{"id":"bbee5606-a496-459f-a6e9-51597be451f3","arxiv_id":"2505.06217","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper introduces SLCA, a spatially localized channel attention mechanism, to fuse frozen SAM segmentation features into medical image classifiers, improving accuracy across three public datasets.","lead":"This paper freezes the Segment Anything Model's image encoder and uses its features to improve medical image classifiers. A new attention module, SLCA, highlights spatially relevant regions, yielding small accuracy gains on three public medical datasets, especially when training labels are scarce.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gains attributed to frozen SAM features are not isolated from the added SLCA/projector capacity; a random-encoder control is needed before the mechanism claim can be accepted.","rationale":"The reader correctly identifies the fixed block choice (patch embedding, blocks 1/16/32, Conv block) as under-validated: it is selected through a RetinaMNIST-only ablation (Table IV) and then applied to all datasets, so some of the reported gain may be selection-on-test noise. However, that concern is about the optimality of one instantiation. The more load-bearing issue is that no experiment isolates the contribution of SAM's pretrained weights from the trainable SLCA modules and projector head. Table III's direct-add row (60.25 vs. 66.33 baseline) shows that naive fusion can severely hurt, and the sigmoid row (66.75) is only slightly above baseline, so the full system's 68.50 depends on the attention/feature-alignment machinery and extra parameters. Since Tables I-II compare complete systems against unmodified backbones and SAMAug-C, a skeptic cannot tell whether the claimed data-efficiency advantage comes from SAM's segmentation prior or from added capacity that any random feature extractor would provide. The paper gives no parameter counts and no error bars, despite stating five runs. These omissions are correctable, and the experiments are inexpensive enough that the needed control is feasible. I would keep the paper conditional rather than reject it: the proposal is clear and the positive ablation evidence supports a direction, but the central causal attribution to SAM remains unproven until a random/weak-pretraining control is run. This is independent support for the reader's conditional verdict rather than a reason to move it.","tokens_in":8040,"tokens_out":5169,"duration_ms":54408,"concrete_test":"Run the exact Swin Transformer v2 + SLCA + projector pipeline on RetinaMNIST and ISIC 2017 at 10% and 100% training data with the frozen SAM ViT-h encoder replaced by (a) a randomly initialized ViT-h with the same architecture and (b) an ImageNet-pretrained ViT-B/16, keeping all hyperparameters, fusion points, and random seeds identical; report accuracy and standard deviation over the same 5 seeds. If the random/ImageNet encoder variants reproduce the gains, the frozen SAM pretraining is not the causal ingredient and the central claim must be re-scoped.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing gap is the absence of any control that isolates SAM's pretrained representation from the extra trainable modules and parameters introduced by the method. In Table III, fusing SAM features into SENet154 by direct addition degrades accuracy from 66.33 to 60.25; replacing SLCA with a plain sigmoid gives only 66.75, while the full SLCA plus projector head reaches 68.50. This pattern suggests most of the gain could come from the attention/projection machinery rather than from the segmentation semantics of the frozen SAM encoder. The paper reports no parameter or FLOP counts for the compared variants, and the headline gains in Tables I-II are mostly 1-3 accuracy points (one 5.75 point at 10% data) without error bars. The reader's block-selection concern is real but secondary: the extraction blocks are fixed and ablated only on RetinaMNIST (Table IV), and a suboptimal block choice would weaken rather than falsify the central claim. The missing pretraining control threatens the causal mechanism itself, because the same pipeline with a randomly initialized or weakly pretrained encoder could conceivably retain the improvements.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a framework that adapts the frozen image encoder of the Segment Anything Model (SAM) for medical image classification. Features are extracted from five layers of the SAM ViT backbone (patch embedding, the 1st/16th/32nd Transformer blocks, and the Conv block), passed through a proposed Spatially Localized Channel Attention (SLCA) module, and injected into the stages of a backbone CNN or Transformer via residual connections. A task-specific projector head further concatenates downsampled SAM features with the backbone's late feature map. Experiments on RetinaMNIST, BreastMNIST, and ISIC 2017 show accuracy and AUC improvements over the base backbones (ResNet152, SENet154, Swin Transformer v2) and over the SAMAug-C baseline, with larger gains at 10% training data. Ablations on RetinaMNIST evaluate the SLCA module, the projector head, alternative fusion strategies, and alternative choices of SAM layers.","tokens_in":8393,"tokens_out":3821,"duration_ms":38668,"significance":"If the reported gains are real and attributable to SAM's pretrained segmentation knowledge, then the paper offers a practical recipe for reusing a frozen segmentation foundation model for classification with modest extra parameters. The method is straightforward and the experimental setup covers three public datasets, three backbone families, and several data fractions, which is a reasonable scope for a first demonstration. However, the central mechanistic claim -- that the improvements come from SAM's segmentation-based representations -- is currently not isolated from the effect of simply adding trainable capacity. The paper itself reports that direct addition of SAM features degrades accuracy (Table III), and the SLCA plus projector head account for the entire improvement; without a control using a randomly initialized or non-segmentation pretrained encoder, the attribution is unsupported. The missing standard deviations and significance tests further weaken the quantitative claims, many of which are within 1-3 accuracy points. The paper is clearly written and the empirical exploration is useful, but these gaps are load-bearing for the paper's abstract-level claim.","major_comments":[{"comment":"The paper attributes the accuracy gains to SAM's pre-trained segmentation-based features, but no control isolates the contribution of SAM's pretrained representation from the additional trainable modules. In Table III, directly adding SAM features to SENet154 drops accuracy from 66.33 to 60.25; replacing SLCA with a plain sigmoid gives 66.75; the full SLCA plus projector head reaches 68.50. This pattern suggests the SLCA/projector machinery, not the segmentation semantics of the frozen encoder, may be responsible for most of the gain. The authors should run the identical pipeline with a randomly initialized ViT-h encoder (and, ideally, with an ImageNet-pretrained ViT that was not trained for segmentation) while keeping all other components fixed. If the gains persist, the claimed role of segmentation-based features is not supported; if they vanish, the claim is validated. This is a necessary experiment before the central conclusion can be accepted.","section":"Section III, Table III and Sections II-A/II-C"},{"comment":"The manuscript states that five runs with different seeds were performed and averages reported, but no standard deviations, confidence intervals, or significance tests are given anywhere. Many of the headline improvements are small (e.g., Table I, Swin Transformer v2 at 100% RetinaMNIST: 68.25 to 69.75; Table II, SENet154 at 10% ISIC 2017: 75.33 to 76.67), and the main data-efficiency claim rests on a 5.75-point gain for a single dataset-backbone combination. Without variance estimates, the reader cannot judge whether these differences are reproducible or within run-to-run noise. The authors should report per-run results or error bars for all entries, and perform a paired significance test (e.g., Wilcoxon signed-rank) for the key comparisons.","section":"Section III, 'Experimental Results' and Tables I-II"},{"comment":"The choice of SAM layers (patch embedding, 1st, 16th, 32nd Transformer blocks, and Conv block) is a fixed design decision that is validated only on RetinaMNIST (Table IV). It is not shown that this selection transfers to BreastMNIST and ISIC 2017, and the text does not clarify whether the selection was made using the test set or a hold-out validation set. If the layer choice was tuned on the same data used for evaluation, the reported numbers are optimistically biased. The authors should either provide the same block-choice ablation on the other two datasets or justify the transferability of the fixed choice, and they should state explicitly how the selection was made.","section":"Section II-C, Table IV"}],"minor_comments":[{"comment":"The text says 'while our SCLA aims to capture...' but the module is named SLCA; this is a typographical error.","section":"Section II-B"},{"comment":"The phrase 'we append the SAM image encoder with an additional task-specific projector head for fine-tuning' is ambiguous. Earlier the SAM encoder is described as frozen; clarify that the projector head is trainable while the SAM encoder remains frozen, and use consistent terminology throughout.","section":"Section II-C"},{"comment":"The 'sigmoid' baseline in Table III is not precisely defined. Since SLCA ends with a sigmoid activation, the reader cannot tell what the 'sigmoid' variant actually changes (e.g., removing the spatial local average pooling, or applying sigmoid directly to the feature map). Please specify the exact operation.","section":"Section III, Ablation Study"},{"comment":"The paper claims SLCA is 'computationally efficient, adding minimal overhead' but reports no parameter counts, FLOPs, or runtime comparisons. A quantitative statement of the added cost would support this claim.","section":"Section III, Implementation Details"},{"comment":"The sentence 'The less the annotations used, the greater the performance improvements' is informal and imprecise; it should be clarified as 'the accuracy gains over the baseline are larger when fewer labeled training samples are used'.","section":"Section III, Experimental Results"},{"comment":"No code or model weights are provided. Given that the method relies on a specific set of extraction layers and fusion hyperparameters, releasing the implementation would materially aid reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is methodologically on the right track but currently under-supports its central attribution claim. The missing random-encoder control is the key gate; without it, the title's promise ('adapting a segmentation foundation model') is not established. The small dataset sizes and lack of error bars also make the level of evidence better suited to a workshop or a methods paper pending a substantial revision. I would encourage the editor to request the control and variance reporting before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, clearly written empirical paper. Freezing SAM and using its features to help standard classifiers via a new attention module produces small but consistent gains on three medical datasets, with the largest gains at 10% labels. It's a useful incremental contribution, not a breakthrough.\n\nWhat's actually new: the specific combination of extracting features from five SAM layers (patch embedding, blocks 1/16/32, conv block) and passing them through the proposed SLCA module before residual fusion into ResNet/SENet/Swin. The ablation against SAMAug-C is fair and the method wins on nearly every metric. That matters because SAMAug-C is the closest prior work and is cited honestly.\n\nThe soft spots are real. The most load-bearing one is attribution. Table III shows that directly adding SAM features to SENet154 drops accuracy from 66.33 to 60.25; replacing SLCA with a plain sigmoid gets only 66.75; the full SLCA plus projector head gets 68.50. The method adds a lot of trainable machinery, so the gains could be coming from the attention and projector rather than from SAM's pretrained segmentation semantics. There is no control with a randomly initialized encoder or a non-segmentation pretrained encoder. Without that, the paper cannot support its claim that segmentation-based features are what helps. The stress-test note is right; this is the main thing I'd want fixed.\n\nSecondary: the block selection is ablated only on RetinaMNIST. It's plausible the chosen layers transfer, but that is unverified on the other datasets. Also, five runs are averaged with no error bars or significance tests, so 1-2 point differences are shaky, and no code is released.\n\nDespite those, the paper is worth a serious referee. The method is clearly described, the empirical trend is consistent, and the data-efficiency result at 10% labels is useful. I'd send it out, but ask for the random-encoder control, error bars, and validation of the block choice on at least one more dataset. If those hold, it's an accept. As is, conditional.","headline":"Solid incremental paper with a clear method and fair comparisons, but the missing random-encoder control prevents the authors from claiming SAM's features are the cause of the gains.","tokens_in":8776,"tokens_out":3041,"would_cite":true,"duration_ms":29896,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a frozen SAM encoder improves medical image classification when its multi-depth features are fused through SLCA attention into standard classifiers, especially under scarce labels.","keywords":["Segment Anything Model","medical image classification","frozen feature extractor","channel attention","spatially localized attention","data efficiency","transfer learning","multi-scale features"],"falsifier":"Run the same training protocol on a held-out medical dataset, comparing the fixed five-block extraction with the best single-block extraction chosen by an ablation on that dataset; if single-block matches or beats the five-block fusion, the paper's core claim that multi-block fusion drives the gain would fail.","tokens_in":7889,"feed_emoji":"🩺","tokens_out":8358,"duration_ms":76710,"temperature":0.7,"pith_summary":"The paper sets out to show that a frozen Segment Anything Model (SAM) image encoder can be repurposed for medical image classification instead of for segmentation. The proposed method extracts feature maps from five depths of SAM's encoder, runs them through a Spatially Localized Channel Attention (SLCA) module, and fuses them into standard classification backbones via residual connections. On RetinaMNIST, BreastMNIST, and ISIC 2017, the fused models report higher accuracy and AUC than their unfused baselines and than SAMAug-C. The largest gains occur under label scarcity, with a 5.75% accuracy improvement for Swin Transformer v2 at 10% of RetinaMNIST training labels. A sympathetic reader would take this as evidence that segmentation features can be imported into classifiers without training the foundation model.","feed_headline":"Frozen SAM boosts medical classification, most with few labels","feed_subtitle":"Multi-block SAM features via SLCA attention lift accuracy across three datasets; gains peak at 5.75 points with 10 percent labels.","key_machinery":"The load-bearing machinery is the frozen SAM image encoder sampled at five depths — patch embedding, Transformer blocks 1, 16, and 32, and the Conv block — together with a fusion pathway. Each extracted feature map $F$ is passed through SLCA, which computes spatially localized channel weights as $\\mathrm{SLCA}(F) = \\sigma(\\mathrm{Conv}_2(\\mathrm{ReLU}(\\mathrm{Conv}_1(\\mathrm{SLAP}(F)))))$, where $\\mathrm{SLAP}$ is spatial local average pooling, then applies the weights to the classifier's intermediate features and adds them by residual connection. A lightweight projector head resizes SAM features to the classifier's channel count and resolution before concatenation. The same five extraction points are used for all backbones and datasets.","core_discovery":"The central claim is that segmentation-oriented features from a frozen SAM image encoder can be injected into existing classifiers at multiple stages without any SAM fine-tuning, and that the way the injection happens matters. Direct additive fusion of SAM features hurts performance, while applying SLCA attention before a residual fusion, plus a small projector head, gives consistent accuracy and AUC gains across three public medical datasets and three classification backbones (ResNet152, SENet154, and Swin Transformer v2). The gains are largest when only 10% of the training labels are used, which the paper reads as evidence of data efficiency. The paper also reports that its fusion approach outperforms the SAMAug-C augmentation baseline on nearly all reported settings.","pith_inferences":["Inference: because the five extraction depths were selected on RetinaMNIST alone, the method would probably gain from per-dataset selection of extraction depths; a search over subsets on BreastMNIST and ISIC 2017 would show whether the fixed choice is the main driver of the reported margins.","Inference: if SLCA's spatial-localization story is right, the same frozen-encoder fusion should transfer to tasks with small localized findings, such as chest X-ray pathology or histology tiles, and might extend to 3D imaging with a volumetric encoder; neither test appears in the paper.","Inference: the comparison to SAMAug-C treats SAM as an augmentation source, whereas this method treats it as a feature source; a natural next experiment is to combine both uses and measure the marginal gain."],"forward_implications":["Using the frozen SAM encoder as a feature source improves classification on RetinaMNIST, BreastMNIST, and ISIC 2017 across ResNet152, SENet154, and Swin Transformer v2.","The gains grow as training labels shrink; for example, at 10% of RetinaMNIST labels, the method improves Swin Transformer v2 accuracy by 5.75 percentage points, suggesting data efficiency.","SLCA fusion outperforms both direct additive fusion and a plain sigmoid attention baseline, and the full method beats SAMAug-C on nearly all tested settings.","Adding the projector head to the SAM feature branch yields a further accuracy gain, indicating that the segmentation encoder benefits from a task-specific readout."],"supporting_citations":[{"why":"Supplies the frozen SAM image encoder whose multi-depth features are the main feature source of the method.","marker":"[4]"},{"why":"The prior SAM-based augmentation baseline that the paper compares against and aims to outperform.","marker":"[13]"},{"why":"ResNet152, one of the classification backbones whose stages receive the SLCA-weighted SAM features.","marker":"[1]"},{"why":"SENet154, another backbone used in the main experiments and in the ablation study of SLCA.","marker":"[2]"},{"why":"Swin Transformer v2, the Transformer backbone used to demonstrate the largest gains at 10% training data.","marker":"[3]"},{"why":"Defines the Vision Transformer architecture whose patch embedding, Transformer blocks, and Conv block are the extraction points.","marker":"[14]"},{"why":"Provides the masked autoencoder pretraining that makes SAM's encoder a useful frozen feature source.","marker":"[15]"},{"why":"The efficient channel attention method that the paper contrasts with SLCA to position its new attention design.","marker":"[20]"},{"why":"Defines the RetinaMNIST and BreastMNIST datasets used for the main classification experiments.","marker":"[21]"}],"fun_headline_variants":["Frozen SAM boosts medical classifiers with few labels","SLCA attention makes frozen SAM work for classification","Multi-stage SAM fusion lifts accuracy at 10% labels","No SAM fine-tuning: features improve medical classification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The fixed set of SAM layers used for feature extraction (patch embedding, blocks 1, 16, 32, and the Conv block) is chosen from an ablation on one dataset and is assumed to transfer to other datasets and backbones without per-dataset revalidation.","fun_headline_variants_meta":{"raw":{"variants":["Frozen SAM boosts medical classifiers with few labels","SLCA attention makes frozen SAM work for classification","Multi-stage SAM fusion lifts accuracy at 10% labels","No SAM fine-tuning: features improve medical classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1371,"prompt_tokens":862,"completion_tokens":509,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":448}},"tokens_in":478,"tokens_out":509,"duration_ms":5286,"temperature":1.0,"reasoning_tokens":448,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:44:56.342751+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same training protocol on a held-out medical dataset, comparing the fixed five-block extraction with the best single-block extraction chosen by an ablation on that dataset; if single-block matches or beats the five-block fusion, the paper's core claim that multi-block fusion drives the gain would fail.","supporting_citations":[{"cited_title":"Boosting medical image classification with segmentation foundation model,","cited_arxiv_id":null,"evidence_quote":"The prior SAM-based augmentation baseline that the paper compares against and aims to outperform."},{"cited_title":"Deep residual learning for image recognition,","cited_arxiv_id":null,"evidence_quote":"ResNet152, one of the classification backbones whose stages receive the SLCA-weighted SAM features."},{"cited_title":"Squeeze-and-excitation networks,","cited_arxiv_id":null,"evidence_quote":"SENet154, another backbone used in the main experiments and in the ablation study of SLCA."},{"cited_title":"Swin Transformer v2: Scaling up capacity and resolution,","cited_arxiv_id":null,"evidence_quote":"Swin Transformer v2, the Transformer backbone used to demonstrate the largest gains at 10% training data."},{"cited_title":"Masked autoencoders are scalable vision learners,","cited_arxiv_id":null,"evidence_quote":"Provides the masked autoencoder pretraining that makes SAM's encoder a useful frozen feature source."},{"cited_title":"ECA-Net: Efficient channel attention for deep convolutional neural networks,","cited_arxiv_id":null,"evidence_quote":"The efficient channel attention method that the paper contrasts with SLCA to position its new attention design."},{"cited_title":"MedMNIST classification decathlon: A lightweight autoML benchmark for medical image analysis,","cited_arxiv_id":null,"evidence_quote":"Defines the RetinaMNIST and BreastMNIST datasets used for the main classification experiments."}],"review_version":1}