REVIEW 3 major objections 5 minor 47 references
Unlocking the Power of SAM 2 for Few-Shot Segmentation
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read FSSAM reuses SAM 2's video memory matching for few-shot segmentation by matching query features against pseudo query memories instead of support features, and reports state-of-the-art mIoU on PASCAL-5i and COCO-20i.
desk verdict A genuinely good SAM 2 adaptation for few-shot segmentation, but a ten-point fold-1 mIoU discrepancy across tables makes the headline 81.0 and the 4.2% margin unsupported until fixed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Because the rough masks are imperfect, two extra modules fix them. Iterative Memory Refinement gradually copies foreground features from a more complete but noisier mask into a cleaner mask. Support-Calibrated Memory Attention uses the clean support foreground as a reference to lower the attention paid to background pixels that leaked into the memory. On the standard PASCAL-5i and COCO-20i benchmarks, the final system reports 81.0% and 62.3% mean IoU in the 1-shot setting, ahead of previous methods. The authors also report ablations showing each module helps, and they include error bars and additional datasets in the appendix.
Extended reading notes
Core claim
The load-bearing assertion, from Section 5.2: 'FSSAM can surpass other baselines by large margins, setting new state-of-the-arts. For example, the 1-shot mIoU score is 4.2% better than that of the best baseline FounFSS, and the FB-IoU score can reach 89.4%.' If the paper is correct, adapting SAM 2 via pseudo query memories, iterative refinement, and support-calibrated attention yields 1-shot mIoU of 81.0 on PASCAL-5i and 62.3 on COCO-20i, with each designed module contributing to the gain in the ablation study.
Load-bearing premise
The design rests on the assumption that the pseudo query masks generated by DINOv2 prototypes are reliable enough to serve as SAM 2 mask prompts: if the prior completely misses the foreground or is dominated by background, the encoded memory is wrong and the method has no correction path. The authors acknowledge this in Appendix E ('the real FG is completely uncovered'). A second load-bearing premise is that cosine similarity between the support foreground prototype and query memory features cleanly separates query FG from query BG in IMR Eq. (8) and SCMA Eq. (12)-(13); the paper shows examples but gives no quantitative failure analysis.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FSSAM, a method to adapt SAM 2 for few-shot segmentation. It identifies an incompatibility between SAM 2's same-object video matching and the different-object matching required by FSS, and addresses it with three components: a Pseudo Prompt Generator (PPG) that creates pseudo query masks encoded as query memory, an Iterative Memory Refinement (IMR) that progressively fuses foreground features into the discriminative memory, and a Support-Calibrated Memory Attention (SCMA) that suppresses background features in the memory during cross attention. The method is evaluated on PASCAL-5i, COCO-20i, LVIS-92i, and PASCAL-Part, reporting 1-shot mIoU of 81.0 on PASCAL-5i and 62.3 on COCO-20i, with component ablations, error bars, and qualitative visualizations. The authors also release code and state that the proposed modules introduce no additional learnable parameters.
Significance. If the reported results are correct, the paper makes a useful contribution by showing how a video foundation model (SAM 2) can be adapted to few-shot segmentation through a pseudo-prompt mechanism, rather than treating SAM 2 as a black-box promptable segmenter. The design is transparent, the modules are simple and parameter-free, and the appendices provide error bars, multi-episode testing, model-size studies, and additional datasets, which are all strengths. However, the central quantitative claim is currently undermined by an unreconciled inconsistency in the PASCAL-5i fold-1 score, and by test-set-based selection of the IMR iteration count. These issues must be resolved before the state-of-the-art claim can be trusted.
major comments (3)
- [Section 5.2, Table 1 vs. Table 3 and Table 11] The same final configuration (PPG+IMR+SCMA, fine-tuned) reports fold-1 mIoU of 84.9 in Table 1, but 74.9 in Table 3 (the full ablation row) and in Table 11 (FSSAM w/ FT). The four-fold mean in Tables 3 and 11 is still reported as 81.0, which is arithmetically impossible with the stated per-fold values (81.6+74.9+81.6+76.0)/4 = 78.5, not 81.0. This inconsistency directly affects the headline claim of a 4.2% margin over FounFSS: if the 74.9 value were correct, the margin would be about 1.7 points. The authors must reconcile these numbers, state which value is correct, and correct the affected means and conclusions. This is a load-bearing issue and cannot be left as an unexplained discrepancy.
- [Section 5.3, Table 4] The IMR iteration count n is selected as the peak of the test-set performance curve: Table 4 reports n=1,2,3,4 on PASCAL-5i and the paper sets n=3 because it gives the highest mean (81.0). This is test-set model selection: the hyperparameter is chosen using the same benchmark on which the final comparison is made, which can inflate the reported accuracy. The authors should either select n on a validation split, report the results for all values of n in the comparison with baselines, or clearly characterize the sensitivity. The effect is not negligible: n=1 gives 79.3 and n=2 gives 80.3, so the choice of n contributes up to 1.7 points to the headline number. Similarly, the SCMA scaling factor alpha in Eq. (13) is empirically set to 10; the selection procedure for alpha should be described.
- [Appendix C.5, Table 10] Table 10 is internally inconsistent. For Matcher, the four fold values 37.1, 32.4, 33.7, 38.1 average to 35.3, not the reported 42.9; for FSSAM, the values 34.7, 37.8, 37.2, 41.1 average to 37.7, not the reported 46.4. Moreover, the FSSAM fold values in Table 10 are identical to the first four folds of the LVIS-92i results in Table 9, which suggests that the rows may have been mis-copied or that the mean column corresponds to a different experimental setting. Since the paper uses these results to claim generalizability beyond PASCAL and COCO, the table must be corrected or the experimental protocol clarified.
minor comments (5)
- [Section 4.1] The Pseudo Prompt Generator is described as a contribution, but Equations (2)-(4) follow AENet's FG/BG prototype subtraction almost directly. The authors do cite AENet, but the novelty should be scoped more carefully: the contribution is the use of such priors as pseudo mask prompts for SAM 2, not the prior-generation mechanism itself.
- [Equations (8) and (13)] The notation AQQ is reused for different quantities: in Eq. (8) it is a similarity map between M_em^FG_Q and a prototype, while in Eq. (12)-(13) it is the query-memory attention map. Please use distinct symbols or clearly distinguish the two roles to improve readability.
- [Appendix C.1, Table 5] The COCO-20i fold-level error bars are fairly large (e.g., fold-0 ranges from 55.8 to 61.8 across seeds), yet the main Table 2 reports only the seed-321 result. Consider reporting the mean over multiple seeds in the main comparison to avoid overstating precision.
- [Table 3] The ablation table rows labeled only with checkmarks are hard to parse because the column order (PPG, IMR, SCMA) is clear in the header but the rows with two checkmarks do not indicate which one is omitted. Explicitly labeling each row, or adding a note, would improve clarity.
- [Abstract and Section 5.2] The phrase 'the 1-shot mIoU can be 4.2% better than the best baseline' is stated as a general result, but it refers specifically to PASCAL-5i. Please state the dataset explicitly in the abstract to avoid ambiguity.
Assumptions & free parameters
free parameters (2)
- SCMA scaling factor alpha =
10
- IMR iteration count n =
3
assumptions (3)
- domain assumption SAM 2's memory attention learned same-object FG-FG matching during video pretraining, and this knowledge is incompatible with matching two different object instances of the same class.
- domain assumption DINOv2 global-average-pooled support prototypes yield reliable FG and BG prior masks for novel classes.
- domain assumption Cosine similarity between the support FG prototype and query memory features separates query FG from query BG well enough to suppress BG attention.
Cite this review
Pith. "Pith review of Unlocking the Power of SAM 2 for Few-Shot Segmentation." pith.science (2026). https://pith.science/paper/HR2S2KVX
@misc{pith2026250514100,
author = {Pith},
title = {Pith review of: Unlocking the Power of SAM 2 for Few-Shot Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/HR2S2KVX}},
note = {Machine review of arXiv:2505.14100}
}
abstract
Few-Shot Segmentation (FSS) aims to learn class-agnostic segmentation on few classes to segment arbitrary classes, but at the risk of overfitting. To address this, some methods use the well-learned knowledge of foundation models (e.g., SAM) to simplify the learning process. Recently, SAM 2 has extended SAM by supporting video segmentation, whose class-agnostic matching ability is useful to FSS. A simple idea is to encode support foreground (FG) features as memory, with which query FG features are matched and fused. Unfortunately, the FG objects in different frames of SAM 2's video data are always the same identity, while those in FSS are different identities, i.e., the matching step is incompatible. Therefore, we design Pseudo Prompt Generator to encode pseudo query memory, matching with query features in a compatible way. However, the memories can never be as accurate as the real ones, i.e., they are likely to contain incomplete query FG, and some unexpected query background (BG) features, leading to wrong segmentation. Hence, we further design Iterative Memory Refinement to fuse more query FG features into the memory, and devise a Support-Calibrated Memory Attention to suppress the unexpected query BG features in memory. Extensive experiments have been conducted on PASCAL-5$^i$ and COCO-20$^i$ to validate the effectiveness of our design, e.g., the 1-shot mIoU can be 4.2% better than the best baseline.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
Relevant intrinsic feature enhancement network for few-shot semantic segmentation
Bao, X., Qin, J., Sun, S., Wang, X., and Zheng, Y. Relevant intrinsic feature enhancement network for few-shot semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 765--773, 2024
work page 2024
-
[3]
High-Performance Few-Shot Segmentation with Foundation Models: An Empirical Study
Chang, S., Zhang, L., and Lu, H. High-performance few-shot segmentation with foundation models: An empirical study. arXiv preprint arXiv:2409.06305, 2024
work page Pith review arXiv 2024
-
[4]
Chen, S., Meng, F., Wu, C., Wei, H., Zhang, R., Wu, Q., Xu, L., and Li, H. No re-train, more gain: Upgrading backbones with diffusion model for few-shot segmentation. arXiv preprint arXiv:2407.16182, 2024
work page Pith review arXiv 2024
-
[5]
Detect what you can: Detecting and representing objects using holistic models and body parts
Chen, X., Mottaghi, R., Liu, X., Fidler, S., Urtasun, R., and Yuille, A. Detect what you can: Detecting and representing objects using holistic models and body parts. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 1971--1978, 2014
work page 1971
-
[6]
K., Winn, J., and Zisserman, A
Everingham, M., Van Gool, L., Williams, C. K., Winn, J., and Zisserman, A. The pascal visual object classes (voc) challenge. International journal of computer vision, 88 0 (2): 0 303--338, 2010
work page 2010
-
[7]
Self-support few-shot semantic segmentation
Fan, Q., Pei, W., Tai, Y.-W., and Tang, C.-K. Self-support few-shot semantic segmentation. In Computer Vision--ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23--27, 2022, Proceedings, Part XIX, pp.\ 701--719. Springer, 2022
work page 2022
-
[8]
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023
arXiv 2023
Show all 47 references
-
[9]
Lvis: A dataset for large vocabulary instance segmentation
Gupta, A., Dollar, P., and Girshick, R. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 5356--5364, 2019
2019
-
[10]
Simultaneous detection and segmentation
Hariharan, B., Arbel \'a ez, P., Girshick, R., and Malik, J. Simultaneous detection and segmentation. In European conference on computer vision, pp.\ 297--312. Springer, 2014
2014
-
[11]
Cost aggregation with 4d convolutional swin transformer for few-shot segmentation
Hong, S., Cho, S., Nam, J., Lin, S., and Kim, S. Cost aggregation with 4d convolutional swin transformer for few-shot segmentation. In European Conference on Computer Vision, pp.\ 108--126. Springer, 2022
2022
-
[12]
Hu, T., Yang, P., Zhang, C., Yu, G., Mu, Y., and Snoek, C. G. Attention-based multi-context guiding for few-shot semantic segmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp.\ 8441--8448, 2019
2019
-
[13]
Msanet: Multi-similarity and attention guidance for boosting few-shot segmentation
Iqbal, E., Safarov, S., and Bang, S. Msanet: Multi-similarity and attention guidance for boosting few-shot segmentation. arXiv preprint arXiv:2206.09667, 2022
2022 arXiv
-
[14]
C., Lo, W.-Y., et al
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4015--4026, 2023
2023
-
[15]
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll \'a r, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In European conference on computer vision, pp.\ 740--755. Springer, 2014
2014
-
[16]
Fecanet: Boosting few-shot semantic segmentation with feature-enhanced context-aware network
Liu, H., Peng, P., Chen, T., Wang, Q., Yao, Y., and Hua, X.-S. Fecanet: Boosting few-shot semantic segmentation with feature-enhanced context-aware network. IEEE Transactions on Multimedia, 2023 a
2023
-
[17]
Matcher: Segment anything with one shot using all-purpose feature matching
Liu, Y., Zhu, M., Li, H., Chen, H., Wang, X., and Shen, C. Matcher: Segment anything with one shot using all-purpose feature matching. arXiv preprint arXiv:2305.13310, 2023 b
2023 arXiv
-
[18]
A simple image segmentation framework via in-context examples
Liu, Y., Jing, C., Li, H., Zhu, M., Chen, H., Wang, X., and Shen, C. A simple image segmentation framework via in-context examples. arXiv preprint arXiv:2410.04842, 2024
2024 arXiv
-
[19]
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 3431--3440, 2015
2015
-
[20]
V-net: Fully convolutional neural networks for volumetric medical image segmentation
Milletari, F., Navab, N., and Ahmadi, S.-A. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pp.\ 565--571. IEEE, 2016
2016
-
[21]
Attention-based joint detection of object and semantic part
Morabia, K., Arora, J., and Vijaykumar, T. Attention-based joint detection of object and semantic part. arXiv preprint arXiv:2007.02419, 2020
2007 arXiv
-
[22]
and Todorovic, S
Nguyen, K. and Todorovic, S. Feature weighting and boosting for few-shot segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 622--631, 2019
2019
-
[23]
Dinov2: Learning robust visual features without supervision
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023
2023 arXiv
-
[24]
S., and Heo, J.-P
Park, S., Lee, S., Hyun, S., Seong, H. S., and Heo, J.-P. Task-disruptive background suppression for few-shot segmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp.\ 4442--4449, 2024
2024
-
[25]
Hierarchical dense correlation distillation for few-shot segmentation
Peng, B., Tian, Z., Wu, X., Wang, C., Liu, S., Su, J., and Jia, J. Hierarchical dense correlation distillation for few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 23641--23651, 2023
2023
-
[26]
Sam 2: Segment anything in images and videos
Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali, C., Ma, T., Khedr, H., R \"a dle, R., Rolland, C., Gustafson, L., et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024
2024 arXiv
-
[27]
One-shot learning for semantic segmentation
Shaban, A., Bansal, S., Liu, Z., Essa, I., and Boots, B. One-shot learning for semantic segmentation. arXiv preprint arXiv:1709.03410, 2017
2017 arXiv
-
[28]
Dense cross-query-and-support attention weighted mask aggregation for few-shot segmentation
Shi, X., Wei, D., Zhang, Y., Lu, D., Ning, M., Chen, J., Ma, K., and Zheng, Y. Dense cross-query-and-support attention weighted mask aggregation for few-shot segmentation. In European Conference on Computer Vision, pp.\ 151--168. Springer, 2022
2022
-
[29]
Vrp-sam: Sam with visual reference prompt
Sun, Y., Chen, J., Zhang, S., Zhang, X., Chen, Q., Zhang, G., Ding, E., Wang, J., and Li, Z. Vrp-sam: Sam with visual reference prompt. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 23565--23574, 2024
2024
-
[30]
Prior guided feature enrichment network for few-shot segmentation
Tian, Z., Zhao, H., Shu, M., Yang, Z., Li, R., and Jia, J. Prior guided feature enrichment network for few-shot segmentation. IEEE transactions on pattern analysis and machine intelligence, 2020
2020
-
[31]
N., Kaiser, ., and Polosukhin, I
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017
2017
-
[32]
Few-shot semantic segmentation with democratic attention networks
Wang, H., Zhang, X., Hu, Y., Yang, Y., Cao, X., and Zhen, X. Few-shot semantic segmentation with democratic attention networks. In European Conference on Computer Vision, pp.\ 730--746. Springer, 2020
2020
-
[33]
Adaptive fss: a novel few-shot segmentation framework via prototype enhancement
Wang, J., Li, J., Chen, C., Zhang, Y., Shen, H., and Zhang, T. Adaptive fss: a novel few-shot segmentation framework via prototype enhancement. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp.\ 5463--5471, 2024 a
2024
-
[34]
Rethinking prior information generation with clip for few-shot segmentation
Wang, J., Zhang, B., Pang, J., Chen, H., and Liu, W. Rethinking prior information generation with clip for few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 3941--3951, 2024 b
2024
-
[35]
H., Zou, Y., Zhou, D., and Feng, J
Wang, K., Liew, J. H., Zou, Y., Zhou, D., and Feng, J. Panet: Few-shot image semantic segmentation with prototype alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 9197--9206, 2019
2019
-
[36]
Focus on query: Adversarial mining transformer for few-shot segmentation
Wang, Y., Luo, N., and Zhang, T. Focus on query: Adversarial mining transformer for few-shot segmentation. Advances in Neural Information Processing Systems, 36: 0 31524--31542, 2023 a
2023
-
[37]
Rethinking the correlation in few-shot segmentation: A buoys view
Wang, Y., Sun, R., and Zhang, T. Rethinking the correlation in few-shot segmentation: A buoys view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 7183--7192, 2023 b
2023
-
[38]
Xiong, Z., Li, H., and Zhu, X. X. Doubly deformable aggregation of covariance matrices for few-shot segmentation. In Computer Vision--ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23--27, 2022, Proceedings, Part XX, pp.\ 133--150. Springer, 2022
2022
-
[39]
Self-calibrated cross attention network for few-shot segmentation
Xu, Q., Zhao, W., Lin, G., and Long, C. Self-calibrated cross attention network for few-shot segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 655--665, 2023
2023
-
[40]
C., Long, C., Li, Z., and Zhao, R
Xu, Q., Lin, G., Loy, C. C., Long, C., Li, Z., and Zhao, R. Eliminating feature ambiguity for few-shot segmentation. In European Conference on Computer Vision, pp.\ 416--433. Springer, 2024 a
2024
-
[41]
Hybrid mamba for few-shot segmentation
Xu, Q., Liu, X., Zhu, L., Lin, G., Long, C., Li, Z., and Zhao, R. Hybrid mamba for few-shot segmentation. Advances in Neural Information Processing Systems, 37: 0 73858--73883, 2024 b
2024
-
[42]
Mianet: Aggregating unbiased instance and general information for few-shot semantic segmentation
Yang, Y., Chen, Q., Feng, Y., and Huang, T. Mianet: Aggregating unbiased instance and general information for few-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 7131--7140, 2023
2023
-
[43]
Bridge the points: Graph-based few-shot segment anything semantically
Zhang, A., Gao, G., Jiao, J., Liu, C., and Wei, Y. Bridge the points: Graph-based few-shot segment anything semantically. Advances in Neural Information Processing Systems, 37: 0 33232--33261, 2024
2024
-
[44]
Pyramid graph networks with connection attentions for region-based one-shot semantic segmentation
Zhang, C., Lin, G., Liu, F., Guo, J., Wu, Q., and Yao, R. Pyramid graph networks with connection attentions for region-based one-shot semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 9587--9595, 2019 a
2019
-
[45]
Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning
Zhang, C., Lin, G., Liu, F., Yao, R., and Shen, C. Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 5217--5226, 2019 b
2019
-
[46]
Few-shot segmentation via cycle-consistent transformer
Zhang, G., Kang, G., Yang, Y., and Wei, Y. Few-shot segmentation via cycle-consistent transformer. Advances in Neural Information Processing Systems, 34: 0 21984--21996, 2021
2021
-
[47]
Llafs: When large language models meet few-shot segmentation
Zhu, L., Chen, T., Ji, D., Ye, J., and Liu, J. Llafs: When large language models meet few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 3065--3075, 2024
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.