REVIEW 4 major objections 5 minor 19 references
SAM-IF: Leveraging SAM for Incremental Few-Shot Instance Segmentation
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read SAM-IF claims that SAM2 can add new object classes from one mask per class by updating only a cosine-similarity classifier's weight matrix.
desk verdict The paper is a thin SAM2 plus iMTFA extension whose own results contradict its headline claim; the missing equations and unquantified evaluation gap make it a desk reject. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the cosine-similarity classifier and its weight matrix W, which stores one normalized prototype vector per class. Mask embeddings from SAM2's decoder enter a small feature extractor, are L2-normalized, and are scored against the normalized class vectors in W with a temperature of gamma = 7. For a novel class, the feature extractor converts the few-shot mask embeddings into a vector w_new; this vector is normalized, averaged over the number of shots, and written directly into W. That weight replacement is the entire incremental update, so no gradient step, no decoder retraining, and no storage of the few-shot examples are needed at test time.
What would settle it
Record the mean cosine similarity between class prototypes for held-out novel classes using the trained feature extractor and SAM2 mask embeddings. If the average between-class similarity is close to or larger than the average within-class similarity for any class pair, the weight-insertion rule cannot separate those classes, and the method's novel-class accuracy would collapse. This measurement is a direct check of the paper's core premise.
Extended reading notes
Core claim
On its own terms, SAM-IF's central discovery is that the incremental few-shot machinery of iMTFA can be transplanted onto SAM2. The paper fine-tunes SAM2's image encoder and mask decoder using randomly sampled foreground points from eroded instance masks, so the model produces class-agnostic instance masks; a separate classifier, trained from scratch, assigns each mask to a base class or to a background class. For novel classes, the method takes the mask embeddings from a few shots, passes them through the feature extractor, normalizes and averages the resulting vectors, and replaces the corresponding rows of the cosine-similarity weight matrix W. This makes category addition a pure weight-update operation. On the 1-shot COCO2014 split, SAM-IF reaches overall AP 17.8 and AP50 27.7; training the image encoder raises base AP from 14.0 to 18.1, and the paper attributes the remaining gap to fragmented masks and COCO's missing instance annotations.
Load-bearing premise
The method depends on the assumption that the internal image features SAM2 produces for one object class are similar enough to each other, and different enough from other classes, that comparing them by cosine similarity can separate the classes.
Editorial extensions
If this is right
- New classes can be added to a deployed segmenter by replacing one weight matrix, so a production system could absorb user-defined categories from a handful of masks without a training run.
- Because inference uses uniformly distributed random points rather than user-supplied boxes or points, the system can find target instances automatically once background predictions are discarded.
- Few-shot learning collapses to a memory operation: storing one normalized vector per new class extends the classifier, so incremental updates require only the embeddings of the few examples.
- The reported 1-shot COCO2014 numbers (overall AP 17.8, AP50 27.7) place SAM-IF near the iMTFA baseline on overall metrics while preserving the ability to add classes incrementally.
Reading between the lines
- The same weight-swap recipe should transfer to any promptable segmenter that emits mask-level embeddings; SAM2 is the test bed, not a requirement of the mechanism.
- Because the paper's own future-work section concedes that SAM embeddings lack structured class features, adding a learned transformation between the mask embedding and the cosine head is a natural extension that could lift novel-class accuracy without changing the incremental protocol.
- The reported base-versus-novel gap (18.1 AP vs 0.5 AP) suggests the cosine classifier works for familiar classes but nearly fails on unseen ones; a stress test with 2, 5, and 10 shots would show whether averaging more prototypes actually improves separation.
- The paper attributes part of the low scores to incomplete COCO annotations and fragmented SAM masks; re-evaluating on a dataset with dense instance annotations could change the numerical comparison with iMTFA.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces SAM-IF, an incremental few-shot instance segmentation method built on SAM2. It fine-tunes SAM2's image encoder and mask decoder with random point prompts for class-agnostic mask prediction, and attaches a cosine-similarity classifier with a background class for multi-class labeling. For novel classes, class weights are computed from mask embeddings and inserted into the classifier without retraining the decoder. The experiments on COCO2014 in the 1-shot setting compare against iMTFA, reporting lower AP on all metrics; the authors attribute the gap to fragmented masks and missing COCO annotations.
Significance. If substantiated, the approach would be a useful SAM-based alternative for incremental few-shot instance segmentation, with the practical appeal of adding new classes without retraining the segmentation decoder. The paper's strengths include a clear task formulation, the use of random point prompts for class-agnostic mask training, and an ablation showing that training the image encoder helps (Table 2). However, the main claim is not currently supported by Table 1, the central update equation is missing, and no code or variance estimates are provided. The significance therefore remains conditional on a substantial revision.
major comments (4)
- [Table 1, Section 4.4] The stated contribution of 'competitive but more reasonable results' is not supported by the reported numbers. SAM-IF is below iMTFA on every reported metric (Overall AP 17.8 vs 20.13; Base AP 18.1 vs 25.9; Novel AP 0.5 vs 2.81; Novel AP50 1.0 vs 4.72), and the gap is largest on the novel classes that the incremental few-shot mechanism is supposed to enable. Section 4.4 proposes missing COCO annotations and fragmented masks as explanations, but provides no quantification. I request an oracle evaluation (e.g., scoring predicted masks against GT masks, or using GT masks as proposals) and corrected metrics; without it, the paper's central claim is unverified.
- [Section 3.2] The central update equation for w_new is absent: the text says 'as shown in the following equation' but no equation is printed, so the reader cannot verify how the novel-class weight vector is computed or how the classifier weight matrix W is replaced. This is the core mechanism of the incremental few-shot claim and must be given explicitly.
- [Section 4.1] The 10 repetitions of the 1-shot evaluation are reported only as means. Since one shot per class is selected randomly, the result may depend heavily on shot selection; the paper should report mean ± standard deviation or per-run results. No code is released, and with the missing equation in Section 3.2 this makes the experiments very hard to reproduce.
- [Section 5] The paper acknowledges that SAM embeddings lack structured feature representations needed for classification, especially for fine-grained or novel classes. Given the very low novel-class AP, the paper should include an experiment that separates classifier error from proposal/segmentation error (e.g., feeding GT masks to the classifier) to determine whether the bottleneck is the classifier or the mask proposals. Without this, the explanation in Section 4.4 is not established.
minor comments (5)
- [Section 3.1] The text says 'The mask encoder produces a low-resolution mask'; this appears to refer to the mask decoder, and the phrase 'cue features' is unclear.
- [Section 3.3] The erosion formula M' = M ◦ K = min(sum(M(i,j) over K), 1) is not a standard definition of binary erosion, which is a min over the neighborhood. The formula should be corrected or clarified.
- [Section 2.2] Several citations appear inaccurate: [6] is cited as FGN but is a one-shot semantic segmentation paper, and [16] is cited for 'Network-in-Network for better semantic segmentation' but the reference is a general NiN paper. The related-work references need a careful pass.
- [Table 1] The row 'SAM-IF Base' reports '-' for novel metrics; its purpose and how it differs from the SAM-IF row should be explained in the text.
- [Figure 1, Figure 2] Figures 1 and 2 are described in the text but are not referenced by number at the points where they are first discussed; please add explicit references.
Circularity Check
No significant circularity: the few-shot classifier weights are computed from labeled support shots and evaluated on a disjoint test split, with the cosine-similarity design adopted from the external iMTFA baseline.
full rationale
The paper's derivation chain is self-contained and non-circular. The few-shot classifier follows iMTFA [3], an external CVPR-published baseline whose design choice is adopted rather than smuggled in via self-citation. Novel-class weights Wavg are computed by averaging normalized support embeddings extracted from COCO training/validation images (Section 3.2), then evaluated on the held-out approximately 5k test images (Section 4.1). Nothing in the reported equations fits any parameter to the test set, and no evaluation metric is defined in terms of a quantity that was also used as a training input. The one place where support features enter the classifier is the weight-update rule itself, which is exactly the standard definition of few-shot prototype classification and does not force the test predictions. The missing w_new equation in Section 3.2 is an exposition gap, not a circular step. The Future Work admission that SAM embeddings are not ideal for classification concerns accuracy and generalization, not derivation independence. There are no load-bearing self-citations; citations [1]-[3] are external prior work. Therefore the central claim does not reduce to its inputs by construction.
Assumptions & free parameters
free parameters (5)
- Cosine similarity scaling factor gamma =
7
- Image encoder and mask decoder learning rate =
0.0001
- Classifier learning rate =
0.005
- Erosion kernel size =
not specified
- NMS threshold and stability score threshold =
not specified
assumptions (4)
- domain assumption Uniformly distributed random point prompts cover all object instances well enough to produce complete masks.
- domain assumption SAM2 mask embeddings are sufficiently class-discriminative for a cosine-similarity classifier.
- domain assumption COCO ground truth annotations are complete enough to give an unbiased evaluation.
- ad hoc to paper Ignoring background loss during mask training while expecting the classifier to suppress background is a valid design.
Cite this review
Pith. "Pith review of SAM-IF: Leveraging SAM for Incremental Few-Shot Instance Segmentation." pith.science (2026). https://pith.science/paper/L7FPBXSV
@misc{pith2026241211034,
author = {Pith},
title = {Pith review of: SAM-IF: Leveraging SAM for Incremental Few-Shot Instance Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/L7FPBXSV}},
note = {Machine review of arXiv:2412.11034}
}
read the original abstract
We propose SAM-IF, a novel method for incremental few-shot instance segmentation leveraging the Segment Anything Model (SAM). SAM-IF addresses the challenges of class-agnostic instance segmentation by introducing a multi-class classifier and fine-tuning SAM to focus on specific target objects. To enhance few-shot learning capabilities, SAM-IF employs a cosine-similarity-based classifier, enabling efficient adaptation to novel classes with minimal data. Additionally, SAM-IF supports incremental learning by updating classifier weights without retraining the decoder. Our method achieves competitive but more reasonable results compared to existing approaches, particularly in scenarios requiring specific object segmentation with limited labeled data.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Eva Rolland, Linus Gustafson, Tsung-Yi Xiao, Spencer Whitehead, Alexander C Berg, Wincent Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643, 2023
arXiv 2023
-
[2]
Sam 2: Segment anything in images and videos
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00...
arXiv 2024
-
[3]
imtfa: Incremental few-shot instance segmentation
Bowen Zhang, Huan Sun, Rui Yang, and Hu Han. imtfa: Incremental few-shot instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 13879– 13888, 2020
work page 2020
-
[4]
Matching networks for one shot learning
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems , 2016
work page 2016
-
[5]
Optimization as a model for few-shot learning
Sharmodeep Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference on Learning Representations (ICLR) , 2017
work page 2017
-
[6]
Rami Ali Shaban, Mohamed M. Elhoseiny, et al. One-shot learning for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017
work page 2017
-
[7]
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS) , 2017
work page 2017
-
[8]
Meta-learning with differentiable convex optimization
Luca Bertinetto, Jiri Valmadre, Joao F Henriques, and et al. Meta-learning with differentiable convex optimization. In Advances in Neural Information Processing Systems (NeurIPS) , 2016
work page 2016
Show all 19 references
-
[9]
Meta networks
Tumai Munkhdalai and Hong Yu. Meta networks. In International Conference on Machine Learning (ICML) , 2017
2017
-
[10]
Human-level concept learning through probabilistic program induction
Brenden M Lake, Ruslan R Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332–1338, 2015
2015
-
[11]
Mask r-cnn
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , pages 2961–2969, 2017
2017
-
[12]
Siamese mask r-cnn for few-shot segmentation
Gabriel Koch, Richard Zemel, et al. Siamese mask r-cnn for few-shot segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018
2018
-
[13]
Learning to segment with few examples
Yuxuan Liu, Yue Huang, and Mingyuan Zhang. Learning to segment with few examples. In European Conference on Computer Vision (ECCV), 2018
2018
-
[14]
Further analysis of few-shot segmentation
Eric Tzeng, Judy Hoffman, Trevor Darrell, et al. Further analysis of few-shot segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2019
2019
-
[15]
Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs
Liang-Chieh Chen, George Papandreou, et al. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
2017
-
[16]
Network in network for medical image segmentation
Zongwei Zhou, Xue Bai, et al. Network in network for medical image segmentation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , 2017
2017
-
[17]
Modeling the background for incremental learning in semantic segmentation
Federico Cermelli, Massimiliano Mancini, Samuel Rota Bulo, Elisa Ricci, and Barbara Caputo. Modeling the background for incremental learning in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9233–9242, 2020
2020
-
[18]
Incremental learning for semantic segmentation
Xialei Liu, Yi Zhu, Zhiwei Xiong, and Xuemin (Sherman) Shen. Incremental learning for semantic segmentation. In IEEE Transactions on Pattern Analysis and Machine Intelligence , 2020
2020
-
[19]
Incremental learning with deep neural networks
Han Zhao, Yuexian Zou, and Jure Leskovec. Incremental learning with deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018. 9
2018
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.