REVIEW 4 major objections 5 minor 27 references
Segment Any Class (SAC): Multi-Class Few-Shot Semantic Segmentation via Class Region Proposals
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A training-free pipeline of frozen DINOv2 features and SAM point prompts claims to outperform gradient-trained multi-class few-shot segmenters on COCO-20i, with the gap widening as class count grows.
desk verdict Plausible training-free multi-class FSS pipeline that beats trained baselines on COCO-20i, but the unspecified filtering thresholds make the headline numbers hard to trust. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the Class Region Proposal (CRP), built from Class-Representative Feature Arrays (CRFAs). Support images pass through frozen DINOv2; pixels of each class (plus a background class) are k-means clustered into $n_{\text{cluster}}=5$ prototype vectors; cosine similarity between the query embedding and each prototype yields per-pixel score maps; Otsu thresholding keeps only high-similarity pixels; inter-class overlap filtering and average-cosine filtering discard conflicting or low-confidence masks. These region proposals become SAM point prompts, with positive points from clustered proposal locations paired with negative points from the background proposal, so the frozen SAM mask decoder does the segmentation.
What would settle it
Run the full pipeline on COCO-20i with the support set's class masks replaced by random masks of the same shape. If mIoU stays far above the chance level, the class signal is not coming from the support annotations and the Class Region Proposal geometry is doing no work; if the score collapses, the mechanism depends on the claimed feature alignment.
Extended reading notes
Core claim
SAC's central claim is that elementary feature geometry in a frozen self-supervised vision model is sufficient to adapt a class-agnostic segmenter to a new multi-class segmentation task. The method clusters support pixels per class (and background) into Class-Representative Feature Arrays, computes cosine-similarity maps between query embeddings and each array, thresholds these maps with Otsu, and uses the surviving regions as class region proposals. Positive point prompts are drawn from each proposal, negative prompts from a background region proposal, and spatially clustered to create prompt sets for SAM; overlapping low-confidence masks are filtered. On COCO-20i, SAC reports 1-way 1-shot mIoU of 49.2, within about two points of DCAMA's 50.9, while at 5-way it reaches 44.0 against DCAMA's 16.7 and Label Anything's 27.7, and at 20-way 37.9 against 4.7 and 13.7 respectively, with no weight updates.
Load-bearing premise
The argument rests on the premise that DINOv2's frozen embeddings place pixels of the same class close together in cosine space on unseen query images, so that the cluster prototypes and the fixed Otsu and overlap thresholds yield point prompts that land on true class instances; if the features do not transfer, or the thresholds are miscalibrated, every downstream mask inherits the error.
Editorial extensions
If this is right
- Adding a new class to SAC requires only adding its support images and re-running the clustering; no retraining or weight update is needed, so previously learned classes cannot be overwritten.
- Segmentation accuracy degrades far more slowly as the number of classes grows: SAC drops from 49.2 mIoU at 1-way to 37.9 at 20-way (1-shot), whereas DCAMA falls from 50.9 to 4.7 and Label Anything from 43.1 to 13.7.
- Because the pipeline is prompt-only, the same frozen SAM and DINOv2 weights can be swapped between tasks online, without storing per-task model copies.
- The ablation attributes most of the gain to the feature extractor: replacing DINOv2 with SAM's image encoder drops 1-way 1-shot mIoU from 45.5 to 26.6, indicating that the frozen features, not the mask decoder, carry the adaptation.
- The method accommodates any N-way K-shot configuration, so the same machinery applies to one-shot, five-shot, or any other support-set size without changing the algorithm.
Reading between the lines
- A testable consequence the paper does not pursue: because the proposal generator is prompt-agnostic, the same CRFAs should drive newer point-promptable segmenters such as SAM 2, which would isolate where the gain comes from.
- The fixed hyperparameters ($n_{\text{cluster}}=5$, Otsu thresholding, overlap and confidence cutoffs) are calibrated on COCO-20i; on a benchmark with dissimilar class scales, the same defaults may miscalibrate, so measuring per-class accuracy by object size would reveal the boundary of the claim.
- The paper implicitly redefines adaptation as prompt selection rather than weight update; under that definition the model cannot forget because there is no learned state to overwrite, but it also cannot repair a weak feature extractor, so the ceiling is whatever DINOv2's frozen features can separate.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Segment Any Class (SAC), a training-free method for multi-class few-shot semantic segmentation. SAC uses DINOv2 features extracted from support images to build class-representative feature arrays, computes per-pixel cosine similarity between query features and these arrays, applies Otsu thresholding and additional filtering to obtain Class Region Proposals, and then automatically generates positive/negative point prompts to drive the frozen SAM model. The authors evaluate SAC on COCO-20i for 1-way to 20-way settings and report that it outperforms gradient-trained baselines (DCAMA and Label Anything) in multi-way settings, with an especially large margin at 20-way, while remaining competitive in 1-way.
Significance. If the reported results are robust, SAC would be a valuable demonstration that prompt-only adaptation of vision foundation models can compete with or exceed gradient-based adaptation for multi-class few-shot segmentation, with advantages such as immunity to catastrophic forgetting and rapid online adaptation. The paper provides a useful conceptual contribution by extending single-class prompt-based methods like Matcher and PerSAM to multi-class settings, and it evaluates on a standard benchmark with 1000 episodes per fold. The ablation study in Table III gives some support for the importance of the proposed filtering components. However, the central quantitative claim currently rests on underspecified and potentially tuned thresholds, and the evaluation lacks statistical grounding, so the significance cannot be fully assessed from the manuscript as written.
major comments (4)
- [Section III-B and Section IV] The two thresholds that control inter-class overlap filtering and the rejection of masks based on average cosine similarity are never specified. The text says masks are removed if overlap 'exceeds a certain threshold' and if average cosine similarity is 'below a predefined threshold', but no values are given and no sensitivity analysis is provided. This is load-bearing because, for classes absent from a query image, Otsu thresholding will still partition the similarity distribution and produce spurious positive regions; whether those regions survive into SAM masks depends entirely on these thresholds. The reported 20-way margin over Label Anything (37.9 vs 13.7) could therefore be a consequence of threshold tuning rather than a general property of the prompt-only approach. The authors should report the exact threshold values and include an analysis showing that the high-N-way advantage is stable across a range of thresholds.
- [Abstract and Section V-C, Table II] The claim of 'superior results over state-of-the-art methods' is overstated relative to the paper's own data. Table I shows that DCAMA outperforms SAC in both 1-way 1-shot (50.9 vs 49.2) and 1-way 5-shot (58.3 vs 58.0), while Table II shows SAC ahead only starting at 2-way. The abstract and Section V-C should be reworded to state that SAC is competitive in 1-way and superior in multi-way settings, so that the central claim matches the evidence.
- [Section IV and Section V-C] The evaluation is limited to a single benchmark (COCO-20i) and no error bars, confidence intervals, or multiple-seed results are reported. Since the experiments use 1000 randomly selected episodes per fold, the differences between methods—especially the smaller gaps at 1-way and 2-way—could be within noise. To support a state-of-the-art claim, the authors should report variance statistics (e.g., standard deviation or confidence interval over episode draws) and ideally evaluate on at least one additional benchmark, such as PASCAL-5i, to demonstrate that the multiclass advantage generalizes.
- [Section III-B] The prompt generation procedure is described at a high level but lacks implementation details needed for reproduction. Specifically, the paper does not state how the positive-point grid is constructed, how many positive points are used per class, how the spatial clustering of positive points is performed (what clustering algorithm and what parameters), or how 'the average cosine similarity inside a mask' is computed. These details are essential for reproducing the method and for assessing whether the reported performance depends on specific heuristic choices beyond the thresholds mentioned above.
minor comments (5)
- [Section I and Section V-D] There are several typographical issues, including 's ´ans' in the Section V-D heading and 'Fig1' without a space in Section III. These should be corrected.
- [References] Reference [13] is malformed, containing repeated 'DINOv' text and an incomplete title; this should be fixed. Also, the paper cites 'MFNET' in Section IV without defining it or providing a reference, so the reader cannot tell what protocol is being followed.
- [Equation (2)] The notation in Equation (2) is unclear: the left-hand side defines R^c as a set indexed over support images, but the right-hand side uses F^c_i and M^c_i without clearly defining how these relate to the original image embeddings and labels. Clarifying the indexing would help readability.
- [Section III-B, Equation (4)] Equation (4) produces n_cluster similarity maps per class, but the text then says 'for each class, we assign each pixel to the class that has the highest cosine similarity score in the similarity map' without explaining how the per-cluster maps are aggregated into a single per-class similarity map. This should be specified.
- [Section IV] The paper states that background class is excluded from mIoU 'in alignment with the MFNET approach', but this choice is not further justified or compared with alternative evaluation protocols; a brief explanation or citation would help.
Circularity Check
No circularity found: SAC's predictions are derived from held-out query features plus support labels, not from the target labels or from a self-citation chain.
full rationale
The paper's derivation chain is self-contained and does not reduce to its inputs by construction. Support-set ground-truth labels enter only in Equation (2), where DINOv2 features are stratified by class masks to build Class-Representative Feature Arrays. Equation (3) clusters those stratified support features, and Equation (4) computes cosine similarity between the frozen query embedding and these CRFAs. The resulting class region proposals are filtered by Otsu thresholding, inter-class overlap filtering, and an average-cosine threshold, and then converted into point prompts for SAM. Query ground-truth labels appear only in the evaluation metric (mIoU), never in any equation that generates proposals, prompts, or masks. Thus there is no self-definitional equality between the claimed prediction and the inputs. The method performs no gradient training on the test classes, and the comparison to DCAMA and Label Anything is an empirical benchmark rather than a derivation forced by the method's assumptions. There are no load-bearing self-citations: references to DINOv2, SAM, Matching Networks, PerSAM, and Matcher are external prior works, and the authors do not invoke any of their own uniqueness theorems or prior results to forbid alternatives. Concerns about the unspecified overlap and average-cosine threshold values, the fixed n_cluster=5, and the transferability of DINOv2 features are correctness, robustness, and reproducibility risks, not circularity, because even if those hyperparameters were tuned on the benchmark, that would be overfitting or evaluation leakage rather than a logical equivalence between inputs and outputs. The conclusion that prompt-only adaptation can outperform gradient-learning methods is an empirical claim supported by the reported experiments, not a statement that is true by definition. Accordingly, no circular step can be exhibited, and the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (4)
- n_cluster =
5
- inter-class overlap threshold =
not disclosed
- average cosine similarity threshold =
not disclosed
- positive-point spatial clustering parameters =
not disclosed
assumptions (5)
- domain assumption DINOv2's frozen features transfer semantic class distinctions to query images not seen during support extraction.
- domain assumption Cosine similarity in DINOv2 embedding space is a valid measure of class-region membership.
- domain assumption SAM produces accurate masks when given point prompts inside a true class region.
- domain assumption Otsu thresholding is appropriate for filtering low-similarity pixels in cosine similarity maps.
- domain assumption The support images are representative of the class appearance distribution in the query images.
Cite this review
Pith. "Pith review of Segment Any Class (SAC): Multi-Class Few-Shot Semantic Segmentation via Class Region Proposals." pith.science (2026). https://pith.science/paper/TWKR27J3
@misc{pith2026241113774,
author = {Pith},
title = {Pith review of: Segment Any Class (SAC): Multi-Class Few-Shot Semantic Segmentation via Class Region Proposals},
year = {2026},
howpublished = {\url{https://pith.science/paper/TWKR27J3}},
note = {Machine review of arXiv:2411.13774}
}
read the original abstract
The Segment-Anything Model (SAM) is a vision foundation model for segmentation with a prompt-driven framework. SAM generates class-agnostic masks based on user-specified instance-referring prompts. However, adapting SAM for automated segmentation -- where manual input is absent -- of specific object classes often requires additional model training. We present Segment Any Class (SAC), a novel, training-free approach that task-adapts SAM for Multi-class segmentation. SAC generates Class-Region Proposals (CRP) on query images which allows us to automatically generate class-aware prompts on probable locations of class instances. CRPs are derived from elementary intra-class and inter-class feature distinctions without any additional training. Our method is versatile, accommodating any N-way K-shot configurations for the multi-class few-shot semantic segmentation (FSS) task. Unlike gradient-learning adaptation of generalist models which risk the loss of generalization and potentially suffer from catastrophic forgetting, SAC solely utilizes automated prompting and achieves superior results over state-of-the-art methods on the COCO-20i benchmark, particularly excelling in high N-way class scenarios. SAC is an interesting demonstration of a prompt-only approach to adapting foundation models for novel tasks with small, limited datasets without any modifications to the foundation model itself. This method offers interesting benefits such as intrinsic immunity to concept or feature loss and rapid, online task adaptation of foundation models.
Figures
Reference graph
Works this paper leans on
-
[13]
F. Li, Q. Jiang, H. Zhang, T. Ren, S. Liu, X. Zou, H. Xu, H. Li, C. Li, J. Yang, L. Zhang, J. Gao, and D. DINOv DINOv DINOv DINOv, “Visual In-Context Prompting Zero-shot Video Object and Part Segmentation Visual Prompting Referring Segmentation 8 Visual Prompting Generic Segmentation,” Tech. Rep., 2023. [Online]. Available: https://github.com/UX-Decoder/DINOv
work page 2023
-
[1]
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” 10
-
[2]
Language Models are Few-Shot Learners,
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Am...
arXiv 2020
-
[3]
LLaMA: Open and Efficient Foundation Language Models,
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “LLaMA: Open and Efficient Foundation Language Models,” 2 2023. [Online]. Available: http: //arxiv.org/abs/2302.13971
arXiv 2023
-
[4]
Masked Autoencoders Are Scalable Vision Learners,
K. He, X. Chen, S. Xie, Y . Li, P. Doll ´ar, and R. Girshick, “Masked Autoencoders Are Scalable Vision Learners,” 11 2021. [Online]. Available: http://arxiv.org/abs/2111.06377
arXiv 2021
-
[5]
DINOv2: Learning Robust Visual Features without Supervision,
M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y . Huang, S.-W. Li, I. Misra, M. Rabbat, V . Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski, “DINOv2: Learning Robust Visual Features without Supe...
arXiv 2023
-
[6]
Generative Adversarial Networks,
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio, “Generative Adversarial Networks,” 6 2014. [Online]. Available: http://arxiv.org/abs/1406.2661
arXiv 2014
-
[7]
Conditional Generative Adversarial Nets,
M. Mirza and S. Osindero, “Conditional Generative Adversarial Nets,” 11 2014. [Online]. Available: http://arxiv.org/abs/1411.1784
arXiv 2014
Show all 27 references
-
[8]
High-Resolution Image Synthesis with Latent Diffusion Models,
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” 12
-
[9]
Segment Anything,
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, P. Doll ´ar, and R. Girshick, “Segment Anything,” 4 2023. [Online]. Available: http://arxiv.org/abs/2304.02643
2023 arXiv
-
[10]
Segment Everything Everywhere All at Once,
X. Zou, J. Yang, H. Zhang, F. Li, L. Li, J. Wang, L. Wang, J. Gao, and Y . J. Lee, “Segment Everything Everywhere All at Once,” 4 2023. [Online]. Available: http://arxiv.org/abs/2304.06718
2023 arXiv
-
[11]
SAM 2: Segment Anything in Images and Videos,
N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ¨adle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V . Alwala, N. Carion, C.-Y . Wu, R. Girshick, P. Doll ´ar, and C. Feichtenhofer, “SAM 2: Segment Anything in Images and Videos,” 8 2024. [Online]. Ava...
2024 arXiv
-
[12]
SAM Fails to Segment Anything? – SAM-Adapter: Adapting SAM in Underperformed Scenes: Camouflage, Shadow, Medical Image Segmentation, and More,
T. Chen, L. Zhu, C. Ding, R. Cao, Y . Wang, Z. Li, L. Sun, P. Mao, and Y . Zang, “SAM Fails to Segment Anything? – SAM-Adapter: Adapting SAM in Underperformed Scenes: Camouflage, Shadow, Medical Image Segmentation, and More,” 4 2023. [Online]. Available: http://arxiv.org/abs/2...
2023 arXiv
-
[14]
VRP-SAM: SAM with Visual Reference Prompt,
Y . Sun, J. Chen, S. Zhang, X. Zhang, Q. Chen, G. Zhang, E. Ding, J. Wang, and Z. Li, “VRP-SAM: SAM with Visual Reference Prompt,” 2 2024. [Online]. Available: http://arxiv.org/abs/2402.17726
2024
-
[15]
Personalize Segment Anything Model with One Shot,
R. Zhang, Z. Jiang, Z. Guo, S. Yan, J. Pan, X. Ma, H. Dong, P. Gao, and H. Li, “Personalize Segment Anything Model with One Shot,” 5
-
[16]
Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching,
Y . Liu, M. Zhu, H. Li, H. Chen, X. Wang, and C. Shen, “Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching,” 5 2023. [Online]. Available: http://arxiv.org/abs/2305.13310
2023 arXiv
-
[17]
Exploring Effective Factors for Improving Visual In-Context Learning,
Y . Sun, Q. Chen, J. Wang, J. Wang, and Z. Li, “Exploring Effective Factors for Improving Visual In-Context Learning,” 4 2023. [Online]. Available: http://arxiv.org/abs/2304.04748
2023
-
[18]
What Makes Good Examples for Visual In-Context Learning?
Y . Zhang, K. Zhou, and Z. Liu, “What Makes Good Examples for Visual In-Context Learning?” 1 2023. [Online]. Available: http://arxiv.org/abs/2301.13670
2023 arXiv
-
[19]
Dense Cross-Query-and-Support Attention Weighted Mask Aggregation for Few-Shot Segmentation,
X. Shi, D. Wei, Y . Zhang, D. Lu, M. Ning, J. Chen, K. Ma, and Y . Zheng, “Dense Cross-Query-and-Support Attention Weighted Mask Aggregation for Few-Shot Segmentation,” 7 2022. [Online]. Available: http://arxiv.org/abs/2207.08549
2022 arXiv
-
[20]
Label Anything: Multi-Class Few-Shot Semantic Segmentation with Visual Prompts,
P. De Marinis, N. Fanelli, R. Scaringi, E. Colonna, G. Fiameni, G. Vessio, and G. Castellano, “Label Anything: Multi-Class Few-Shot Semantic Segmentation with Visual Prompts,” 7 2024. [Online]. Available: http://arxiv.org/abs/2407.02075
2024 arXiv
-
[21]
A Novel Benchmark for Few-Shot Semantic Segmentation in the Era of Foundation Models,
R. Bensaid, V . Gripon, F. Leduc-Primeau, L. Mauch, G. B. Hacene, and F. Cardinaux, “A Novel Benchmark for Few-Shot Semantic Segmentation in the Era of Foundation Models,” Tech. Rep. [Online]. Available: https://github.com/RedaBensaidDS/Foundation FewShot
-
[22]
SegGPT: Segmenting Everything In Context,
X. Wang, X. Zhang, Y . Cao, W. Wang, C. Shen, and T. Huang, “SegGPT: Segmenting Everything In Context,” 4 2023. [Online]. Available: http://arxiv.org/abs/2304.03284
2023 arXiv
-
[23]
Matching Networks for One Shot Learning,
O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and D. Wierstra, “Matching Networks for One Shot Learning,” 6 2016. [Online]. Available: http://arxiv.org/abs/1606.04080
2016 arXiv
-
[24]
Panoptic Segmentation,
A. Kirillov, K. He, R. Girshick, C. Rother, and P. Doll ´ar, “Panoptic Segmentation,” 1 2018. [Online]. Available: http://arxiv.org/abs/1801. 00868
2018
-
[2018]
Available: http://arxiv.org/abs/1810.04805
[Online]. Available: http://arxiv.org/abs/1810.04805
-
[2021]
Available: http://arxiv.org/abs/2112.10752
[Online]. Available: http://arxiv.org/abs/2112.10752
-
[2023]
Available: http://arxiv.org/abs/2305.03048
[Online]. Available: http://arxiv.org/abs/2305.03048
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.