{"id":"38b904a3-aff8-4d6c-a2f3-95222cf45515","arxiv_id":"2411.13774","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A training-free prompting method that adapts SAM to multi-class few-shot segmentation and reports higher mIoU than trained baselines on COCO-20i as the number of classes grows.","lead":"Segment Any Class (SAC) automatically adapts SAM, a foundation model for image segmentation, to segment several object classes at once using only a few labeled examples and no model training. It reports accuracy gains over trained methods on the COCO-20i benchmark, especially when many classes must be segmented at once.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"High-N-way superiority rests on unstated overlap and average-cosine thresholds; without their values or a sensitivity analysis, the reported 20-way gain over LA could be tuning rather than a property of the prompt-only method.","rationale":"The reader's conditional verdict is well justified. The strongest claim is empirical: a frozen SAM with automatically generated point prompts outperforms gradient-trained multi-class FSS models, especially at high N-way. I considered whether the main risk is instead that DINOv2 features might not transfer, but DINOv2 is trained on broad data and the paper's ablation shows it is dominant; feature quality affects all methods relying on such backbones. The more specific and load-bearing gap is the false-positive suppression mechanism for absent classes, which is exactly what determines multi-way performance. When a class is absent, the similarity map to that class's CRFAs is likely low-valued and possibly unimodal; Otsu thresholding will still split it, generating candidate positive points. Whether those candidates produce final masks depends on the two thresholds that are never reported: the mask-overlap threshold in inter-class filtering and the average-cosine threshold. Without these values, the reported 20-way result cannot be independently reproduced, and it is impossible to tell whether the gap over LA arises from a robust property of the method or from a narrow threshold setting. This is consistent with the reader's identification of the unspecified thresholds as part of the weakest assumption. A sensitivity sweep directly tests the brittleness: if the reported numbers sit on a plateau, the conditionality can be relaxed; if they are isolated peaks, the central claim should be treated as unverified. Since the paper does not provide code or threshold values, the CONDITIONAL verdict remains appropriate; my concern does not move the verdict but strengthens the reasons for it.","tokens_in":9728,"tokens_out":13668,"duration_ms":159230,"concrete_test":"Run a sensitivity sweep on COCO-20i fold-0 for 5-way and 20-way 1-shot: vary the overlap threshold over a grid such as [0.1, 0.9] and the average-cosine threshold over a range covering plausible values, holding n_cluster=5 and the DINOv2-L and SAM-H backbones fixed. Plot mIoU as a heat map. If the Table II numbers sit in a narrow high-peak region (e.g., mIoU drops by more than 5 points within ±0.05 of the chosen thresholds), the central claim is brittle; if a wide plateau exists, the missing threshold values are less concerning. Additionally, verify whether a single fixed threshold pair across all folds and all N-way settings reproduces the reported means, as the 'any N-way K-shot' claim requires.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, that SAC outperforms gradient-trained multi-class FSS methods and especially excels at high N-way settings, depends on suppressing false positives for classes absent from a query image. Section III-B describes only two mechanisms for this: inter-class filtering with an unspecified overlap threshold, and discarding masks whose average cosine similarity falls below an unspecified predefined threshold. Neither value is reported, and no sensitivity analysis is provided. For an absent class, Otsu thresholding of the per-class similarity map has no principled way to know the class is absent; it will still partition the similarity distribution and select spurious positive locations. Whether those spurious proposals survive into SAM masks depends entirely on the two unspecified thresholds. The reported 20-way mIoU of 37.9 versus LA's 13.7 could therefore reflect threshold values tuned to suppress false positives rather than a general property of the prompt-only approach. Because the paper claims the method is training-free and works for any N-way K-shot configuration, the absence of these parameter values and of any robustness analysis is a load-bearing gap, not a cosmetic one.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Segment Any Class (SAC), a training-free method for multi-class few-shot semantic segmentation. SAC uses DINOv2 features extracted from support images to build class-representative feature arrays, computes per-pixel cosine similarity between query features and these arrays, applies Otsu thresholding and additional filtering to obtain Class Region Proposals, and then automatically generates positive/negative point prompts to drive the frozen SAM model. The authors evaluate SAC on COCO-20i for 1-way to 20-way settings and report that it outperforms gradient-trained baselines (DCAMA and Label Anything) in multi-way settings, with an especially large margin at 20-way, while remaining competitive in 1-way.","tokens_in":9918,"tokens_out":3270,"duration_ms":33920,"significance":"If the reported results are robust, SAC would be a valuable demonstration that prompt-only adaptation of vision foundation models can compete with or exceed gradient-based adaptation for multi-class few-shot segmentation, with advantages such as immunity to catastrophic forgetting and rapid online adaptation. The paper provides a useful conceptual contribution by extending single-class prompt-based methods like Matcher and PerSAM to multi-class settings, and it evaluates on a standard benchmark with 1000 episodes per fold. The ablation study in Table III gives some support for the importance of the proposed filtering components. However, the central quantitative claim currently rests on underspecified and potentially tuned thresholds, and the evaluation lacks statistical grounding, so the significance cannot be fully assessed from the manuscript as written.","major_comments":[{"comment":"The two thresholds that control inter-class overlap filtering and the rejection of masks based on average cosine similarity are never specified. The text says masks are removed if overlap 'exceeds a certain threshold' and if average cosine similarity is 'below a predefined threshold', but no values are given and no sensitivity analysis is provided. This is load-bearing because, for classes absent from a query image, Otsu thresholding will still partition the similarity distribution and produce spurious positive regions; whether those regions survive into SAM masks depends entirely on these thresholds. The reported 20-way margin over Label Anything (37.9 vs 13.7) could therefore be a consequence of threshold tuning rather than a general property of the prompt-only approach. The authors should report the exact threshold values and include an analysis showing that the high-N-way advantage is stable across a range of thresholds.","section":"Section III-B and Section IV"},{"comment":"The claim of 'superior results over state-of-the-art methods' is overstated relative to the paper's own data. Table I shows that DCAMA outperforms SAC in both 1-way 1-shot (50.9 vs 49.2) and 1-way 5-shot (58.3 vs 58.0), while Table II shows SAC ahead only starting at 2-way. The abstract and Section V-C should be reworded to state that SAC is competitive in 1-way and superior in multi-way settings, so that the central claim matches the evidence.","section":"Abstract and Section V-C, Table II"},{"comment":"The evaluation is limited to a single benchmark (COCO-20i) and no error bars, confidence intervals, or multiple-seed results are reported. Since the experiments use 1000 randomly selected episodes per fold, the differences between methods—especially the smaller gaps at 1-way and 2-way—could be within noise. To support a state-of-the-art claim, the authors should report variance statistics (e.g., standard deviation or confidence interval over episode draws) and ideally evaluate on at least one additional benchmark, such as PASCAL-5i, to demonstrate that the multiclass advantage generalizes.","section":"Section IV and Section V-C"},{"comment":"The prompt generation procedure is described at a high level but lacks implementation details needed for reproduction. Specifically, the paper does not state how the positive-point grid is constructed, how many positive points are used per class, how the spatial clustering of positive points is performed (what clustering algorithm and what parameters), or how 'the average cosine similarity inside a mask' is computed. These details are essential for reproducing the method and for assessing whether the reported performance depends on specific heuristic choices beyond the thresholds mentioned above.","section":"Section III-B"}],"minor_comments":[{"comment":"There are several typographical issues, including 's ´ans' in the Section V-D heading and 'Fig1' without a space in Section III. These should be corrected.","section":"Section I and Section V-D"},{"comment":"Reference [13] is malformed, containing repeated 'DINOv' text and an incomplete title; this should be fixed. Also, the paper cites 'MFNET' in Section IV without defining it or providing a reference, so the reader cannot tell what protocol is being followed.","section":"References"},{"comment":"The notation in Equation (2) is unclear: the left-hand side defines R^c as a set indexed over support images, but the right-hand side uses F^c_i and M^c_i without clearly defining how these relate to the original image embeddings and labels. Clarifying the indexing would help readability.","section":"Equation (2)"},{"comment":"Equation (4) produces n_cluster similarity maps per class, but the text then says 'for each class, we assign each pixel to the class that has the highest cosine similarity score in the similarity map' without explaining how the per-cluster maps are aggregated into a single per-class similarity map. This should be specified.","section":"Section III-B, Equation (4)"},{"comment":"The paper states that background class is excluded from mIoU 'in alignment with the MFNET approach', but this choice is not further justified or compared with alternative evaluation protocols; a brief explanation or citation would help.","section":"Section IV"}],"recommendation":"major_revision","confidential_remarks":"The central idea is interesting and the reported multi-way results are promising, but the missing threshold values and lack of statistical evaluation are exactly the kind of load-bearing gaps that should be fixed before publication. I would like to see the authors either provide the threshold values and a sensitivity analysis, or soften the state-of-the-art claim accordingly. The paper's scope is appropriate for a venue interested in foundation-model adaptation, but as written the quantitative claims are not yet fully supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick read on arXiv 2411.13774 (SAC). The paper does something genuinely new: it takes the PerSAM/Matcher prompt-only idea and pushes it to multi-class N-way FSS, using DINOv2 prototypes per class, a background prototype, and two filtering stages. On COCO-20i it reports mIoU of 49.2/46.6/44.0/37.9 for 1/2/5/20-way, against DCAMA's 50.9/31.7/16.7/4.7 and LA's 43.1/34.6/27.7/13.7. That graceful degradation is the paper's best evidence, and it is a real result if the numbers hold.\n\nThe soft spots are real and one is load-bearing. The two filtering mechanisms — inter-class overlap threshold and average-cosine threshold — are never given values, and the ablation shows inter-class filtering contributes about 6.5 mIoU at 1-way and 8.3 at 5-way. Without those values or a sensitivity analysis, the high-N-way gains could be a tuned artifact. The stress-test note is right: for a class absent from the query, Otsu will still split the similarity distribution, and the only thing preventing false positives is those two thresholds. The paper compensates with a clean experimental setup (1000 episodes per fold, held-out validation, no training on test classes), but there are only two trained baselines, no error bars, and no code. The 'intrinsic immunity to forgetting' phrase is overstatement — prompt-only means no learned weights to forget, but the prompt generation itself can still be miscalibrated to new classes.\n\nAlso, the comparison set is narrow; missing VRP-SAM and SegGPT in the multi-class table (they are quoted for 1-way in the text but not in Table II). Minor: reference [13] has a corrupted author list.\n\nIs it a serious thinker? Yes. The method is coherent, the write-up is honest about dependence on DINOv2 quality, and the ablation is informative. It deserves a referee, but the referee should demand the threshold values, a sensitivity study over them, and ideally at least a second benchmark or error bars. This is a good workshop-grade paper with the potential to be a solid conference paper after that work.\n\nMy call: send to peer review, but expect major revision.","headline":"Plausible training-free multi-class FSS pipeline that beats trained baselines on COCO-20i, but the unspecified filtering thresholds make the headline numbers hard to trust.","tokens_in":10443,"tokens_out":2120,"would_cite":true,"duration_ms":19951,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free pipeline of frozen DINOv2 features and SAM point prompts claims to outperform gradient-trained multi-class few-shot segmenters on COCO-20i, with the gap widening as class count grows.","keywords":["Segment Anything Model","multi-class few-shot semantic segmentation","in-context learning","foundation model adaptation","training-free prompting","class region proposals","cosine similarity","COCO-20i"],"falsifier":"Run the full pipeline on COCO-20i with the support set's class masks replaced by random masks of the same shape. If mIoU stays far above the chance level, the class signal is not coming from the support annotations and the Class Region Proposal geometry is doing no work; if the score collapses, the mechanism depends on the claimed feature alignment.","tokens_in":9514,"feed_emoji":"🎯","tokens_out":6763,"duration_ms":62963,"temperature":0.7,"pith_summary":"This paper sets out to show that a vision foundation model can be adapted to multi-class few-shot semantic segmentation without updating any weights. Instead of training a mask decoder or prompt encoder, the method, Segment Any Class (SAC), uses a frozen feature extractor (DINOv2) to build per-class prototype clusters from support images, computes per-pixel cosine similarity on the query image, and converts the resulting Class Region Proposals into positive and negative point prompts for the frozen segmentation model SAM. The authors report that this prompt-only pipeline beats the gradient-trained baselines DCAMA and Label Anything on the COCO-20i benchmark, and that the gap widens as the number of classes grows (e.g., 37.9 mIoU for SAC versus 13.7 for Label Anything and 4.7 for DCAMA at 20-way 1-shot). If the claim holds, task adaptation of foundation models for new classes need not rewrite weights, which would make adaptation immune to catastrophic forgetting and fast enough for online use.","feed_headline":"Prompt-only method beats trained few-shot segmenters","feed_subtitle":"Frozen DINOv2 features become SAM point prompts, keeping 37.9 mIoU at 20-way where trained baselines fall to single digits.","key_machinery":"The load-bearing machinery is the Class Region Proposal (CRP), built from Class-Representative Feature Arrays (CRFAs). Support images pass through frozen DINOv2; pixels of each class (plus a background class) are k-means clustered into $n_{\\text{cluster}}=5$ prototype vectors; cosine similarity between the query embedding and each prototype yields per-pixel score maps; Otsu thresholding keeps only high-similarity pixels; inter-class overlap filtering and average-cosine filtering discard conflicting or low-confidence masks. These region proposals become SAM point prompts, with positive points from clustered proposal locations paired with negative points from the background proposal, so the frozen SAM mask decoder does the segmentation.","core_discovery":"SAC's central claim is that elementary feature geometry in a frozen self-supervised vision model is sufficient to adapt a class-agnostic segmenter to a new multi-class segmentation task. The method clusters support pixels per class (and background) into Class-Representative Feature Arrays, computes cosine-similarity maps between query embeddings and each array, thresholds these maps with Otsu, and uses the surviving regions as class region proposals. Positive point prompts are drawn from each proposal, negative prompts from a background region proposal, and spatially clustered to create prompt sets for SAM; overlapping low-confidence masks are filtered. On COCO-20i, SAC reports 1-way 1-shot mIoU of 49.2, within about two points of DCAMA's 50.9, while at 5-way it reaches 44.0 against DCAMA's 16.7 and Label Anything's 27.7, and at 20-way 37.9 against 4.7 and 13.7 respectively, with no weight updates.","pith_inferences":["A testable consequence the paper does not pursue: because the proposal generator is prompt-agnostic, the same CRFAs should drive newer point-promptable segmenters such as SAM 2, which would isolate where the gain comes from.","The fixed hyperparameters ($n_{\\text{cluster}}=5$, Otsu thresholding, overlap and confidence cutoffs) are calibrated on COCO-20i; on a benchmark with dissimilar class scales, the same defaults may miscalibrate, so measuring per-class accuracy by object size would reveal the boundary of the claim.","The paper implicitly redefines adaptation as prompt selection rather than weight update; under that definition the model cannot forget because there is no learned state to overwrite, but it also cannot repair a weak feature extractor, so the ceiling is whatever DINOv2's frozen features can separate."],"forward_implications":["Adding a new class to SAC requires only adding its support images and re-running the clustering; no retraining or weight update is needed, so previously learned classes cannot be overwritten.","Segmentation accuracy degrades far more slowly as the number of classes grows: SAC drops from 49.2 mIoU at 1-way to 37.9 at 20-way (1-shot), whereas DCAMA falls from 50.9 to 4.7 and Label Anything from 43.1 to 13.7.","Because the pipeline is prompt-only, the same frozen SAM and DINOv2 weights can be swapped between tasks online, without storing per-task model copies.","The ablation attributes most of the gain to the feature extractor: replacing DINOv2 with SAM's image encoder drops 1-way 1-shot mIoU from 45.5 to 26.6, indicating that the frozen features, not the mask decoder, carry the adaptation.","The method accommodates any N-way K-shot configuration, so the same machinery applies to one-shot, five-shot, or any other support-set size without changing the algorithm."],"supporting_citations":[{"why":"Supplies the frozen class-agnostic mask generator that turns the automatically generated point prompts into binary masks.","marker":"[9]"},{"why":"Supplies the frozen feature extractor whose embeddings are clustered into Class-Representative Feature Arrays and matched to query pixels by cosine similarity.","marker":"[5]"},{"why":"Provides the strongest gradient-trained multi-class few-shot segmentation baseline that SAC claims to surpass at high N-way.","marker":"[19]"},{"why":"Provides the other gradient-trained multi-class baseline with a trainable prompt encoder, used as the Label Anything comparison.","marker":"[20]"},{"why":"Contributes the single-class prompt-only SAM adaptation whose spatial clustering of positive points SAC extends to multi-class prompts.","marker":"[16]"},{"why":"Establishes the prompt-only one-shot SAM adaptation paradigm that SAC generalizes from one class to many classes.","marker":"[15]"},{"why":"Provides the cosine-similarity matching of query and support features that anchors SAC's affinity-based class region proposal mechanism.","marker":"[23]"}],"fun_headline_variants":["Training-free SAC beats fine-tuned few-shot segmenters","Zero-gradient SAC wins few-shot segmentation","Frozen features + SAM prompts outdo fine-tuned models","No training, just prompts: SAC adapts SAM for new classes","Class-region prompts make SAM a few-shot learner"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument rests on the premise that DINOv2's frozen embeddings place pixels of the same class close together in cosine space on unseen query images, so that the cluster prototypes and the fixed Otsu and overlap thresholds yield point prompts that land on true class instances; if the features do not transfer, or the thresholds are miscalibrated, every downstream mask inherits the error.","fun_headline_variants_meta":{"raw":{"variants":["Training-free SAC beats fine-tuned few-shot segmenters","Zero-gradient SAC wins few-shot segmentation","Frozen features + SAM prompts outdo fine-tuned models","No training, just prompts: SAC adapts SAM for new classes","Class-region prompts make SAM a few-shot learner"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000688,"raw_usage":{"total_tokens":3150,"prompt_tokens":1010,"completion_tokens":2140,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":2061}},"tokens_in":626,"tokens_out":2140,"duration_ms":21510,"temperature":1.0,"reasoning_tokens":2061,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:53:58.783774+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full pipeline on COCO-20i with the support set's class masks replaced by random masks of the same shape. If mIoU stays far above the chance level, the class signal is not coming from the support annotations and the Class Region Proposal geometry is doing no work; if the score collapses, the mechanism depends on the claimed feature alignment.","supporting_citations":[{"cited_title":"Dense Cross-Query-and-Support Attention Weighted Mask Aggregation for Few-Shot Segmentation","cited_arxiv_id":"2207.08549","evidence_quote":"Provides the strongest gradient-trained multi-class few-shot segmentation baseline that SAC claims to surpass at high N-way."},{"cited_title":"Personalize Segment Anything Model with One Shot,","cited_arxiv_id":null,"evidence_quote":"Establishes the prompt-only one-shot SAM adaptation paradigm that SAC generalizes from one class to many classes."}],"review_version":1}