{"id":"7f51f348-2f25-4a29-9ddc-b783d39c9cec","arxiv_id":"2511.20027","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SAM-MI improves open-vocabulary segmentation by injecting aggregated SAM masks as low- and high-frequency guidance into CLIP cost maps, with sparse text-guided point prompts for speed.","lead":"An image-segmentation framework that feeds masks from the Segment Anything Model into CLIP's open-vocabulary cost maps, while using sparse text-guided points to cut SAM's prompt cost by 96%. It reports consistent but small accuracy gains on standard benchmarks and a 16.7% MESS gain over Grounded-SAM, though that comparison is not training-matched.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 gains over CAT-Seg may not be statistically robust: improvements are 0.5–1.1 mIoU with no error bars, and the w/o DMI ablation exactly matches published CAT-Seg numbers, suggesting the baseline was not retrained under identical conditions.","rationale":"The reader's verdict is CONDITIONAL, and I agree that the paper needs additional verification. However, the reader identified the MESS training mismatch and TSPP transfer as the weakest assumptions; my review points to a more fundamental issue with the central Table 2 claim: the small effect size and the lack of a properly controlled baseline. The exact match between the 'w/o DMI' ablation and the published CAT-Seg numbers suggests the baseline was not retrained, which undermines the controlled comparison. This is a concrete, technical concern that can be settled by a multi-seed retraining experiment. If the gains disappear under controlled conditions, the paper's core contribution is not established; if they persist, the concern is resolved. The verdict remains CONDITIONAL because the current evidence is not airtight, but the paper's method and ablations are otherwise coherent and the qualitative analyses are useful.","tokens_in":19500,"tokens_out":11268,"duration_ms":118231,"concrete_test":"Retrain CAT-Seg ViT-B/16 and SAM-MI ViT-B/16 from scratch in the same codebase, with the same hardware, data pipeline, and training recipe (80K iterations, batch size 4, resolution 384×384 for CLIP), using at least 3 random seeds. Report mean and standard deviation of mIoU on A-847, PC-459, A-150, PC-59, and PAS-20. If the mIoU gain of SAM-MI over CAT-Seg is less than one standard deviation or is not consistently positive across seeds, the central claim is unsupported. As a secondary check, verify that the 'w/o DMI' row in Table 3 can actually be reproduced from an independent run under these settings; if it cannot, the ablation is not a controlled comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that injecting aggregated SAM masks into CLIP cost maps improves open-vocabulary segmentation over the same backbone without SAM. Table 2 shows SAM-MI exceeding CAT-Seg by only +0.5/+0.8/+1.1 mIoU on A-847/PC-459/A-150 (ViT-B/16), and +0.5/+0.3 on PC-59/PAS-20. These absolute gains are small, and the paper reports no multiple seeds, standard deviations, or significance tests, so the improvement could be within run-to-run variance.\n\nMore concerning, Table 3's 'w/o DMI' row is numerically identical to CAT-Seg's published results for all five benchmarks (12.0/19.0/31.8/57.5/94.6). Such an exact match strongly implies the authors did not re-run a w/o DMI baseline in their own environment but copied the CAT-Seg numbers from the original paper. While using published baselines is common practice, the comparison is then not controlled for training code, hardware, data preprocessing, or random seed. Given the small effect size, a properly controlled retraining of the baseline could plausibly erase the claimed advantage. The evaluation on MESS suffers from a similar baseline mismatch (Grounded-SAM is zero-shot while SAM-MI is COCO-Stuff trained), but the Table 2 claim is the most load-bearing and is vulnerable to this statistical and control concern.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"SAM-MI is a framework for open-vocabulary semantic segmentation that combines CLIP-based cost maps with frozen SAM masks. Three components are proposed: TSPP learns sparse text-guided point prompts for SAM, reducing the number of prompts by ~96% compared to a dense 32x32 grid; SMAgg aggregates over-segmented SAM masks using coarse text-mask overlap; and DMI injects the aggregated masks into low-frequency and high-frequency stages of a CAT-Seg-style cost aggregation network. The method is trained on COCO-Stuff and evaluated on A-847, PC-459, A-150, PC-59, PAS-20, and the MESS benchmark. The paper reports gains over CAT-Seg on the five standard benchmarks (e.g., +1.1 mIoU on A-150) and a 16.7% relative improvement over Grounded-SAM on MESS, with a 1.6x speedup.","tokens_in":19990,"tokens_out":10058,"duration_ms":89837,"significance":"If the empirical claims are reliable, the mask-injected paradigm is a useful and general alternative to mask-driven and prompt-driven ways of coupling SAM with CLIP-based OVSS: using SAM masks as soft guidance inside cost-map aggregation is conceptually clean and could transfer to other cost-map architectures. TSPP's sparse prompting also gives a very large latency reduction with little accuracy loss, which is practically valuable. The paper is clearly written and the ablations are organized in a way that, in principle, isolates each module. However, the current evaluation does not establish the central claim. The gains over the same backbone are small (0.5-1.1 mIoU absolute) with no error bars; the w/o DMI ablation row exactly reproduces published CAT-Seg numbers, suggesting the baseline was not retrained under the same conditions; and the headline MESS comparison is confounded because SAM-MI is COCO-Stuff-trained while Grounded-SAM is used zero-shot. A controlled rerun plus uncertainty quantification could turn this into a solid contribution, but as it stands the evidence is not yet convincing.","major_comments":[{"comment":"The 'w/o DMI' row reports exactly the same five numbers (12.0/19.0/31.8/57.5/94.6) as the CAT-Seg ViT-B/16 row in Table 2. Removing DMI should leave a CAT-Seg-like model, so the numbers should be close, but an exact match across all five benchmarks strongly suggests that the authors copied the published CAT-Seg results rather than running this ablation in their own pipeline. This matters because the paper's central claim — that mask injection improves OVSS — rests on the difference between SAM-MI and this row. Please rerun the w/o DMI configuration under the same training code, data, schedule, and hardware, and report mean ± std over multiple seeds. If the run was already performed, please provide logs or other evidence.","section":"§4.3, Tables 3 and 4"},{"comment":"The improvements over CAT-Seg are small: +0.5 mIoU on A-847, +0.8 on PC-459, +1.1 on A-150, +0.9 on PC-59, and +0.6 on PAS-20. No variance, confidence intervals, or multiple-seed results are reported. Given that the w/o DMI ablation appears to reproduce published CAT-Seg numbers exactly, it is uncertain whether the differences are due to the proposed modules or to run-to-run variation. The paper should include at least three seeds for SAM-MI and the controlled baseline, and ideally a paired significance test. The Limitations section itself notes that the benchmarks contain incomplete/incorrect annotations, which further underscores the need for statistical robustness when claiming small gains.","section":"§4.2, Table 2"},{"comment":"The headline MESS claim — a 16.7% relative mIoU improvement over Grounded-SAM — is not an apples-to-apples comparison. SAM-MI is trained on COCO-Stuff, whereas Grounded-SAM is an off-the-shelf zero-shot pipeline (Grounding DINO + SAM) with no COCO-Stuff training. The training-data advantage alone could explain much of the difference. To support the claimed cross-domain benefit of the mask-injected framework, the MESS table should include a COCO-Stuff-trained baseline using the same CLIP backbone (e.g., CAT-Seg or the SAM-MI model without DMI/SMAgg). The abstract and conclusion should be reworded so that the comparison is not presented as an architectural improvement.","section":"Abstract and §4.2, Table 1"}],"minor_comments":[{"comment":"Typos: 'Renmin Univeristy' in the affiliations; 'even even more significant' in §4.3; 'xFurthermore' before Table 10; 'denotes as M' in Eq. (8). Please proofread.","section":"General"},{"comment":"The text states that in the earth-monitoring domain Grounded-SAM reaches 25.97 mIoU and SAM-MI reaches 37.66, a 45.0% relative improvement. These averages do not match the values in Table 1 for either the five or six datasets in that domain. Please recompute or correct the text.","section":"§4.2, MESS discussion"},{"comment":"The latency for the uniform-grid configuration is inconsistent: Table 3 reports 5458 ms for w/o TSPP, while Table 5 reports 5967 ms for the same 32x32 uniform grid on A-150. Clarify the exact dataset, image size, hardware, and measurement setup for each latency number.","section":"Tables 3 and 5"},{"comment":"The probability-target formula is typeset ambiguously: it is not clear whether P_k is in the numerator or denominator. Please write the expression explicitly and define all symbols (e.g., d is the distance transform, M_gt^k is the ground-truth mask).","section":"Eq. (3)"},{"comment":"Please state whether reported numbers are the best over runs or the mean, and report standard deviations where feasible. Also, since no code is provided, a reproducibility statement or plan to release code would strengthen the paper.","section":"§4.2 and Tables"}],"recommendation":"major_revision","confidential_remarks":"Editor: the exact identity of the w/o DMI ablation with published CAT-Seg numbers is a red flag that should be checked before this paper proceeds further. I recommend asking the authors to provide training logs or code to confirm that the ablation was actually run. The small effect sizes and the confounded MESS comparison make the current manuscript unsuitable for acceptance without a controlled re-evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SAM-MI is a solid engineering paper for coupling SAM with CLIP-based segmentation. The sparse-prompting trick works, the modular ablations are coherent, and the latency numbers look credible. But the headline claims overstate the story: the w/o DMI ablation appears to be copied from the CAT-Seg paper, and the MESS comparison pits a COCO-Stuff-trained model against a zero-shot one. Once those are accounted for, the method buys you roughly 0.5–1.1 mIoU on some standard benchmarks plus a real speedup — still worth having, but not the 16.7% story.\n\nWhat is actually new: TSPP learns text-conditioned sampling probabilities to prompt SAM, cutting point count by 96% relative to grid sampling while keeping mIoU about the same, and giving roughly a 6x speedup on the SAM decoding path. That is a practical contribution with consistent ablation support in Tables 5–7. SMAgg and DMI are reasonable extensions of the cost-aggregation line (CAT-Seg); the low/high-frequency split is a sensible way to inject mask cues without hard fusion, and the component-wise ablations are internally consistent.\n\nWhere it is soft: the w/o DMI row in Table 3 exactly matches the published CAT-Seg numbers on all five benchmarks. That strongly suggests they did not retrain a same-code baseline, so the gains over CAT-Seg in Table 2 are not controlled for training pipeline, data preprocessing, or random seed. The absolute gains are also small enough that run-to-run variance could matter, and no error bars or multiple seeds are reported. The MESS comparison is similar in spirit: Grounded-SAM is zero-shot, while SAM-MI is trained on COCO-Stuff. The 16.7% relative improvement is therefore not evidence that mask-injected prompting beats prompt-driven SAM; it is evidence that training on the target distribution helps. That claim needs a training-matched baseline before it means anything.\n\nTo be fair, the paper is honest about its limitations, and the failure-case and SAM-variant analyses are useful. If I were working on SAM-based OVSS, I would cite the TSPP speedup; I would not cite the MESS superiority claim.\n\nIt deserves a serious referee — the engineering is clean and the evaluation questions are checkable — but a good review should insist on a retrained CAT-Seg baseline, error bars, a training-matched MESS comparison, and code release. As submitted, the paper is a useful technical report, not a demonstrated state-of-the-art result.","headline":"Useful SAM-CLIP integration with a real speedup, but the headline gains shrink once you notice the baselines are borrowed and the MESS comparison is trained-vs-zero-shot.","tokens_in":20329,"tokens_out":2509,"would_cite":true,"duration_ms":27132,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SAM-MI claims open-vocabulary segmentation improves when SAM's masks are injected into CLIP's pixel-text cost maps at separate low and high frequencies, rather than being hard-combined with labels.","keywords":["open-vocabulary semantic segmentation","Segment Anything Model","mask injection","cost maps","sparse point prompts","CLIP","Decoupled Mask Injection","Shallow Mask Aggregation"],"falsifier":"A controlled experiment on a novel-category benchmark (e.g., classes not in COCO-Stuff, with no mask-skeleton supervision for TSPP) comparing SAM-MI against its own SAM-less backbone: if the gap disappears or reverses when TSPP is replaced by uniform grid sampling at the same point budget, the claim that sparse text-guided prompting preserves accuracy fails. Also, retraining Grounded-SAM on COCO-Stuff and re-running MESS would test whether the 16.7% lead survives matched training.","tokens_in":19466,"feed_emoji":"🎯","tokens_out":6340,"duration_ms":56270,"temperature":0.7,"pith_summary":"The paper tries to establish a new way to couple the Segment Anything Model (SAM) with CLIP-based open-vocabulary segmentation: instead of using SAM's masks as fixed region proposals that are then labeled (mask-driven) or prompting SAM with per-class boxes (prompt-driven), SAM-MI uses SAM's masks as soft guidance injected into the pixel-text cost maps that CLIP already produces. The injection is decoupled into low-frequency (global, mask-interior context) and high-frequency (edge and detail) components, which the authors argue fixes two failure modes: SAM's over-segmentation and the rigid, error-prone combination of fixed masks with labels. The support is a set of mIoU gains — 4.2%, 4.2%, and 3.5% relative over the same SAM-less backbone on ADE20K-847, PC-459, and ADE20K-150 with a ViT-B/16 CLIP, and 16.7% relative over Grounded-SAM on the cross-domain MESS benchmark — plus a 1.6x inference speedup from a text-guided sparse point prompter that reduces SAM's prompt count by 96%. If true, this gives practitioners a cheap, general recipe for adding SAM's universal segmentation ability to any cost-map-based OVSS model.","feed_headline":"Inject SAM masks into CLIP cost maps to gain 16.7% over Grounded-SAM","feed_subtitle":"A sparse prompt sampler cuts SAM's point count by 96%, and decoupled mask injection fixes over-segmentation.","key_machinery":"Three modules carry the argument. Decoupled Mask Injection (DMI) splits the guidance: low-frequency injection uses mask-pooling and cross-attention to model intra-mask and inter-mask context in the coarse cost aggregation stage; high-frequency injection concatenates mask embeddings with shallow CLIP features and passes them through a depthwise convolution plus MLP to sharpen edges in the upsampling decoder. Shallow Mask Aggregation (SMAgg) merges SAM's over-segmented proposals using text-guided masks (from the TSPP head) and a matching-score threshold (0.5), reducing mask redundancy. Text-guided Sparse Point Prompter (TSPP) learns text-specific sampling probabilities from cost maps, supervis","core_discovery":"The paper's central discovery is that decoupled mask injection — adding SAM's masks as low-frequency context and high-frequency detail to CLIP's cost maps — is a better way to equip open-vocabulary segmentation with SAM than labeling fixed masks or prompting SAM with boxes. This turns SAM's universal masks from competitive region proposals into corrective guidance, which is robust to SAM's over-segmentation and low-quality masks. The ablations confirm each component matters: removing the high-frequency injection costs about 0.3 mIoU, removing the low-frequency injection costs more, and removing the aggregation step costs 2.2 mIoU on ADE20K-150. The sparse prompter matches dense grid sampling","pith_inferences":["The decoupling principle likely transfers beyond semantic segmentation: the same low/high-frequency injection could improve open-vocabulary panoptic or instance segmentation, where the boundary-detail channel matters even more.","TSPP's reliance on skeleton-derived targets trained on COCO-Stuff is a weakness for truly open classes; a purely self-supervised point selector (e.g., based on cost-map uncertainty) would make the framework annotation-free.","The MESS comparison with Grounded-SAM is not perfectly controlled: SAM-MI is trained on COCO-Stuff while Grounded-SAM is not, so part of the 16.7% gain may be training-data, not the mask-injected architecture; re-running with matched training would clarify.","If one replaced the frozen SAM-H with MobileSAM, the accuracy drop is small (about 0.4 mIoU on A-150) while latency drops by roughly 300 ms, pointing to a deployable variant that keeps most of the benefit."],"forward_implications":["If the mask-injected recipe generalizes, any future CLIP-based OVSS model can plug in a frozen SAM with the three modules and obtain consistent mIoU gains without retraining the VLM from scratch.","Because TSPP cuts SAM's prompt count by 96% and the inference latency by about 1.6x versus Grounded-SAM, SAM-powered OVSS becomes practical on a single GPU for deployment.","The gains are larger when the CLIP backbone is mid-size (ViT-B/16): 4-5% relative improvements, whereas a larger ViT-L/14 sees only 1.8-2.5% gains, suggesting mask injection compensates for weaker cost maps.","The ablation removing SMAgg (a 2.2 mIoU drop on A-150) shows that over-segmentation is the main liability of SAM-based OVSS, so aggregation is as important as injection.","Cross-domain results on MESS (earth monitoring: +45% relative over Grounded-SAM) indicate the benefit is strongest on specialized domains where CLIP's cost maps are noisy."],"fun_headline_variants":["Decoupled mask injection lifts open-vocab segmentation by 16.7%","Sparse SAM prompts + mask injection beat Grounded-SAM by 16.7%","Inject SAM masks smartly: 16.7% mIoU gain over Grounded-SAM","SAM masks as guidance, not labels: 16.7% better segmentation","Mask-injected framework: 16.7% mIoU boost with 1.6x speedup"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole framework depends on TSPP's hand-crafted probability targets — that useful SAM prompts are points near ground-truth mask skeletons with point budgets proportional to mask size; if unseen classes and domains violate this prior, the sparse prompts will miss objects and the injected masks will be too few or wrong, and the MESS comparison is partly confounded by training data.","fun_headline_variants_meta":{"raw":{"variants":["Decoupled mask injection lifts open-vocab segmentation by 16.7%","Sparse SAM prompts + mask injection beat Grounded-SAM by 16.7%","Inject SAM masks smartly: 16.7% mIoU gain over Grounded-SAM","SAM masks as guidance, not labels: 16.7% better segmentation","Mask-injected framework: 16.7% mIoU boost with 1.6x speedup"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000798,"raw_usage":{"total_tokens":3383,"prompt_tokens":812,"completion_tokens":2571,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":2454}},"tokens_in":556,"tokens_out":2571,"duration_ms":17642,"temperature":1.0,"reasoning_tokens":2454,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T20:21:21.444002+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled experiment on a novel-category benchmark (e.g., classes not in COCO-Stuff, with no mask-skeleton supervision for TSPP) comparing SAM-MI against its own SAM-less backbone: if the gap disappears or reverses when TSPP is replaced by uniform grid sampling at the same point budget, the claim that sparse text-guided prompting preserves accuracy fails. Also, retraining Grounded-SAM on COCO-Stuff and re-running MESS would test whether the 16.7% lead survives matched training.","supporting_citations":[],"review_version":1}