{"id":"67bd5d6a-4442-4b52-9e25-1dc3448971e2","arxiv_id":"2411.14723","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ESC-Net combines CLIP with pre-trained SAM decoder blocks fed by pseudo prompts from image-text correlations to reach state-of-the-art open-vocabulary segmentation on three major benchmarks.","lead":"ESC-Net is a one-stage model for open-vocabulary image segmentation that blends CLIP's image-text understanding with SAM's segmentation blocks, guided by pseudo prompts derived from image-text correlations. It reports state-of-the-art scores on ADE20K, PASCAL-VOC, and PASCAL-Context while keeping inference costs on par with the fastest prior one-stage method.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim depends on an unreported pseudo-prompt threshold α and no released code; a sensitivity sweep should gate acceptance.","rationale":"I read the paper as an empirical systems contribution. The architecture is plausible: it reuses SAM's mask-decoder transformer blocks with CLIP features and pseudo prompts, and the authors include helpful controls such as random-init SAM, no-SAM, and prompt-type ablations, plus a limitations paragraph. Those controls argue against fabrication or gross internal inconsistency. The reader's weakest assumption—that pretrained SAM blocks transfer to CLIP features—is real, and Table 2 makes it central. But the sharpest, most falsifiable concern is that the PPG threshold α, which directly controls the pseudo masks and points that drive those SAM blocks, is never reported and no code is released. Even a well-designed transfer story cannot be checked without this hyperparameter. The random-init comparison shows the gain is not simply extra capacity, but it does not show robustness to prompt-generation settings. My concern does not move the verdict away from CONDITIONAL; the paper should be accepted only if the authors provide code or the missing α and demonstrate that results are stable across reasonable α values. I do not see grounds to reject: the tables are internally consistent and the ablations are appropriately designed. 'UNCHANGED' is therefore the right recommendation relative to the reader's CONDITIONAL verdict, with the condition now made more explicit.","tokens_in":13420,"tokens_out":5806,"duration_ms":65823,"concrete_test":"Obtain the authors' code (or reimplement PPG from Section 3.3) and run a threshold sweep α ∈ {0.1, 0.3, 0.5, 0.7, 0.9} on A-847 and PC-459 using the CLIP ViT-L/14 recipe, keeping all other settings fixed. If the Table 1 values reproduce within 0.5 mIoU and mIoU varies by less than 1 point across the sweep, the unreported α is not load-bearing. If mIoU varies by more than about 1 point, or if no default α can be identified, the SOTA claim should be reported as conditional on α.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is an empirical SOTA result, so it stands or falls on the reproducibility of the reported pipeline. Section 3.3 defines the Pseudo Prompt Generator: the softmax correlation map is binarized 'using a predefined threshold value α' to produce pseudo masks and, subsequently, pseudo points. However, α is never reported in the implementation details (§4.3) or in any ablation, and no code is released. This threshold is not a minor implementation detail: it determines exactly which regions become SAM prompts, and Table 2 shows that SAM blocks contribute the bulk of the gain (A-847: 18.1 with pretrained SAM vs 4.8 without SAM). If α is mis-set, the pseudo prompts can be nearly empty or nearly full-image, changing the input to the SAM blocks and therefore the final mIoU. The random-init control in Table 2 is a good-faith check that the gain is not merely extra parameters, but it tests initialization rather than prompt quality. Without a reported α or code, the reader cannot determine whether the headline improvements are robust or tied to a specific unstated hyperparameter choice. This is an unverified empirical claim, not evidence of fabrication, but it makes the central SOTA assertion conditional on information the paper does not provide.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ESC-Net, a one-stage open-vocabulary semantic segmentation model that combines a CLIP vision-language encoder with pre-trained SAM mask-decoder transformer blocks. Image-text correlation maps are binarized and clustered into pseudo points and pseudo masks, which are fed through SAM's prompt encoder and then into several pre-trained SAM blocks to refine CLIP image features. A Vision-Language Fusion (VLF) module further refines the correlation maps, and a U-Net-style decoder produces the final masks. The method is evaluated on ADE20K, PASCAL-VOC, and PASCAL-Context, reporting state-of-the-art mIoU with both CLIP ViT-B/16 and ViT-L/14 while maintaining inference time comparable to the one-stage baseline CAT-Seg. Ablations cover the effect of SAM blocks, prompt-type combinations, and model efficiency.","tokens_in":13584,"tokens_out":4290,"duration_ms":40721,"significance":"The core idea of reusing SAM decoder blocks with pseudo prompts generated from CLIP correlation maps is a plausible and potentially efficient alternative to two-stage mask-proposal pipelines. The random-init SAM control in Table 2 is an honest and informative baseline that strengthens the causal claim that the gain comes from pre-trained SAM weights rather than added parameters. If the reported results are reproducible, the method offers a meaningful improvement over CAT-Seg on several benchmarks. However, the manuscript currently lacks critical implementation details, particularly the binarization threshold alpha, and contains ambiguities in the SAM-block formulation, which prevent full validation of the central SOTA claim.","major_comments":[{"comment":"The binarization threshold alpha is never reported. Section 3.3 states that the softmax probability mask is binarized 'using a predefined threshold value alpha' to produce pseudo masks and pseudo points, but no value or sensitivity analysis is given anywhere in Sections 4.3 or 4.5. Since Table 2 shows that the SAM blocks contribute the bulk of the accuracy (A-847 mIoU moves from 4.8 without SAM to 18.1 with pretrained SAM), the quality of the pseudo prompts is a load-bearing component of the reported SOTA numbers. Please report the value of alpha and include an ablation over a range of alpha values.","section":"Section 3.3, Pseudo Prompt Generator"},{"comment":"Equation (2) appears inconsistent with the surrounding text. The text states that the SAM block applies prompt self-attention SA and bidirectional cross-attention BCA, but the equation writes (F_v^n)' = BCA(SA(F_l^n), F_v), where F_l^n is the n-th language feature vector. It does not show the embedded pseudo prompt features P_sparse^n and P_dense^n produced by the prompt encoder described in Section 3.3. This makes the exact computation of the SAM block ambiguous and unreproducible. Please clarify the notation and provide the precise computation that includes the prompt embeddings.","section":"Section 3.4, Eq. (2)"},{"comment":"The abstract claims ESC-Net 'outperforms prior methods in both efficiency and accuracy.' Table 4 shows that relative to CAT-Seg, ESC-Net has more parameters (451.3M vs 433.7M) and higher inference GFLOPs (2203.5 vs 2121.1); only inference time is marginally lower (0.76 s vs 0.78 s). The efficiency claim should be revised to 'comparable efficiency' or otherwise qualified, as the current wording is contradicted by the paper's own efficiency table.","section":"Abstract and Section 4.4, Table 4"},{"comment":"No error bars or multiple-seed results are reported in any table. Several reported improvements are small (e.g., +0.3 mIoU on PAS-20 with ViT-B/16 in Table 1), so without variance estimates it is unclear whether these gains, and the overall SOTA claim, are statistically meaningful. Please report means and standard deviations over at least three random seeds, or otherwise justify why a single run is sufficient.","section":"Section 4.5 and Table 2"},{"comment":"The key hyperparameters N_o=5 pseudo prompts per class and N=4 ESC blocks are selected without any ablation. Since these directly control the prompt count, the number of SAM blocks, and the computational cost, the paper should include an ablation over N and N_o to justify the chosen configuration and to show the sensitivity of the reported results to these choices.","section":"Section 4.3, Implementation Details"}],"minor_comments":[{"comment":"The checkmark layout in Table 3 is garbled: row (e) is described in the text as using all three prompt types, but the table row shows only two checkmarks. Please fix the table so that each row clearly indicates which prompt types are used.","section":"Table 3"},{"comment":"The last column header in Table 2 reads 'PAS-20' but should likely be 'PAS-20b' for consistency with Table 1 and the dataset description in Section 4.1.","section":"Table 2"},{"comment":"The sentence 'Unlike PC-59, PAS-20b is defined without including the \"background\" class' is confusing because the preceding sentences describe PASCAL-VOC and PASCAL-Context without establishing a contrast with PC-59. Please clarify the relationship between PAS-20b and PC-59.","section":"Section 4.1, Datasets"},{"comment":"The qualitative activation maps in Figure 6 would be more convincing if accompanied by quantitative localization metrics or by the same visual comparison for the random-init SAM variant.","section":"Section 4.5, Figure 6"},{"comment":"The limitations paragraph is a useful addition, but the discussion of low-resolution feature levels could be tied more concretely to the mask-resolution behavior observed in the experiments, for example by reporting boundary metrics or high-resolution variants.","section":"Limitations paragraph"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and relevant problem, and the random-init SAM baseline is a good scientific practice. The main blockers are reproducibility: the unreported alpha threshold, the ambiguous Eq. (2), and the absence of code or seed variance. The efficiency claim in the abstract is also overstated relative to Table 4. These issues are fixable within the scope of a revision, so I recommend major revision rather than rejection. If the authors can provide the missing hyperparameter and a corrected, complete formulation, the paper may be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuine one-stage OVSS method, not a repackaging. The core idea—generating pseudo points and masks from CLIP correlation maps and running them through pretrained SAM decoder blocks—is not present in CAT-Seg or EBSeg, and the paper backs it with an honest random-init control. The reported mIoU gains (e.g., +2.1 on A-847 with ViT-L/14) are consistent with the ablations. My main reservation is not the idea but the missing details around the pseudo-prompt threshold alpha. Section 3.3 says the correlation map is binarized with a 'predefined threshold value alpha', but the value is never given in the implementation details or in any ablation. Since Table 2 shows nearly all the gain comes from the pretrained SAM blocks (18.1 vs 5.9 random init vs 4.8 no SAM), the quality of the pseudo prompts is load-bearing. A sensitivity sweep on alpha would tell us whether the result is stable or tuned to a specific operating point. No code is released, so I can't check this myself.\n\nA few smaller issues. The abstract claims the method outperforms prior work 'in both efficiency and accuracy,' but Table 4 shows essentially the same GFLOPs, parameters, and inference time as CAT-Seg (2203 vs 2121 GFLOPs, 451 vs 434 M params). That's comparable, not superior. Also, Eq. (2) writes (F_v^n)' = BCA(SA(F_l^n), F_v), which looks wrong: the input to the prompt self-attention should be the embedded pseudo prompts (P_sparse/P_dense), not the language features. If that's a typo it should be fixed; if it's intentional, it needs explanation. Minor: Table 2's header repeats 'PAS-20' instead of 'PAS-20b'.\n\nOn the plus side: the paper reuses CAT-Seg's cost aggregation baseline and credits it; the ablation in Table 3 (point vs bbox vs mask prompts) is useful; and the random-init control is the right experimental check for whether the gain is just extra parameters. The VLF module with its order-agnostic linear attention is a reasonable addition, even if not heavily novel.\n\nBottom line: the main claim is believable and the work belongs in the OVSS conversation. But as it stands, the missing alpha and the lack of code make the headline result conditional. I'd accept it for review with a clear request: report alpha, add a sensitivity sweep, release code, and fix the notation. A serious referee can sort that out in a round.","headline":"Convincing one-stage OVSS method with real gains from SAM decoder blocks; the unreported pseudo-prompt threshold and missing code are the only things keeping me from a clean yes.","tokens_in":14232,"tokens_out":2953,"would_cite":true,"duration_ms":68204,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ESC-Net claims that one-stage open-vocabulary semantic segmentation can reuse pretrained SAM decoder blocks, driven by pseudo prompts from CLIP correlation maps, to beat two-stage and prior one-stage methods at comparable cost.","keywords":["open-vocabulary semantic segmentation","segment anything model","CLIP","pseudo prompts","vision-language fusion","one-stage segmentation","pixel-level classification","efficient inference"],"falsifier":"A decisive experiment would take ESC-Net and swap in the decoder blocks of a different promptable segmentation model, or swap CLIP features for a different frozen vision-language encoder's features without retraining; if A-847 mIoU collapses to near the no-SAM baseline (4.8), then the gain is not generic promptable-decoder transfer but a CLIP-specific SAM adaptation. A cheaper check already in the paper: with randomly initialized SAM blocks the A-847 score is only 5.9, so any variant that keeps that score while removing the pretrained weights would refute the claim that pretrained SAM blocks carry the result.","tokens_in":13138,"feed_emoji":"🧩","tokens_out":5926,"duration_ms":54092,"temperature":0.7,"pith_summary":"ESC-Net is a one-stage model for open-vocabulary semantic segmentation: it labels every pixel of an image using class names that were never seen during training. The paper's central claim is that the heavy mask-proposal stage of prior two-stage systems can be replaced by reusing the decoder blocks of SAM, the segment-anything model, and prompting them with pseudo points and masks read off from CLIP's image-text correlation map. This yields more spatially precise and denser masks than the previous one-stage correlation methods, and it does so at roughly the same inference cost as the CAT-Seg baseline rather than the much higher cost of running SAM's image encoder. On standard benchmarks the paper reports gains on every test set with both CLIP ViT-B/16 and CLIP ViT-L/14, for example A-847 mIoU rising from 16.0 to 18.1 and PC-459 from 23.8 to 27.0 against the prior best. If correct, this makes high-quality open-vocabulary segmentation practical at roughly one-stage cost.","feed_headline":"SAM decoder blocks, prompted by CLIP, beat two-stage segmentation","feed_subtitle":"One-stage ESC-Net reports gains on six benchmarks at CAT-Seg-level cost.","key_machinery":"The load-bearing object is the SAM mask-decoder transformer block, reused with pretrained weights and wrapped together with a Pseudo Prompt Generator (PPG) into an 'ESC block'. PPG turns the CLIP correlation map for each class into a binarized, k-means-clustered set of candidate regions, then selects for each region the highest-probability point and the region mask as pseudo prompts; these enter through SAM's prompt encoder as sparse and dense prompt embeddings. The SAM block then applies prompt self-attention and bidirectional image-prompt cross-attention, updating CLIP's image features class-by-class in parallel batches. The paper's ablation attributes the performance jump to the pretrained SAM weights: with SAM blocks removed A-847 mIoU is 4.8, with randomly initialized SAM blocks it is 5.9, and with pretrained blocks it is 18.1.","core_discovery":"The paper's discovery is that the promptable segmentation machinery of SAM can be transplanted, with its pretrained weights, into a one-stage correlation pipeline built on CLIP. Instead of generating full candidate masks with SAM's image encoder and then classifying them, ESC-Net computes a per-class cosine-similarity map between CLIP image and text features, thresholds and clusters it into pseudo-object regions, and converts those regions into point and mask prompts. These prompts are embedded by SAM's prompt encoder and fed, together with CLIP's image features, into the pretrained SAM decoder blocks, which refine the CLIP features with class-agnostic spatial aggregation. A Vision-Language Fusion module then re-correlates the refined image features with text and produces the final masks. The authors claim this achieves state-of-the-art results on ADE20K, PASCAL-VOC, and PASCAL-Context with both CLIP ViT-B/16 and CLIP ViT-L/14, while keeping parameters (451.3M), inference time (0.76 s), and GFLOPs (2203.5) comparable to the much simpler CAT-Seg baseline rather than to the heavier two-stage pipelines.","pith_inferences":["Editorial inference: the large gap between random-init (5.9) and pretrained (18.1) SAM blocks suggests the pretrained decoder is supplying a learned spatial prior about object boundaries, not just a trainable architecture; a further test would be to freeze the SAM blocks and see whether the prompt generator alone can adapt to CLIP features.","Editorial inference: because ESC-Net still refines low-resolution CLIP features, the paper's own limitation statement implies that pairing it with a lightweight boundary-refinement module could recover the boundary quality of two-stage methods without paying for their full cost.","Editorial inference: the recipe could transfer to other promptable segmentation decoders, and the transfer would be testable by swapping in a decoder pretrained on a different task or domain; if the performance jump over random init persists, the effect is generic promptable-decoder transfer, and if only SAM's decoder shows it, the effect is specific to SAM's training distribution."],"forward_implications":["A dedicated mask proposal generator can be replaced by a pretrained SAM decoder fed with pseudo prompts, so open-vocabulary segmentation no longer has to pay for SAM's full image encoder at inference.","Because all prompt operations are batched per class, the model scales to hundreds of classes at roughly constant per-class marginal cost, limited mainly by the memory of the expanded correlation maps.","The method inherits CLIP's open vocabulary: new classes can be added by changing text prompts without retraining the segmentation head.","On the standard benchmarks, the reported gains hold across all six evaluation splits with both ViT-B/16 and ViT-L/14 backbones, so the benefit is not tied to one model scale."],"supporting_citations":[{"why":"Supplies the pretrained mask-decoder transformer blocks and prompt encoder that ESC-Net reuses for spatial aggregation.","marker":"[19]"},{"why":"Supplies the CLIP vision and language encoders whose correlation map forms the pseudo-prompt source.","marker":"[28]"},{"why":"The one-stage correlation-based baseline whose cost-aggregation design and CLIP fine-tuning recipe ESC-Net adopts and outperforms.","marker":"[5]"},{"why":"Provides the k-means clustering whose region splitting turns correlation maps into pseudo point and mask prompts.","marker":"[12]"},{"why":"Supplies the Swin transformer structure used in the VLF module's vision attention for memory efficiency.","marker":"[23]"},{"why":"Supplies the linear attention used in the VLF module's language attention to handle order-agnostic variable class counts.","marker":"[18]"},{"why":"Provides the COCO-Stuff training set with 171 annotated categories used to train ESC-Net.","marker":"[2]"},{"why":"Supplies the ADE20K validation sets A-150 and A-847 used in the headline quantitative claims.","marker":"[39]"},{"why":"Supplies PASCAL-VOC evaluation (PAS-20, PAS-20b) and the mIoU metric used across benchmarks.","marker":"[9]"},{"why":"Supplies the PASCAL-Context evaluation sets PC-59 and PC-459 used for the large-vocabulary gains.","marker":"[25]"}],"fun_headline_variants":["ESC-Net: one-stage SAM+CLIP for open-vocab segmentation","SAM decoder blocks power one-stage open-vocab segmentation","Prompted SAM decoder blocks replace two-stage pipelines","One-stage ESC-Net: SAM prompts from CLIP, record accuracy","Fusing SAM and CLIP in one stage cuts cost, boosts accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire performance gain rests on the assumption that SAM's pretrained decoder blocks, trained on SAM image features, still perform useful spatial aggregation when fed CLIP image features and pseudo prompts, even though they never saw CLIP features during pretraining.","fun_headline_variants_meta":{"raw":{"variants":["ESC-Net: one-stage SAM+CLIP for open-vocab segmentation","SAM decoder blocks power one-stage open-vocab segmentation","Prompted SAM decoder blocks replace two-stage pipelines","One-stage ESC-Net: SAM prompts from CLIP, record accuracy","Fusing SAM and CLIP in one stage cuts cost, boosts accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000616,"raw_usage":{"total_tokens":2857,"prompt_tokens":937,"completion_tokens":1920,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":1830}},"tokens_in":553,"tokens_out":1920,"duration_ms":12505,"temperature":1.0,"reasoning_tokens":1830,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:59:09.749872+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive experiment would take ESC-Net and swap in the decoder blocks of a different promptable segmentation model, or swap CLIP features for a different frozen vision-language encoder's features without retraining; if A-847 mIoU collapses to near the no-SAM baseline (4.8), then the gain is not generic promptable-decoder transfer but a CLIP-specific SAM adaptation. A cheaper check already in the paper: with randomly initialized SAM blocks the A-847 score is only 5.9, so any variant that keeps that score while removing the pretrained weights would refute the claim that pretrained SAM blocks carry the result.","supporting_citations":[{"cited_title":"Cat- seg: Cost aggregation for open-vocabulary semantic seg- mentation","cited_arxiv_id":null,"evidence_quote":"The one-stage correlation-based baseline whose cost-aggregation design and CLIP fine-tuning recipe ESC-Net adopts and outperforms."},{"cited_title":"Transformers are rnns: Fast autoregressive transformers with linear attention","cited_arxiv_id":null,"evidence_quote":"Supplies the linear attention used in the VLF module's language attention to handle order-agnostic variable class counts."},{"cited_title":"Semantic under- standing of scenes through the ade20k dataset","cited_arxiv_id":null,"evidence_quote":"Supplies the ADE20K validation sets A-150 and A-847 used in the headline quantitative claims."},{"cited_title":"The pascal visual object classes (voc) challenge","cited_arxiv_id":null,"evidence_quote":"Supplies PASCAL-VOC evaluation (PAS-20, PAS-20b) and the mIoU metric used across benchmarks."},{"cited_title":"The role of context for object detection and semantic segmentation in the wild","cited_arxiv_id":null,"evidence_quote":"Supplies the PASCAL-Context evaluation sets PC-59 and PC-459 used for the large-vocabulary gains."}],"review_version":1}