{"id":"db4e6739-205e-4807-9a92-06fddf3f24f5","arxiv_id":"2505.11676","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"DPSeg adds Stable Diffusion generated visual prompts to CLIP text prompts to build cost volumes for open-vocabulary semantic segmentation, reporting state-of-the-art mIoU on five benchmarks.","lead":"DPSeg is a new image-segmentation method that pairs text descriptions with AI-generated example images as prompts to label every pixel in a photo. It reports higher accuracy than previous open-vocabulary segmentation systems on five public benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DPSeg's SOTA margin vanishes at M=40 templates (Table 4), and M=80 was chosen by test-set ablation; the headline comparison may be cherry-picked.","rationale":"The reader's chosen weakest assumption is Stable Diffusion visual-prompt fidelity, but I do not think that is the most load-bearing premise: Table 5 explicitly tests degraded visual prompts (up to 80% Gaussian noise) and shows the method still beats SED, so the architecture is at least somewhat robust to imperfect prompts. The more decisive vulnerability is the experimental protocol. The paper itself contains the evidence: Table 4 shows that with 40 templates the ConvNeXt-B configuration scores below SED on all five benchmarks, and only reaches SOTA after the template count is increased to 80. Because Table 4 is an ablation run directly on the test benchmarks and no validation-based selection is described, the reported SOTA numbers may be an artifact of test-set template selection rather than a genuine architectural advantage. The second-pass Inference II result compounds this by reusing each test image's predicted masks as visual prompts, a test-time adaptation not present in the compared baselines. The numerical inconsistencies in Sec. 5.3 and the absence of code or error bars mean the central claim cannot currently be checked. I therefore keep the reader's CONDITIONAL verdict, but the binding condition should be a validation-fixed template protocol and released artifacts, not merely additional robustness analysis of visual prompts.","tokens_in":14961,"tokens_out":6501,"duration_ms":66700,"concrete_test":"Re-run Table 1 with a validation-fixed template schedule: select M on the COCO-Stuff validation set, or fix M=40 (the next configuration reported in Table 4), and report only Inference I for both DPSeg and SED under identical template sets. If DPSeg falls below SED on any of the five benchmarks, the paper's SOTA claim is not supported as stated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline claim is 'significantly outperforms existing SOTA' (Abstract, Table 1). The most load-bearing condition for that claim is that the reported configuration is not selected using test labels. The paper gives no evidence for this. In Sec. 5.5, Table 4 ablates the number of prompt templates directly on the five evaluation benchmarks and picks 80 because it yields the best test mIoU; no validation split or fixed template budget is described. This is not a cosmetic issue: with M=40 the ConvNeXt-B model scores 10.1/16.8/29.6/55.9/91.8 on A-847/PC-459/A-150/PC-59/PAS-20, which is below the SED baseline listed in Table 1 (11.4/18.6/31.6/57.3/94.4). DPSeg only surpasses SED at M=80 (12.0/19.5/32.9/58.1/96.0). Hence the entire SOTA margin is contingent on a hyperparameter that appears to have been selected on the test benchmarks. The top-line Inference II result adds a further test-image-specific second pass (Sec. 4.3), a transductive refinement not used by any compared method. The manuscript's numerical inconsistencies (Sec. 5.3 quotes gains of +1.1/+1.5/+2.5/+1.1 where Table 1 shows 0.6/0.9/1.3/0.8, and an average Inference-II gain of +0.87% that no row supports) reinforce that the reported numbers are not independently checkable. Without code, seeds, or a validation-based template protocol, the SOTA claim is unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DPSeg, an open-vocabulary semantic segmentation method that generates visual prompts by running Stable Diffusion on text templates, averages the visual and text prompt embeddings in CLIP space, computes a pixel-level cost volume, and feeds it along with multi-scale visual cost volumes into a cost volume-guided decoder. A two-pass semantic-guided refinement replaces visual prompts with crops from first-pass predicted masks. The authors report state-of-the-art mIoU on A-847, PC-459, A-150, PC-59, and PAS-20 in both ConvNeXt-B and ConvNeXt-L configurations.","tokens_in":15323,"tokens_out":5607,"duration_ms":50961,"significance":"If the reported results are reproducible, the paper makes a useful contribution: it provides empirical evidence (Figs. 2 and 3) that visual prompts reduce the CLIP modality gap for OVSS, and the ablations on prompt strategies, cost volume fusion, and decoder guidance are well designed. The proposed architecture is coherent, and the reported gains are internally plausible for a method that adds intra-modal cues and multi-scale cost volumes. The value of the work is, however, currently contingent on fixing the test-set selection of the template count, providing a fair comparison for the two-pass inference, and correcting the numerical inconsistencies in the experimental section.","major_comments":[{"comment":"The number of prompt templates M is ablated directly on the five test benchmarks and M=80 is selected because it gives the best test mIoU; no validation split or fixed template budget is described. This is load-bearing: at M=40 the ConvNeXt-B configuration scores 10.1/16.8/29.6/55.9/91.8 on A-847/PC-459/A-150/PC-59/PAS-20, below the SED baseline (11.4/18.6/31.6/57.3/94.4) in Table 1, while at M=80 it exceeds SED. The claimed advantage over the strongest baseline therefore appears to be an artifact of test-set ablation unless a validation-based protocol for choosing M is provided.","section":"§5.5, Table 4"},{"comment":"The headline 'Inference II' results are obtained from a two-pass procedure in which the first-pass predicted masks on the test image are used to crop and replace the visual prompts for the second pass. This makes the comparison with single-pass methods such as SED, CAT-Seg, and SAN not apples-to-apples, because the method is effectively using test-image content (its own predictions) to refine itself. The authors should report Inference I as the primary result in the abstract and conclusion, or add a comparison in which all methods are evaluated under a comparable single-pass protocol.","section":"§4.3, Table 1"},{"comment":"The quantitative reporting is internally inconsistent. Section 5.3 claims gains of +1.1%, +1.5%, +2.5%, and +1.1% over SED on 'A-187', PC-459, A-150, and PC-59, but Table 1 shows Inference I ConvNeXt-B deltas of +0.6, +0.9, +1.3, and +0.8 (plus +1.6 on PAS-20), and 'A-187' is not a dataset used in the paper. The claimed average Inference-II gain of +0.87% over Inference I is not supported by Table 1, whose per-dataset gains are 0.5/0.6/0.4/0.3/0.9 (mean 0.54); the large-backbone gain of +0.92% is also unsupported (mean 0.70). These discrepancies must be corrected before the results can be taken as reliable.","section":"§5.3, Table 1"},{"comment":"The paper does not provide code, a release of the visual prompt bank, or multi-seed results with error bars. The visual prompt bank (M=80 Stable Diffusion images per category) is a core input to the method, and without it the exact reported numbers cannot be reproduced or independently verified. Please release the prompt bank and code, and report mean and standard deviation over at least three training runs for the main configurations.","section":"§5.1, §5.2, §5.5"}],"minor_comments":[{"comment":"The citation for PFENet appears to be incorrect: in Sec. 2.3, PFENet is cited as [68], but reference [68] is 'Extract free dense labels from CLIP' by Zhou et al.; the PFENet paper is not in the reference list.","section":"§2.3, References"},{"comment":"The SCAN paper is cited twice as [26] and [27] with different venues; please consolidate to a single reference.","section":"Table 1, References [26] and [27]"},{"comment":"The cosine similarity experiment is described only as using 'more than one hundred samples'; please specify the number of images, the number of categories, and whether any error bars or confidence intervals are shown, since Fig. 2 appears to show raw scatter points.","section":"§3.1, Fig. 2"},{"comment":"The t-SNE figure reports distances without stating which embedding pairs are averaged or how the image feature is obtained, which makes the headline distance of 0.18 hard to interpret.","section":"Fig. 1"},{"comment":"The text says 'an average increase of 1.8% mIoU' when all scales are incorporated, but the row deltas in Table 3 average 1.3; please reconcile this number with the table.","section":"§5.5, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about M=80 being selected on the test benchmarks is correct and is the main reason I cannot recommend acceptance. The two-pass inference and the numerical inconsistencies further weaken the headline claim. There are also citation hygiene issues (duplicate SCAN references and a likely PFENet citation error) that should be fixed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the dual-prompt idea is the real thing, but the SOTA margin is softer than the abstract implies. Here's what I see.\n\nWhat's new: using Stable Diffusion to generate per-category visual prompts, then averaging visual and text embeddings before computing cost volumes (Eq. 1) is a clean and sensible way to reduce the CLIP modality gap. The motivation experiments in Sec. 3.1/3.2 are convincing—visual prompts do sit closer to image features and produce sharper cost maps. The multi-scale visual cost volumes (Eq. 2) and the decoder integration are reasonable architecture choices. The ablations in Tables 2 and 3 are coherent and show each piece helping.\n\nWhere it gets shaky: Table 4 sweeps the number of prompt templates directly on the five evaluation benchmarks and picks 80 because it gives the best mIoU. That's test-set-based hyperparameter selection, and it matters: at M=40 the same ConvNeXt-B model scores below SED on every dataset (e.g., 10.1 vs 11.4 on A-847). The entire SOTA gap only appears at M=80. No validation split or fixed-budget argument is given.\n\nInference II adds another comparison problem. It uses the test image's own first-pass segmentation to crop and replace visual prompts before the second pass. That is transductive—it consumes the test image's predicted structure—and none of the compared methods get a similar second pass. The +0.5 to +1.1 mIoU from Inference II is a different kind of gain than a better learned representation.\n\nThe paper also has internal inconsistencies. Sec. 5.3 quotes gains of +1.1/+1.5/+2.5/+1.1 over SED, but Table 1 shows 0.6/0.9/1.3/0.8 for the ConvNeXt-B rows. The claimed +0.87% average Inference-II gain is not what the table averages to. The PFENet citation in Sec. 2.3 points to ref [68], which is MaskCLIP, not PFENet. No code or visual prompt bank is released, so the numbers can't be independently checked.\n\nThe core idea is worth testing. I'd send this to a serious referee, but with the expectation of major revision: release code and the SD prompt bank, pick M on a validation split or justify a fixed budget, present Inference II as a transductive extension rather than part of the comparison, and fix the numerical reporting. The paper is not circular or incoherent—it's just not ready to be taken at face value.","headline":"DPSeg's dual-prompt idea is genuinely new, but the SOTA claim rests on template counts chosen on the test sets and a transductive second pass, so the headline numbers are not trustworthy as reported.","tokens_in":15896,"tokens_out":3831,"would_cite":false,"duration_ms":35215,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adding Stable Diffusion-generated visual prompts to CLIP text embeddings closes the text-image gap and yields state-of-the-art open-vocabulary segmentation.","keywords":["open-vocabulary semantic segmentation","dual-prompt cost volume","visual prompting","Stable Diffusion","CLIP","cost volume-guided decoder","semantic-guided prompt refinement","multi-scale feature fusion"],"falsifier":"Construct a rare-category subset of a benchmark like ADE20K-847, score each Stable Diffusion visual prompt for visible fidelity, and check whether DPSeg's per-category IoU drops below a text-only cost volume baseline whenever the synthetic prompt is obviously misrendered; if it does not, the faithfulness premise is not the actual driver, and if it does, the premise is violated.","tokens_in":14738,"feed_emoji":"🖼️","tokens_out":9224,"duration_ms":84684,"temperature":0.7,"pith_summary":"The paper's central claim is that open-vocabulary segmentation is held back by the text-image modality gap in CLIP's embedding space, and that this gap can be largely bridged by generating a visual prompt for each category with Stable Diffusion and averaging it with the text prompt before computing pixel-level cost volumes. The resulting DPSeg system reports improved mIoU over prior cost-volume methods on ADE20K-847, Pascal Context-459, ADE20K-150, Pascal Context-59, and Pascal VOC, with a second inference pass that crops detected objects and uses them as refined visual prompts adding further gains. A sympathetic reader would take away that synthetic visual prompts are a practical, cheap source of category prototypes that improves both seen and unseen class segmentation.","feed_headline":"Text plus image prompts beat text-only open-vocab segmentation","feed_subtitle":"Stable Diffusion reference images close CLIP's text-image gap, sharpening rare-class segmentation on five benchmarks.","key_machinery":"The load-bearing mechanism is the dual-prompt cost volume: a pixel-level cosine-similarity map between image features and a unified prompt embedding formed by averaging CLIP text embeddings with Stable Diffusion visual embeddings for each category. Around it, DPSeg adds (1) multi-scale visual cost volumes $F_v^j$ that inject intra-modality, category-level similarity at 4x, 8x, and 16x downsampling without upsampling the initial cost volume, and (2) a two-pass semantic-guided refinement in which the first pass's segmentation masks crop detected objects and substitute them as the visual prompts for the second pass.","core_discovery":"The paper establishes that visual prompts live closer to image features than text prompts do in CLIP space, and that this intra-modality proximity translates into cost volumes with sharper spatial-semantic structure. On that basis, DPSeg defines the dual-prompt cost volume $F_c(x,y,k,m)$ as the cosine similarity between the image feature at each location and the averaged text-plus-visual prompt embedding $R = \\mathrm{Avg}(V+T)$ (Eq. 1), and additionally derives multi-scale visual cost volumes $F_v^j$ from global-average-pooled visual prompt prototypes matched against intermediate image features (Eq. 2). A cost volume-guided decoder consumes these volumes stage by stage, and a semantic-guided refinement pass replaces generic visual prompts with cropped objects from the first segmentation output. In the paper's reported numbers, this combination outperforms the compared methods on all five benchmarks in both base and large backbone configurations.","pith_inferences":["An extension the paper leaves implicit: per-category gains should track Stable Diffusion's fidelity for that category, so a confidence-weighted blend of text and visual prompts could rescue classes whose synthetic renderings are poor.","If the two-pass refinement works because the second pass sees instance-level crops, the same idea could be applied to open-vocabulary detection or panoptic segmentation, where detected masks already provide instance-level prompts; the paper only evaluates semantic segmentation.","The modality-gap diagnosis is independent of the decoder, so the same dual-prompt embeddings could be plugged into other cost-volume architectures, with the decoder contribution being additive rather than strictly necessary.","The paper's numbers are dataset averages; reporting per-category IoU broken down by generation quality would turn the faithfulness assumption into a testable design guide."],"forward_implications":["Synthetic text-to-image generation becomes a direct source of category prototypes, reducing dependence on hand-collected reference images for unseen classes.","Averaging prompt embeddings before similarity, rather than fusing separately computed cost volumes, is the better fusion strategy for text and visual prompts.","A second inference pass that re-prompts with detected object crops yields consistent gains over a single pass, so test-time self-refinement is a viable accuracy lever.","Multi-scale visual cost volumes preserve small-object detail better than naive cost volume upsampling, giving a concrete decoder design for fine-grained open-vocabulary segmentation.","Even degraded visual prompts keep the method above a text-only baseline, suggesting the dual-prompt design is not brittle to prompt quality."],"supporting_citations":[{"why":"Supplies the pre-trained CLIP text and image encoders whose modality gap is measured and whose embeddings define every cost volume.","marker":"[37]"},{"why":"Generates the synthetic visual prompts from text templates, the central new ingredient of the dual-prompt scheme.","marker":"[38]"},{"why":"Introduces pixel-level cost volumes for open-vocabulary segmentation and the text-template strategy that DPSeg extends.","marker":"[7]"},{"why":"Provides the strongest compared cost-volume baseline and the hierarchical encoder setting that DPSeg builds on.","marker":"[52]"},{"why":"Contributes mask-adapted CLIP practices and text template choices that DPSeg follows for prompt construction.","marker":"[23]"},{"why":"Supplies the COCO-Stuff training data whose 171 classes define the seen categories for the supervised part of training.","marker":"[3]"},{"why":"Provides the ADE20K evaluation benchmarks where the central performance claim is measured.","marker":"[67]"}],"fun_headline_variants":["Visual prompts close CLIP gap, sharpening open-vocab segmentation","Dual prompts: text plus image beat text-only for open-vocab segmentation","DPSeg: dual prompt cost volumes shrink CLIP domain gap","From text to visual prompts: better open-vocabulary segmentation","Visual prompts from images fix CLIP text-image misalignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Stable Diffusion-generated images faithfully represent every queried category, including rare unseen ones, so averaging their embeddings with text embeddings improves rather than pollutes the prompt signal.","fun_headline_variants_meta":{"raw":{"variants":["Visual prompts close CLIP gap, sharpening open-vocab segmentation","Dual prompts: text plus image beat text-only for open-vocab segmentation","DPSeg: dual prompt cost volumes shrink CLIP domain gap","From text to visual prompts: better open-vocabulary segmentation","Visual prompts from images fix CLIP text-image misalignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000231,"raw_usage":{"total_tokens":1463,"prompt_tokens":903,"completion_tokens":560,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":468}},"tokens_in":519,"tokens_out":560,"duration_ms":5546,"temperature":1.0,"reasoning_tokens":468,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:50:19.120554+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a rare-category subset of a benchmark like ADE20K-847, score each Stable Diffusion visual prompt for visible fidelity, and check whether DPSeg's per-category IoU drops below a text-only cost volume baseline whenever the synthetic prompt is obviously misrendered; if it does not, the faithfulness premise is not the actual driver, and if it does, the premise is violated.","supporting_citations":[{"cited_title":"Sed: A simple encoder-decoder for open- vocabulary semantic segmentation","cited_arxiv_id":null,"evidence_quote":"Provides the strongest compared cost-volume baseline and the hierarchical encoder setting that DPSeg builds on."},{"cited_title":"Coco- stuff: Thing and stuff classes in context","cited_arxiv_id":null,"evidence_quote":"Supplies the COCO-Stuff training data whose 171 classes define the seen categories for the supervised part of training."},{"cited_title":"Scene parsing through ade20k dataset","cited_arxiv_id":null,"evidence_quote":"Provides the ADE20K evaluation benchmarks where the central performance claim is measured."}],"review_version":1}