{"id":"f9226375-0f5d-42ae-a727-2ae095b6ad18","arxiv_id":"2412.10292","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A lightweight cross-attention between text and query tokens in the mask decoder improves prompt-aligned mask proposals for two-stage open-vocabulary segmentation.","lead":"The paper adds a text-guidance step to the mask suggestion stage of open-vocabulary segmentation models, so the masks are generated with the prompt in mind. This boosts accuracy by 1 to 3 points on five benchmarks and lets models segment abstract prompts like place names.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-prompt inference (M=1) is never quantitatively tested, while Eq. 5's softmax over text tokens is trained with M around 171; the arbitrary-prompt claim may fail under this distribution shift.","rationale":"The concern is load-bearing because the abstract and Sec. 4.3 explicitly claim arbitrary-prompt capability, which is what distinguishes PMP from a 1-3 mIOU adapter. The benchmark gains are consistent, and the first-stage recall improvements in Table 4 provide credible support for the narrower claim that proposals improve on known class lists; I do not dispute those numbers. The weak point is the extrapolation from a softmax attention module trained with many text tokens to inference with one text token. This is an internal property of Eq. 5, not a disagreement with community consensus, and it is testable. The reader's weakest assumption (transferability to abstract/proprietary prompts) is essentially the same concern; I sharpen it by identifying the M-dependence of the text-query softmax as the specific failure mode. The paper's own statements do not resolve this: Appendix A.2 lists failure cases but no statistics, and Appendix A.5 limits precision, not vocabulary transfer. Therefore the verdict should remain CONDITIONAL, pending a quantitative single-prompt evaluation. If the M=1 test shows no degradation, the concern is resolved and the arbitrary-prompt claim gains support.","tokens_in":19521,"tokens_out":8707,"duration_ms":82503,"concrete_test":"Run OVSeg+PMP on COCO-Stuff val in two configurations: (A) the standard protocol with all 171 class-name text tokens; (B) a single-prompt protocol where each image is evaluated once per class name with M=1, measuring first-stage recall and final mIOU. If per-prompt scores fall substantially below the all-token protocol, the softmax-over-M module does not transfer to single arbitrary prompts. As a second check, fine-tune PMP with M=1 prompts and verify whether the qualitative successes in Figs. 3 and 7 persist beyond hand-picked examples.","verdict_should_be":"UNCHANGED","load_bearing_attack":"PMP's central claim is that it enables mask retrieval for arbitrary single prompts such as 'love' or 'Times Square' (Sec. 4.3, Figs. 3 and 7). This claim is supported only by a few qualitative examples. There is a concrete architectural reason to doubt it. In Eq. 5, Q' = softmax(Q K_t^T) V_t, with the softmax taken over the M text tokens. During COCO-Stuff training, M is the number of class names (171) or the number of nouns in a caption; at inference for a single arbitrary prompt, M=1. With M=1, the softmax returns a single all-ones weight, so every query receives the same projected text vector, and the text-conditioned query features entering the subsequent image cross-attention cannot differentiate queries or route them to different text concepts. The model is neither trained nor quantitatively evaluated in this M=1 regime: Table 1 and Table 4 evaluate with full class lists (e.g., 150/847 classes), and Appendix A.2 reports only anecdotal failure cases without frequency. The paper's own limitation statement (A.5) addresses mask precision, not vocabulary transfer. Thus the arbitrary-prompt generalization claim rests on an untested distribution shift.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Prompt-Guided Mask Proposal (PMP), a lightweight modification to query-based mask-generator decoders (Mask2Former/MaskFormer style) in two-stage open-vocabulary segmentation. The method inserts a text-query cross-attention step before the standard image cross-attention in each decoder layer, so that mask proposals are conditioned on the input text prompt. The authors combine PMP with OVSeg, ODISE, SAN, and FC-CLIP, and report consistent mIOU gains of roughly 0.6–3.9 points on ADE-847, PC-459, ADE-150, PC-59, and Pascal VOC, plus panoptic segmentation gains on ADE20K and COCO. The paper also presents qualitative examples for prompts such as \"Yellowstone,\" \"love,\" and \"Times Square,\" arguing that PMP enables mask retrieval for arbitrary single prompts rather than only benchmark class lists. The main claimed contribution is improved first-stage mask proposal quality conditioned on text, supported by first-stage recall ablations in Table 4.","tokens_in":19756,"tokens_out":6048,"duration_ms":57431,"significance":"If the reported gains hold, PMP is a useful and simple adapter for two-stage open-vocabulary segmentation models, and the breadth of the evaluation—four different methods/backbones across five benchmarks, plus panoptic results—is a genuine strength. The first-stage recall improvements in Table 4 are particularly informative because they show the effect is in proposal generation rather than only in the second-stage classifier. However, the paper's most distinctive claim, that the method supports arbitrary single text prompts in the style of the qualitative figures, is not quantitatively evaluated, and the architectural change has an untested distribution-shift issue at M=1. The empirical contribution is solid but currently supported without variance estimates or a controlled single-prompt benchmark.","major_comments":[{"comment":"The central claim that PMP enables mask retrieval for arbitrary single prompts such as \"love,\" \"Times Square,\" or \"MIT CSAIL\" is supported only by qualitative examples. All quantitative tables evaluate full benchmark class lists (M=150–847 text tokens). For a single prompt, M=1 in Eq. (5): the softmax over text tokens is degenerate, every query receives the same projected text vector, and the subsequent image cross-attention produces the same attention weights across queries except through the residual X_{l-1}. This is a structurally different regime from the M≈171 training distribution, and the manuscript neither trains nor quantitatively evaluates in this regime. Appendix A.2 reports failure cases only anecdotally, and the limitation statement in Appendix A.5 addresses mask precision rather than vocabulary transfer. Please add a quantitative single-prompt/arbitrary-prompt evaluation (for example, mask IoU or recall against manually annotated regions for a set of proper nouns and abstract words), and consider whether training should sample variable M to cover the M=1 case.","section":"Sec. 4.3, Figs. 3 and 7; Eq. (5)"},{"comment":"The central quantitative claim of consistent gains is presented without error bars, multiple seeds, or significance tests. Some reported gains are small (for example, 0.6 mIOU for FC-CLIP + PMP on VOC, and 0.6 mIOU for ODISE + PMP on PC-459), and the manuscript does not state whether the baseline numbers were re-run in the same codebase or taken from the original papers. Without variance information or a statement about experimental control, it is hard to assess whether the smaller gains exceed noise. Please provide per-seed results or confidence intervals and clarify the origin of the baseline numbers.","section":"Table 1"},{"comment":"The training-time construction of the text tokens is unspecified. The paper does not state whether the text input during COCO-Stuff training is the full set of 171 class names, extracted nouns from captions, or a sampled subset, nor how M is distributed across training examples. Since the behavior of Eq. (5) depends on M, and since the paper argues for transfer to arbitrary prompts, this detail is necessary both for reproducibility and for assessing the M=1 distribution shift described above.","section":"Sec. 4.2 and A.3"}],"minor_comments":[{"comment":"The text says the four candidate decoding strategies are compared in Table 5, but the strategy comparison is presented in Table 3; Table 5 contains the backbone and hyperparameter ablations.","section":"Sec. 4.4"},{"comment":"The index notation \"{t_j}_{N}^{M}=1\" is malformed; it should indicate text tokens indexed j=1,...,M.","section":"Eq. (5)"},{"comment":"The text states that PMP brings \"0.1s\" of added latency, but the reported inference times (1.03s vs 1.02s) imply 0.01s of total added latency, and the first-stage difference is also 0.01s.","section":"Appendix A.4"},{"comment":"In the OVSeg paragraph, \"to combine PMP with ODISE\" should read \"to combine PMP with OVSeg.\"","section":"Sec. 4.2"},{"comment":"There are numerous typographical and grammatical errors, including \"the in the second stage,\" \"summmer,\" \"ensambling,\" and \"demonstracts\"; the manuscript should be carefully proofread before resubmission.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable empirical engineering contribution, and the consistent direction of the reported gains is encouraging. The decisive issue is the gap between the headline arbitrary-prompt claim and the quantitative evaluation, which currently only covers full class lists. If the authors add a controlled single-prompt evaluation and variance estimates, the paper could become acceptable; without those, the contribution is a modest decoder modification whose main advertised advantage remains unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PMP is a clean, lightweight plug-in: one text-query cross-attention block before each standard cross-attention in a Mask2Former-style decoder, applied to four two-stage open-vocabulary segmenters. The gains are consistent—1–3 mIOU across five benchmarks, with larger first-stage recall improvements in Table 4. That is genuine, useful engineering, and the design is properly ablated against concat and text-as-query alternatives.\n\nThe paper's central claim, however, is that PMP handles arbitrary text prompts like 'Yellowstone' or 'love.' That claim is currently supported only by qualitative examples, and the stress-test note points to a real architectural reason for doubt. Training uses M≈171 text tokens (or caption nouns); the quantitative eval uses full class lists (M=150–847). Inference with a single prompt is M=1, where the softmax in Eq. 5 over text tokens degenerates: every query gets the same text vector. Residuals and image cross-attention may still produce diverse masks, but no experiment measures this regime. The limitation section (A.5) addresses mask precision, not vocabulary transfer. A small quantitative study with single prompts—even recall@20 on a hand-built set—would resolve the gap.\n\nOther soft spots are minor by comparison: no error bars or significance tests; training-prompt sampling is unspecified; baseline numbers appear copied from prior papers. All fixable, but they matter for a 1–3 point claim.\n\nWho should read this: anyone building two-stage open-vocabulary segmentation on Mask2Former. The plug-in is practical and likely to be adopted. But the 'arbitrary prompts' framing overestimates what is demonstrated. The paper deserves a serious referee; the right outcome is major revision with the missing single-prompt evaluation, not rejection.","headline":"Consistent 1–3 mIOU gains for a clean plug-in, but the arbitrary single-prompt claim is untested and likely distribution-shifted (M=1 vs. M≈171 training).","tokens_in":20305,"tokens_out":4233,"would_cite":true,"duration_ms":651578,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T45"],"pacs":[],"model":"deepseek-v4-flash","headline":"Two-stage open-vocabulary segmentation improves when mask proposals read the text prompt.","keywords":["open-vocabulary segmentation","prompt-guided mask proposal","text-query cross-attention","two-stage segmentation","query-based transformer decoder","CLIP","zero-shot image segmentation","mask proposal recall"],"falsifier":"Run the trained PMP pipeline on a set of abstract and proprietary prompts (for example \"love,\" \"parking,\" \"Washington\") with human-annotated ground-truth masks; if first-stage recall or final mIOU is no better than the class-agnostic baseline on that set, the central claim of prompt-guided transfer fails.","tokens_in":19324,"feed_emoji":"🎯","tokens_out":4836,"duration_ms":42889,"temperature":0.7,"pith_summary":"This paper targets a blind spot in two-stage open-vocabulary segmentation: the first stage proposes masks without looking at the text prompt, so for arbitrary prompts the correct region may never appear. It introduces Prompt-guided Mask Proposal (PMP), a cross-attention added to transformer-based mask decoders that feeds text-token information into the learned query tokens before each decoding step. Wrapped around existing models, PMP raises mIOU by about 1-3 points on five benchmarks and, qualitatively, segments concepts like \"Yellowstone\" or \"love\" that class-agnostic proposals miss. The gain is meant to come from better first-stage recall rather than from changing the second-stage classifier.","feed_headline":"Prompt-aware masks lift open-vocabulary segmentation by 1-3 mIOU","feed_subtitle":"When mask proposals read the text prompt, even prompts like \"Yellowstone\" or \"love\" can be segmented.","key_machinery":"The load-bearing object is a text-query cross-attention block inserted before the standard cross-attention in the transformer decoder. Given query features $Q_l$, the text tokens are projected to keys and values $K_t, V_t$; the queries first attend to the text to produce $Q'_l = \\mathrm{softmax}(Q_l K_t^\\top)V_t$, and this prompt-conditioned query then attends to the image features. This is what makes the $N$ mask embeddings prompt-specific without changing the number of queries, and it can be dropped into Mask2Former-style decoders used by several existing two-stage models.","core_discovery":"The paper's central claim is that the missing-mask failure of two-stage open-vocabulary segmentation is fixable at the proposal stage: if the transformer decoder conditions its learned queries on the input text, the first-stage mask proposals become prompt-specific, and the frozen CLIP classifier can then retrieve regions that class-agnostic proposals never contained. On its own terms, this is a lightweight plug-in that yields absolute mIOU gains of roughly 1-3 points over OVSeg, FC-CLIP, SAN, and ODISE across ADE-847, PC-459, ADE-150, PC-59, and VOC, and enables qualitative segmentation of prompts such as \"Yellowstone\" and \"love.\"","pith_inferences":["Beyond the paper's experiments, the qualitative evidence suggests the mechanism transfers to prompts far outside COCO-Stuff vocabulary, but that transfer is not quantified; a natural next step is to build a benchmark of abstract and proprietary prompts with human-annotated masks and measure first-stage recall there.","Because the improvement concentrates in first-stage recall, pairing PMP with a better proposal-to-prompt matching or ranking step in the second stage could convert more of that recall into final mIOU than the current geometric-mean classifier does.","Training the same text-query cross-attention on a larger, more diverse set of captions and prompt pairs would likely strengthen the prompt-conditioning transfer, a testable extension the paper's framing suggests but does not run.","The cross-attention design is generic enough that it could be applied to other query-based architectures beyond the four baselines tested, though the paper only demonstrates it on those models."],"forward_implications":["Because PMP is a decoder-level change, any two-stage model built on Mask2Former can absorb it without touching its frozen CLIP classifier or retraining the second stage.","The reported first-stage recall gains are larger than the final mIOU gains (for example, OVSeg recall mIOU rises by 4.3-8.5 points across the five benchmarks), so improvements in proposal generation are the main driver of the final result.","Prompt-specific proposals make the pipeline usable for open-ended prompts such as proper nouns, adjectives, and full captions, not just the class names used in training.","The added cost is small: the paper reports roughly 0.01 seconds of extra first-stage latency per image while keeping the rest of the pipeline unchanged.","The same plug-in also improves panoptic segmentation on ADE20K and COCO when added to ODISE and FC-CLIP, so the benefit is not limited to semantic segmentation benchmarks."],"supporting_citations":[{"why":"Supplies the Mask2Former query-based transformer decoder and pixel-decoder machinery that PMP modifies.","marker":"[16]"},{"why":"Provides the CLIP text and image embeddings used for text-conditioned proposals and second-stage classification.","marker":"[46]"},{"why":"OVSeg is the primary two-stage baseline; PMP is attached to its Mask2Former first stage for most ablations.","marker":"[36]"},{"why":"FC-CLIP is a two-stage baseline whose mask decoder PMP replaces, giving some of the largest absolute gains.","marker":"[66]"},{"why":"SAN is a query-based open-vocabulary model whose queries PMP augments with text-guided cross-attention.","marker":"[62]"},{"why":"ODISE is a diffusion-feature two-stage baseline that PMP improves in both semantic and panoptic settings.","marker":"[59]"},{"why":"COCO-Stuff is the training dataset, supplying both class names and captions used as text prompts.","marker":"[4]"}],"fun_headline_variants":["Prompts steer mask proposals, adding 1-3 mIOU to open-vocab segmentation","PMP: text-conditioned masks lift two-stage open-vocab segmentation by 1-3 mIOU","When mask proposals read text, open-vocab segmentation adds 1-3 mIOU","Prompt-guided mask proposals give two-stage segmentation a 1-3 mIOU edge","Text-prompted masks boost open-vocab segmentation by 1-3 mIOU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that a decoder trained on COCO-Stuff class names and captions learns to condition masks on any CLIP text embedding, so the first-stage recall gains seen on standard benchmarks also hold for arbitrary test-time words like \"love\" or \"Times Square\"; the evidence for that transfer is qualitative only.","fun_headline_variants_meta":{"raw":{"variants":["Prompts steer mask proposals, adding 1-3 mIOU to open-vocab segmentation","PMP: text-conditioned masks lift two-stage open-vocab segmentation by 1-3 mIOU","When mask proposals read text, open-vocab segmentation adds 1-3 mIOU","Prompt-guided mask proposals give two-stage segmentation a 1-3 mIOU edge","Text-prompted masks boost open-vocab segmentation by 1-3 mIOU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001766,"raw_usage":{"total_tokens":6977,"prompt_tokens":963,"completion_tokens":6014,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":5890}},"tokens_in":579,"tokens_out":6014,"duration_ms":35874,"temperature":1.0,"reasoning_tokens":5890,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:58:40.246239+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained PMP pipeline on a set of abstract and proprietary prompts (for example \"love,\" \"parking,\" \"Washington\") with human-annotated ground-truth masks; if first-stage recall or final mIOU is no better than the class-agnostic baseline on that set, the central claim of prompt-guided transfer fails.","supporting_citations":[{"cited_title":"Masked-attention mask transformer for universal image segmentation","cited_arxiv_id":null,"evidence_quote":"Supplies the Mask2Former query-based transformer decoder and pixel-decoder machinery that PMP modifies."},{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"Provides the CLIP text and image embeddings used for text-conditioned proposals and second-stage classification."},{"cited_title":"Open-vocabulary semantic segmentation with mask-adapted clip","cited_arxiv_id":null,"evidence_quote":"OVSeg is the primary two-stage baseline; PMP is attached to its Mask2Former first stage for most ablations."},{"cited_title":"Side adapter network for open-vocabulary semantic segmentation","cited_arxiv_id":null,"evidence_quote":"SAN is a query-based open-vocabulary model whose queries PMP augments with text-guided cross-attention."},{"cited_title":"Open-vocabulary panop- tic segmentation with text-to-image diffusion models","cited_arxiv_id":null,"evidence_quote":"ODISE is a diffusion-feature two-stage baseline that PMP improves in both semantic and panoptic settings."},{"cited_title":"Coco- stuff: Thing and stuff classes in context","cited_arxiv_id":null,"evidence_quote":"COCO-Stuff is the training dataset, supplying both class names and captions used as text prompts."}],"review_version":1}