{"id":"28f9e308-92cb-4354-a05b-773c6db6740e","arxiv_id":"2412.00153","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"ROSE uses patch-wise perception in a large multimodal model to predict dense masks and generate open-set category names without predefined prompts.","lead":"A new segmentation model, ROSE, treats every image patch as a candidate region so a large multimodal model can predict dense masks and generate category names without predefined category prompts. The paper reports competitive results on three benchmarks, but the refinement step that produces the highest scores is selected using ground-truth labels.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core open-set claim is not evaluated: ROSE's category generation is tested only on closed ADE/COCO vocabularies, with the paper's own Limitation admitting no open-set benchmark; until a held-out-category evaluation is provided, the central 'eliminates predefined categories' claim is unverified.","rationale":"The reader's oracle-guided CSR critique is valid and should be addressed; however, I do not think it is the most load-bearing issue because CSR is an enhancement and the direct ROSE numbers already support 'competitive' performance. The condition that is least secure for the central claim is the open-set capability: the method is never quantitatively evaluated on categories outside the training vocabulary, and the paper's own Limitation says so. The evaluation protocol compares generated names against the closed ADE/COCO label sets used in training, so high scores do not demonstrate open-set generalization. I therefore recommend keeping the CONDITIONAL verdict while adding a required held-out-category or open-set benchmark. The oracle issue and the §4.2/Table 2 numeric inconsistencies (43.2 vs 51.0 for ADE mIoU; 51.6 vs 57.4 for +CSR; 34.4 vs 36.3 for COCO mAP) should also be reconciled, but they do not change the conditional status.","tokens_in":16792,"tokens_out":6178,"duration_ms":58796,"concrete_test":"Run a held-out-category evaluation: train ROSE on the current data, then evaluate on COCO/ADE categories excluded from training (or on an open-vocabulary segmentation benchmark such as OVSeg or COCO-O), and report mask mIoU plus exact-match and semantic-similarity category accuracy for the generated names. If generated categories are not correct for unseen classes, the open-set claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central contribution is free-vocabulary segmentation: given only a task-level instruction, ROSE should generate category names and masks without a predefined candidate set. The quantitative evaluation never tests this. Semantic segmentation is evaluated on ADE-20k val using the ADE vocabulary; instance segmentation is evaluated on COCO val using COCO categories; referring segmentation provides the description. ADE-20k and COCO are in the training set per §4.1, so the generated categories are being compared against exactly the label sets seen during training. The only evidence for open-set behavior is qualitative (Fig. 3, Fig. 9), and the paper's Limitation states: 'lacking a comprehensive benchmark limits our ability to fully evaluate model performance across diverse open-set scenarios.' Furthermore, the category-generation mechanism (Sec. 3.3) produces noun concepts through an LLM instruction, but no metric measures whether the generated noun is semantically valid or matches a true unseen category. The oracle-guided CSR issue in Supplementary A.1 is a real separate concern about the magnitude of the +CSR gains, but even if CSR is fixed or removed, the open-set capability remains unsupported by quantitative evidence. For the central claim to hold, ROSE must be shown to segment and name objects outside its training vocabulary; that condition is the least secure part of the paper's argument.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ROSE, a large multimodal model for dense open-set segmentation. The method treats each image patch as an independent region of interest and predicts an objectness score, a mask embedding, and a category embedding per patch. Dense masks are decoded via SAM, while category names are generated by the LLM through a custom instruction-response paradigm, with no explicit candidate category set. A conversation-based segmentation refinement (CSR) mechanism is added to iteratively improve masks and categories. Experiments are reported on ADE-20k semantic segmentation, COCO instance segmentation, and RefCOCO/+/g referring segmentation, together with ablations of the segmentation framework, super-patch design, LoRA parameters, and refinement mechanisms.","tokens_in":17074,"tokens_out":6992,"duration_ms":56011,"significance":"If the claims were fully validated, the patch-wise perception design could be a useful building block for dense segmentation with LMMs, and the idea of decoding category embeddings into noun concepts through the LLM is a distinctive alternative to similarity-based open-vocabulary classification. The paper provides a clear architecture description, a reasonable set of ablations, and an explicit limitation statement. However, the significance is substantially reduced by three issues: the open-set capability is never quantitatively evaluated on categories outside the training vocabulary; the reported +CSR gains are obtained with ground-truth-guided selection of refinement targets; and Section 4.2 contains direct numerical contradictions with Table 2. These issues leave the central claims unverified in their current form.","major_comments":[{"comment":"The central claim that ROSE performs open-set, free-vocabulary segmentation is not evaluated. The only quantitative evaluations are on ADE-20k val, COCO val, and RefCOCO/+/g, and all of these datasets (with their label vocabularies) are used in training per §4.1. There is no held-out-category experiment: the model never has to segment or name a category that was not seen during training. The paper's own Limitation admits that 'lacking a comprehensive benchmark limits our ability to fully evaluate model performance across diverse open-set scenarios.' As a result, the abstract's claim of 'category prediction independent of closed-set constraints or predefined categories' is not supported by any measured evidence. I recommend adding a quantitative zero-shot protocol, e.g., training on a subset of categories and evaluating on disjoint held-out categories, with a metric that matches generated category names to ground-truth labels.","section":"§4.1, §4.2, §A.4, Limitation"},{"comment":"The +CSR improvements for semantic and instance segmentation rely on ground-truth information to select which predictions to refine. For semantic segmentation, the five categories with the highest Union-minus-Intersection are chosen from a confusion matrix computed against the ground truth; for instance segmentation, the ten predictions with IoU below 50 are selected using an IoU matrix against ground truth (Supplementary Algorithm 1 and the accompanying text). In deployment the model must decide autonomously when and what to refine, so these numbers describe an oracle-guided upper bound, not the autonomous refinement capability claimed in §3.4. Please either evaluate with model-driven selection (e.g., based on predicted confidence) or clearly re-label the +CSR results as oracle-refined.","section":"Supplementary A.1, §4.2, Table 2"},{"comment":"There are direct numerical contradictions between the prose and Table 2. The text reports 43.2 mIoU for ROSE and 51.6 mIoU with CSR on ADE-20k, while Table 2 lists 51.0 and 57.4; for COCO instance segmentation the text reports 34.4 mAP while Table 2 lists 36.3; and the text claims CSR 'achieves SOTA' on semantic segmentation, but Mask2former's 57.7 exceeds ROSE+CSR's 57.4. Since the text and table cannot both be correct, the reported results are internally inconsistent. The authors must reconcile these numbers and indicate which are authoritative.","section":"§4.2 vs Table 2"},{"comment":"The open-category generation mechanism is not quantitatively evaluated. The instruction-response paradigm decodes the category embedding E_cat into a noun phrase, but no metric measures whether a generated noun is semantically valid or correctly matches a novel category. The only quantitative semantic-segmentation evaluation stacks predictions into a fixed dataset-specific N-channel output (§A.4), which requires matching generated names to the known vocabulary. Thus the 'category prediction independent of closed-set constraints' claim rests entirely on qualitative examples (Fig. 3, Fig. 9). Please provide an evaluation that measures the accuracy and quality of generated category names on categories not seen during training.","section":"§3.3, §A.4"}],"minor_comments":[{"comment":"The LoRA ablation is trained with 20% of the iterations of the main model, so the reported ADE-20k and COCO numbers are not directly comparable to the main results; please state whether the observed trend holds with full training or present the comparison on a normalized basis.","section":"§4.3, Table 5"},{"comment":"The patch size p is never specified; only '482 predicting patches' is given. Since S = floor(L/p), the number of patches should be a perfect square or otherwise derivable; please clarify how 482 is obtained, especially in relation to the super-patch design.","section":"§4.1"},{"comment":"The super-patch role assignment is introduced but it is not clear how the role is incorporated into the patch analyzer; Fig. 5 in the supplement shows the spatial arrangement but not how the role labels are encoded or used in the loss.","section":"§3.2"},{"comment":"The claim of 'multiple SOTAs in refcoco/+/g datasets' is stronger than Table 2 supports: on RefCOCO testA, GLaMM-7B exceeds ROSE, and on RefCOCOg test, GSVA-13B exceeds ROSE. Please qualify the claim.","section":"§4.2"},{"comment":"The notation y_txt is used both for the overall text-generation output in Eq. (1) and for the counting task in §3.5, while y_cat denotes the category generation; this overlapping notation is confusing and should be separated.","section":"Eq. (1) and §3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's title and abstract overstate the open-set contribution relative to what is evaluated. The lack of a held-out-category experiment and the oracle-based refinement selection are the two main scientific concerns; both are addressable in a revision, so I do not recommend rejection. The numerical inconsistencies in §4.2 are a separate author responsibility and should be corrected before any further consideration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: ROSE is a genuine system-level novelty — patch-wise RoI prediction with LLM-generated category names and SAM decoding, without a predefined candidate list. The ablation in Table 3 makes a real case that patch-wise mask-embedding decoding beats stacking <SEG> tokens for dense prediction. The unified results across semantic, instance, and referring segmentation are competitive, especially on referring. The paper deserves a serious referee.\n\nBut the central claim is not actually tested. The paper promises open-set category generation, yet every quantitative experiment uses ADE-20k and COCO, whose label sets are in the training data per Sec 4.1. The model is never asked to segment and name a category it has not seen. The only evidence is qualitative (Fig 3, Fig 9), and the Limitation section admits there is no open-set benchmark. That is not a minor omission; it is the paper's headline. Without a held-out-category evaluation, 'eliminates predefined categories' is an assertion, not a result.\n\nSecond, the CSR numbers are inflated by oracle selection. Supplementary A.1 shows the refinement targets are chosen using ground-truth confusion matrices and IoU. The +CSR columns in Table 2 therefore show an upper bound, not what the model does autonomously. The prose claims 'refine the five worst categories' — that is only possible with a GT oracle. If CSR must self-initiate, these numbers do not transfer.\n\nThird, the paper is sloppy with its own numbers. Section 4.2 says ROSE gets 43.2 mIoU on ADE-20k and 34.4 mAP on COCO, but Table 2 lists 51.0 and 36.3. The CSR gains in prose (8.6 mIoU, 2.0 mAP) don't match the table (6.4, 2.8). This looks like an earlier checkpoint or a typo, but it undermines trust.\n\nWhat holds up: the core architecture is sensible, the ablation supports the design choices, and the referring segmentation results are solid. The self-citation of PixelLM and GKC is not problematic per se. Code is promised but not released; no error bars anywhere.\n\nVerdict: conditionally acceptable if the authors (1) evaluate on genuinely unseen categories, (2) re-run CSR without oracle selection or clearly label it as an upper bound, and (3) reconcile the table/prose numbers. As it stands, I would not cite the open-set claim, but I would send it to a careful referee.","headline":"A plausible patch-wise dense segmentation LMM whose headline open-set claim and CSR gains are undercut by the evaluation design; worth refereeing, but needs a real open-set benchmark and an honest refinement protocol.","tokens_in":17622,"tokens_out":2760,"would_cite":false,"duration_ms":22925,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that ROSE eliminates predefined category inputs and performs dense segmentation by generating category names from patch-level embeddings.","keywords":["ROSE","open-set segmentation","dense prediction","large multimodal model","patch-wise perception","category generation","segmentation refinement"],"falsifier":"Run the full ROSE plus refinement pipeline with the model itself choosing which masks to refine, using its own objectness scores or predicted-category confidence, and compare the resulting ADE-20k mIoU and COCO mAP with Table 2. If the gains disappear when ground-truth confusion and IoU are not used to pick refinement targets, the refinement claim as an autonomous capability is falsified.","tokens_in":16585,"feed_emoji":"🧩","tokens_out":6429,"duration_ms":55795,"temperature":0.7,"pith_summary":"ROSE's claim is that dense open-set segmentation can be freed from predefined category prompts. Instead of matching masks against a fixed list of class names, the paper proposes a large multimodal model that treats every image patch as a potential object, scores it for object presence, decodes a mask from a learned embedding, and lets the language model generate a category name in words. The same framework reports results on semantic, instance, and referring segmentation, with a conversation-based refinement step that improves masks and labels by revisiting earlier predictions. This matters because current open-vocabulary segmenters only select from categories someone has already supplied; ROSE claims to be a genuine generator of both masks and category names.","feed_headline":"Dense segmentation without any predefined category list","feed_subtitle":"Patch-wise perception lets a multimodal model generate both masks and category names in one pass.","key_machinery":"The load-bearing mechanism is the patch-wise perception process. The image is divided into non-overlapping patches; for each patch, three MLP heads read the language model's object-level embedding and output an objectness score, a mask embedding, and a category embedding. At inference, objectness scores filter patches to a top-N set; mask embeddings feed a frozen promptable mask decoder to create dense masks; category embeddings are re-inserted into the language model through a fixed instruction such as 'decode this linguistic embedding into a noun concept,' so category names are generated as words. A 3x3 super-patch grouping assigns detecting roles for small, medium, large, and stuff regions, improving instance-scale behavior. A second stage, conversation-based segmentation refinement, concatenates the image with the previous mask and a corrective instruction, then generates a revised mask and category.","core_discovery":"The central discovery, as the paper states it, is that patch-wise perception makes dense prediction stable in a segmentation large multimodal model. Where stacking many special segmentation tokens causes the model to lose local image detail, assigning each image patch its own objectness score, mask embedding, and category embedding keeps each prediction local and lets a frozen promptable mask decoder produce the final mask. The category embedding is then treated as a linguistic token inserted into a fixed instruction template, so the language model outputs the category name autoregressively without any candidate category list at inference. The paper also introduces a conversation-based refinement paradigm that concatenates the previous mask with the image and a textual correction prompt, reporting that this raises semantic segmentation from 43.2 to 51.6 mIoU on the evaluated protocol and improves referring segmentation by roughly 12.6 points on average.","pith_inferences":["We infer that if ROSE's category generation works without a fixed vocabulary, the same category-embedding prompt could be reused for open-vocabulary detection and retrieval without retraining.","We infer that the CSR gains in Table 2 are upper bounds for a self-refining system, since a deployed version would need a confidence-based trigger to select refinement targets on its own.","We infer that granularity failures such as 'plant' versus 'tree' point to a missing evaluation axis: a proper open-set benchmark would need to count near-synonym and hierarchy errors, not just exact-label IoU.","We infer that the fixed 3x3 super-patch role assignment is one design point and could be learned end-to-end rather than fixed by scale and type."],"forward_implications":["A single model can produce dense and sparse masks from a task-level prompt without a predefined category list, so open-set segmentation no longer requires a separate vocabulary selector.","Category names are generated as language, allowing the model to label objects it has not been explicitly trained to name, at least within the language model's vocabulary.","A second conversational round that receives the previous mask and a correction prompt can improve boundaries and categories, making segmentation an interactive process.","Patch-level objectness scores provide confidence values, letting the same framework output ranked instance predictions rather than only semantic classes.","Because the model is trained on semantic, instance, and referring data together, the paper claims competitive performance in all three tasks within one set of weights."],"supporting_citations":[{"why":"Supplies the pretrained vision-language backbone that ROSE fine-tunes with LoRA.","marker":"[39]"},{"why":"Supplies the embedding-to-mask decoding approach that ROSE extends from sparse to patch-wise dense prediction.","marker":"[38]"},{"why":"Supplies the frozen promptable mask decoder that turns mask embeddings into final masks.","marker":"[36]"},{"why":"Supplies the grid-as-prediction-unit idea behind treating each image patch as a region of interest.","marker":"[72]"},{"why":"Supplies the multi-scale super-patch strategy that ROSE adapts for instance segmentation.","marker":"[63]"},{"why":"Supplies the text-embedding space that the category embeddings are aligned to.","marker":"[85]"},{"why":"Supplies the method for generating defective masks from ground-truth masks to train the refinement stage.","marker":"[14]"}],"fun_headline_variants":["Patch-wise LMM does dense masks and open categories","One LMM, any category name, dense masks","ROSE: dense segmentation that names its own categories","Open-set segmentation without category prompts","Patch-wise perception unlocks open-set segmentation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The performance gains attributed to conversation-based refinement assume the model can decide which predictions to refine, but in the experiments ground-truth confusion and IoU select those targets, so autonomous selection is never demonstrated.","fun_headline_variants_meta":{"raw":{"variants":["Patch-wise LMM does dense masks and open categories","One LMM, any category name, dense masks","ROSE: dense segmentation that names its own categories","Open-set segmentation without category prompts","Patch-wise perception unlocks open-set segmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1239,"prompt_tokens":889,"completion_tokens":350,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":281}},"tokens_in":505,"tokens_out":350,"duration_ms":3478,"temperature":1.0,"reasoning_tokens":281,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:06:33.107429+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full ROSE plus refinement pipeline with the model itself choosing which masks to refine, using its own objectness scores or predicted-category confidence, and compare the resulting ADE-20k mIoU and COCO mAP with Table 2. If the gains disappear when ground-truth confusion and IoU are not used to pick refinement targets, the refinement claim as an autonomous capability is falsified.","supporting_citations":[{"cited_title":"Segment any- thing","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen promptable mask decoder that turns mask embeddings into final masks."},{"cited_title":"SOLO: Segmenting objects by locations","cited_arxiv_id":null,"evidence_quote":"Supplies the grid-as-prediction-unit idea behind treating each image patch as a region of interest."},{"cited_title":"Pixellm: Pixel reasoning with large multimodal model","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-scale super-patch strategy that ROSE adapts for instance segmentation."}],"review_version":1}