{"id":"82bcdd67-ef25-42e7-9316-a5fd2b57b0d7","arxiv_id":"2608.11681","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A synonym-aware training pipeline with GPT-based caption reconstruction improves COCO open-vocabulary segmentation, but only under a target-vocabulary-assisted protocol that leaks the novel class names.","lead":"This paper trains an open-vocabulary image segmenter using auto-generated masks, captions, and synonyms from pre-trained models. On COCO it beats older methods, but the protocol reveals the test class names during training, which weakens the open-vocabulary claim.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The controlled baseline CGG† may be confounded: Section 3.2 says CGG uses BERT while MCCF uses CLIP, and Section 4.1 does not state which text encoder CGG† uses, so the 6.5/6.8 AP gain could be a text-encoder effect rather than the proposed losses.","rationale":"The reader's verdict focused on the target-vocabulary-assisted pseudo-labeling protocol, which is an external-validity concern about what the benchmark measures. The concern raised here is internal to the paper's controlled comparison: the strongest claim is not merely that MCCF beats historical baselines, but that the proposed losses add 6.5/6.8 novel AP over identical pseudo-mask supervision. If CGG† differs from MCCF in the text encoder, the comparison is not controlled and the claim is unsupported. The paper's own statements in Section 3.2 make this a live possibility, and the definition of CGG† in Section 4.1 does not resolve it. Because a single unstated configuration detail determines whether the central claim holds, the appropriate verdict is UNVERDICTED pending clarification, rather than CONDITIONAL acceptance of the current evidence. This is a request for a missing control condition, not an accusation.","tokens_in":18075,"tokens_out":10735,"duration_ms":104103,"concrete_test":"Run CGG† with the CLIP text encoder while keeping the original CGG losses, the same Grounded-SAM pseudo-masks, and the same training schedule as MCCF; report constrained and generalized novel AP. If this baseline already reaches roughly 51/50 novel AP, the proposed losses add little beyond the CLIP encoder switch; if it remains near 45/43, the current comparison is confounded by the encoder change. Also state explicitly whether the Table 4 baseline uses BERT or CLIP.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim, quantified as 'Compared with CGG†, MCCF improves novel-class AP by 6.5 and 6.8' (Section 4.2, Table 2), depends on CGG† being a control that differs from MCCF only in the proposed training objectives. This is not established. Section 3.2 states that CGG 'employs BERT for textual embeddings' while MCCF 'use[s] the CLIP text encoder,' and MCCF's classification, grounding, and consistency losses all use CLIP text embeddings (Eqs. 2, 4, 5). Section 4.1 defines CGG† as CGG trained with the same Grounded-SAM pseudo-mask annotations 'while retaining the original CGG loss functions' but does not specify which text encoder CGG† uses. If CGG† retains BERT, the reported +6.5/+6.8 AP and the Table 4 ablation increments (45.1→49.5→50.9→51.6) combine the BERT→CLIP switch with the proposed losses; the first Table 4 row already replaces the grounding loss, which is defined using CLIP text embeddings, so the increment is not attributable to L_gr alone. The same ambiguity affects the CLIP-filtering ablation in Table 5. Thus, as written, the paper does not demonstrate that the proposed multimodal supervision, rather than a text-encoder change, produces the headline gain. If CGG† already uses CLIP, this concern is resolved, but the paper must state that explicitly.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MCCF, a Mask2Former-based framework for open-vocabulary instance segmentation (OVIS) and open-set panoptic segmentation (OSPS). It automatically generates pseudo segmentation masks with Grounded SAM using the target novel-class vocabulary as prompts, pseudo captions with LLaVA, and visually grounded synonyms with CLIP-based filtering. During training it adds three objectives to the CGG baseline: an extended grounding loss, a semantic consistency loss, and a GPT-based caption reconstruction loss. On COCO, the method reports large gains over CGG and a controlled baseline CGG† in OVIS, and gains over CGG in OSPS.","tokens_in":18411,"tokens_out":6937,"duration_ms":64539,"significance":"If the reported controlled comparison is valid, the paper provides evidence that synonym-aware grounding and caption reconstruction can add meaningful improvements beyond stronger pseudo-mask quality. The introduction of CGG† as a control is a good design choice, and the cumulative ablation in Table 4 is internally consistent. The CLIP-filtering ablation in Table 5 also clearly demonstrates the value of visual grounding for synonym selection. However, the target-vocabulary-assisted protocol means the experiments do not measure open-vocabulary generalization in the usual zero-shot sense, and the CGG† baseline is not fully specified. The paper is honest about the protocol in Section 4.1, but the framing of the contributions needs to be adjusted to match what is actually measured.","major_comments":[{"comment":"The text encoder used by CGG† is not specified. Section 3.2 states that CGG uses BERT while MCCF uses the CLIP text encoder, and the classification, grounding, and consistency losses (Eqs. 2, 4, 5) all operate on CLIP text embeddings. If CGG† retains BERT, the headline +6.5/+6.8 AP improvements in Table 2 and the ablation increments in Table 4 conflate the text-encoder change with the proposed losses. The authors must state explicitly which text encoder CGG† uses; if it is BERT, an additional control using the CLIP text encoder with the original CGG losses is required.","section":"Section 4.1 (CGG†) and Section 3.2"},{"comment":"The target novel-class names are used as text prompts for Grounded SAM to generate pseudo-masks, so the exact test vocabulary is injected into the training pipeline. The evaluation therefore does not assess generalization to categories never seen during training, and the abstract's claims about 'unseen object categories' are not supported by the experiments. The paper should reformulate the claims in terms of a target-vocabulary-assisted protocol, and ideally report an additional experiment in which pseudo-masks are generated without the target-vocabulary prompts (e.g., from captions only) to quantify the resulting drop in novel-class AP.","section":"Section 3.3.1 and Section 4.1 ('Pseudo-label vocabulary')"},{"comment":"The open-set panoptic segmentation comparison is not controlled like the OVIS comparison. CGG in Table 3 is the original method without the Grounded-SAM pseudo-mask annotations, so the unknown-class PQ gains (18.0, 11.5, and 7.3 points under the 20%, 10%, and 5% settings) conflate the effect of the pseudo-masks with the effect of the proposed losses. A CGG†-style baseline using the same pseudo-mask annotations should be reported for the OSPS settings to support the claim that the proposed objectives are responsible for the improvement.","section":"Table 3 (OSPS)"}],"minor_comments":[{"comment":"In Eq. (5), the notation f_i^T (t_i^nov - t_i^syn) is unclear about the dimension of f_i; if f_i is the decoder embedding for instance i, state this and clarify whether the loss is averaged over instances.","section":"Section 3.3.2, Eq. (5)"},{"comment":"The relationship between the GPT-2 tokenizer/embedding layer and the CLIP text encoder should be clarified: the classification and grounding losses use CLIP text embeddings, while the caption reconstruction loss appears to use GPT-2 embeddings; state how these two text representations are reconciled.","section":"Section 4.1 (Implementation Details)"},{"comment":"The column headers in Table 3 are confusing: 'PQTh' appears twice, and the superscripts 'Th' and 'St' are not consistently defined in the caption; the asterisk footnote is also difficult to parse.","section":"Table 3"},{"comment":"The average number of synonyms retained per class and the sensitivity of the results to the threshold τ=0.4 are not reported; this would help assess the stability of the filtering step.","section":"Section 3.3.1 (Synonym filtering)"},{"comment":"The title contains a stray space in 'Open-V ocabulary'; this should be corrected.","section":"Title"},{"comment":"The reference for Loshchilov (2019) is incomplete; it should list all authors and the full proceedings.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical study with a promising idea and a commendable attempt at a controlled baseline. The two main risks are (i) the unspecified text encoder in CGG†, which could invalidate the headline controlled comparison, and (ii) the target-vocabulary-assisted protocol, which limits the open-vocabulary interpretation. Both are fixable with additional experiments or clear restatements. If the authors provide a CLIP-encoder control and an honest reframing, the paper could be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. This is a solid incremental paper: the authors combine Grounded SAM pseudo-masks, LLaVA pseudo-captions, CLIP-filtered synonyms, and three extra training losses on top of CGG, and they get real-looking gains on COCO OVIS and OSPS. The controlled baseline CGG† is the right instinct — retraining CGG with the same pseudo-masks lets you separate pseudo-mask quality from the new objectives. That part of the design is genuinely good.\n\nThe problem is that the control is under-specified. Section 3.2 says CGG uses BERT and MCCF uses CLIP. Section 4.1 defines CGG† as CGG with the same Grounded-SAM masks but \"retaining the original CGG loss functions,\" and never says which text encoder it uses. If CGG† keeps BERT, then the headline +6.5/+6.8 novel AP gain — and every increment in Table 4 — bundles the BERT→CLIP switch together with the proposed losses. The first ablation row already replaces the grounding loss with L_gr from Eq. (4), which is built on CLIP embeddings. So the paper, as written, does not demonstrate that the new losses produce the gain. This is not minor; it is the load-bearing attribution claim.\n\nOther weaknesses are smaller. No error bars, no code, and the CLIP filtering threshold τ=0.4 is chosen empirically without a sweep. The target-vocabulary-assisted protocol is disclosed in the body and the conclusion, so that is honest, but the abstract calls it \"open-vocabulary\" without the caveat, which will mislead skimmers. The OSPS known-class trade-off is reported, which I appreciate.\n\nWhat the paper does well: the pipeline is clearly described, the ablations are internally coherent (each added loss nudges novel AP up), and they show failure cases plus inference-cost overhead. On the citation side, self-citations are present but mostly legitimate related work; no red flag there.\n\nWho is this for? Researchers using pseudo-label/teacher-student training for segmentation, especially anyone thinking about how much test vocabulary you can legitimately feed into training. The paper is worth discussing as a case study, but I would not yet trust the specific attribution claim.\n\nRecommendation: send it to peer review. The empirical work is extensive and the confound is fixable — a rerun of CGG† with the CLIP text encoder, or even a clear sentence pinning down the encoder, would settle it. But the current version should not be accepted as-is; the central comparison needs that control resolved.","headline":"A competent, incremental pseudo-label pipeline for open-vocabulary segmentation whose headline gain over its own control is not yet isolated: the control (CGG†) may have a different text encoder than the proposed method, so the +6.5/+6.8 AP claim could be a BERT-to-CLIP effect.","tokens_in":18980,"tokens_out":2995,"would_cite":false,"duration_ms":32180,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding CLIP-filtered synonyms, a semantic-consistency loss, and a masked-caption-reconstruction loss lifts novel-class mask average precision by 6.5 and 6.8 points over the same pseudo-mask supervision.","keywords":["open-vocabulary instance segmentation","open-set panoptic segmentation","multimodal pseudo-labeling","CLIP-guided synonym filtering","semantic consistency loss","caption reconstruction loss","target-vocabulary-assisted protocol","vision-language alignment"],"falsifier":"Train the proposed model with pseudo-masks generated from a vocabulary disjoint from the test classes, such as generic descriptions or a different held-out split, and evaluate on the original novel classes; if novel-class AP falls back to the CGG† level, the reported gains come from target-vocabulary assistance rather than from the three proposed losses.","tokens_in":17832,"feed_emoji":"🖼️","tokens_out":9508,"duration_ms":79962,"temperature":0.7,"pith_summary":"This paper tries to show that open-vocabulary instance and panoptic segmentation can be improved without extra manual annotations by supervising the model with multimodal pseudo-labels: pseudo masks, pseudo captions, and visually verified synonyms generated by pre-trained vision-language models. On the COCO benchmark under its target-vocabulary-assisted protocol, the method reports 51.6 novel-class AP in the constrained setting and 50.4 in the generalized setting, compared with 45.1 and 43.6 for CGG retrained with the same pseudo masks. The three added objectives, extended grounding with synonyms, semantic consistency, and masked caption reconstruction, account for the gap. If this is right, language-side supervision derived from captions and synonyms adds real segmentation ability on top of better pseudo-mask quality, at about 2.4% extra inference FLOPs.","feed_headline":"Three added training losses lift novel-class mask AP by 6.5 points","feed_subtitle":"Same pseudo-masks, new losses: open-vocabulary segmentation improves through captions and CLIP-verified synonyms.","key_machinery":"The core mechanism is joint training supervision built from three automatically generated signal types and three losses. Pseudo masks from a text-prompted segmenter give novel-class visual supervision; pseudo captions from a vision-language assistant add lexical diversity; CLIP text-image similarity with a top-1 ranking and a threshold of 0.4 filters candidate synonyms to those actually consistent with the masked region. The losses are an extended grounding loss $L_{gr}$ that matches region features to both novel-class names and synonyms, a semantic consistency loss that penalizes $(f_i^T(t_{nov} - t_{syn}))^2$ to pull embeddings of synonymous words together, and a generative caption reconstruction loss $L_{recon}$ that reconstructs masked caption tokens conditioned on image features. Together they make the model treat a class name and its paraphrases as the same visual concept. All auxiliary generators are removed at inference, leaving the segmenter plus CLIP text embeddings.","core_discovery":"The paper's central claim is that a multimodal pseudo-labeling pipeline together with three complementary training losses improves generalization to novel classes beyond what higher-quality pseudo masks alone provide. The pipeline uses a text-prompted segmenter to create pseudo masks for novel classes, a vision-language assistant to write descriptive pseudo captions, and CLIP similarity to keep only visually grounded synonyms. During training, the segmenter is pushed to align region features with both class names and synonyms, to keep synonym and class embeddings consistent, and to reconstruct masked caption tokens from visual features. In the controlled comparison against CGG retrained with the same pseudo masks, novel-class AP rises from 45.1 to 51.6 (constrained) and from 43.6 to 50.4 (generalized); in open-set panoptic segmentation, unknown-class PQ improves by 18.0, 11.5, and 7.3 points under the 20%, 10%, and 5% unknown settings, with a small drop on known classes.","pith_inferences":["A fair test of genuine open-vocabulary generalization would generate pseudo-masks with a vocabulary disjoint from the test classes; the paper's protocol supplies the exact test class names, so the absolute AP numbers partly credit that assistance.","The CLIP-filtering design is portable: any candidate word list could be validated against masked image regions the same way, so the technique could transfer to other weakly supervised segmentation pipelines.","The small known-class drop in panoptic settings suggests the method trades some base-class precision for unknown-class coverage, a trade that may be worth making in applications that value rare-object detection more than base-class exactness.","Because the caption reconstruction loss depends on masked-token prediction, richer or multi-view pseudo captions could plausibly increase the lexical diversity the model is exposed to, possibly enlarging the gain beyond what this paper reports."],"forward_implications":["Training with CLIP-verified synonyms should make the segmenter more tolerant of paraphrases at test time, reducing failures when a class is named in words other than the training label.","Because the auxiliary generators are removed at inference, the added capability costs only about 2.4% more GFLOPs and 7.9% more parameters, so the approach can be used where inference efficiency matters.","The method shifts accuracy toward novel and unknown classes at a small cost to known classes in panoptic settings, so users can expect better coverage of rare categories in exchange for slightly lower base-class PQ.","Removing CLIP-based synonym filtering drops constrained novel AP from 51.6 to 47.1, indicating that visual verification of candidate words is a load-bearing part of the pipeline."],"supporting_citations":[{"why":"Supplies the CGG baseline, its caption grounding and generation losses, and the controlled CGG† comparison with the same pseudo masks.","marker":"Wu et al., 2023"},{"why":"Provides Grounded SAM, the text-prompted segmenter that generates the novel-class pseudo masks.","marker":"Ren et al., 2024"},{"why":"Provides LLaVA, the vision-language assistant that generates the descriptive pseudo captions.","marker":"Liu et al., 2023"},{"why":"Provides CLIP embeddings used for the classification space and for filtering visually grounded synonyms.","marker":"Radford et al., 2021"},{"why":"Provides Mask2Former, the base architecture and mask losses on which the model is built.","marker":"Cheng et al., 2022"},{"why":"Provides Grounding DINO, the open-set detector inside Grounded SAM that produces the boxes refined into pseudo masks.","marker":"Liu et al., 2025a"}],"fun_headline_variants":["Multimodal pseudo-labels lift open-vocab mask AP by 6.5 points","CLIP, LLaVA, and Grounded SAM: +6.5 AP for unseen classes","Novel-class mask AP up 6.5 with multimodal pseudo-label pipeline","From pseudo-masks to pseudo-captions: open-vocab segmentation gains 6.5","Open-vocab segmentation: three losses + multimodal pseudo-labels = +6.5 AP"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes it is acceptable to tell the pseudo-mask generator the exact names of the classes that will be tested, so the reported gains reflect help from the target vocabulary rather than purely never-seen-category generalization.","fun_headline_variants_meta":{"raw":{"variants":["Multimodal pseudo-labels lift open-vocab mask AP by 6.5 points","CLIP, LLaVA, and Grounded SAM: +6.5 AP for unseen classes","Novel-class mask AP up 6.5 with multimodal pseudo-label pipeline","From pseudo-masks to pseudo-captions: open-vocab segmentation gains 6.5","Open-vocab segmentation: three losses + multimodal pseudo-labels = +6.5 AP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001209,"raw_usage":{"total_tokens":4985,"prompt_tokens":960,"completion_tokens":4025,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":3906}},"tokens_in":576,"tokens_out":4025,"duration_ms":27397,"temperature":1.0,"reasoning_tokens":3906,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:30:58.500401+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the proposed model with pseudo-masks generated from a vocabulary disjoint from the test classes, such as generic descriptions or a different held-out split, and evaluate on the original novel classes; if novel-class AP falls back to the CGG† level, the reported gains come from target-vocabulary assistance rather than from the three proposed losses.","supporting_citations":[],"review_version":1}