{"id":"78702eba-d6ba-4580-821f-f545ffdd1abb","arxiv_id":"2507.10118","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"DEARLi improves semi-supervised panoptic segmentation by fusing CLIP zero-shot mask classification with a SAM-based decoder warm-up, reaching 29.9 PQ on ADE20K with 158 labeled images.","lead":"The paper introduces DEARLi, a semi-supervised panoptic segmentation method that combines CLIP zero-shot classification with SAM-based decoder warm-up. It reports state-of-the-art results on ADE20K with only 158 labeled images, beating prior semi-supervised segmentation methods by large margins.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's unverified '8x less GPU memory' claim, which contradicts the conclusion's '8x fewer GPUs', is a load-bearing and falsifiable assertion without any controlled measurement in the paper.","rationale":"The reader's weakest_assumption concerned the robustness of CLIP zero-shot ensembling to poor mask candidates. This concern is explicitly identified and mitigated by the paper itself (Section 3.3) and is empirically supported by ablations (Table 7), the oracle-mask validation (Figure 3), and gains on underrepresented classes (Figure 5). The paper even acknowledges the residual risk in Section E of the supplement. I therefore do not find this the most load-bearing issue. Instead, the strongest threat to the central claim as stated in the abstract is the unsupported '8x less GPU memory' assertion. The paper's conclusion switches to '8x fewer GPUs,' indicating a possible conflation of GPU count with memory. No controlled measurement is reported, and the only memory-related statement (Section 4.1) says backbone fine-tuning roughly doubles memory, which does not imply an 8x reduction. This is a concrete, falsifiable claim that can be checked directly, and if false or unverifiable it would materially misrepresent the method's practical advantage. Since the reader already gave a CONDITIONAL verdict, my concern reinforces that condition without moving the verdict; the authors should either provide the measurement or correct the abstract.","tokens_in":23900,"tokens_out":12710,"duration_ms":138622,"concrete_test":"Run a controlled memory benchmark: train DEARLi and SemiVL on the same GPU (A100-40GB) with identical batch size (8 labeled + 8 unlabeled), crop resolution (e.g., 640x640 for ADE20K), number of iterations, and framework; measure peak GPU memory via torch.cuda.max_memory_allocated. Also record the number of GPUs used in the original SemiVL experiments. If the memory ratio is not approximately 8x, or if the 8x figure reflects GPU count rather than memory, the abstract's claim must be rewritten to state the measured quantity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim includes 'requiring 8x less GPU memory,' but the paper never reports a memory measurement. Section 4 states only that all frozen-backbone experiments run on a single A100-40GB GPU, and Section 4.1 notes backbone fine-tuning 'roughly doubles' memory. The conclusion instead says '8x fewer GPUs,' which is a different quantity (GPU count, not memory). No controlled comparison of peak GPU memory or GPU count between DEARLi and SemiVL is provided, and the methods differ in batch size, resolution, and backbone training strategy. If the 8x figure derives from using 1 GPU versus 8 GPUs, the abstract's phrasing is misleading; if it derives from memory, the evidence is absent. Because this claim is part of the headline result, it must be either substantiated by a controlled measurement or corrected before the abstract can be accepted as written.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DEARLi, a three-stage semi-supervised panoptic segmentation method built on a frozen CLIP (ConvNeXt) backbone and a Mask2Former decoder. Recognition is enhanced by ensembling the mask-transformer's mask-wide posteriors with zero-shot, mask-pooled CLIP posteriors, while localization is enhanced by a class-agnostic decoder warm-up stage using SAM pseudo-labels. Training then proceeds with a Mean Teacher consistency loss on labeled and unlabeled images. The method is evaluated on ADE20K, COCO-Panoptic, and COCO-Objects across low-label regimes, reporting consistent improvements over baselines and large margins over the semi-supervised semantic segmentation state of the art (SemiVL). The supplement provides additional ablations, three-seed variance, and a limitations discussion.","tokens_in":24017,"tokens_out":4947,"duration_ms":52076,"significance":"The paper is a well-executed empirical study of foundation-model exploitation for low-label panoptic segmentation. It introduces a sensible decoupling of recognition and localization enhancement, validates each component with detailed ablations, and provides reproducible code. The reported gains are substantial and the three-seed variance in the supplement is a strength. However, the headline efficiency claim ('8x less GPU memory' in the abstract, '8x fewer GPUs' in the conclusion) is not measured anywhere in the paper, and the headline semantic-segmentation comparison partly relies on the SAM-augmented DEARLi variant, which the authors themselves admit may be an unfair comparison with SemiVL. These issues affect the abstract's central claims and need correction, although the core technical contributions and the main empirical results remain defensible.","major_comments":[{"comment":"The claim of an 8x reduction in GPU memory or GPU count is not supported by any measurement in the manuscript. Section 4 states only that all frozen-backbone experiments run on a single A100-40GB GPU, and Section 4.1 notes that backbone fine-tuning 'roughly doubles' memory. No controlled comparison of peak GPU memory or GPU count between DEARLi and SemiVL is reported, and the methods differ in batch size, resolution, and backbone training strategy. The conclusion's phrasing '8x fewer GPUs' is a different quantity from the abstract's '8x less GPU memory.' This is a load-bearing headline claim and must be either substantiated with a controlled measurement or corrected/removed before the abstract can be accepted as written.","section":"Abstract and Conclusion; Section 4.1"},{"comment":"The abstract's headline semantic-segmentation number (38.9 mIoU on ADE20K with 158 labeled images) belongs to DEARLi, which includes the class-agnostic SAM warm-up. In Section 4.2 the authors state that comparing DEARLi with SemiVL 'might be unfair' because DEARLi distills knowledge from SAM, and they present DEAR as the fair comparison (36.5 mIoU on the same partition). The abstract and the 'outperforms the state of the art' phrasing do not carry this qualification. Please either report DEAR as the primary comparison or explicitly note in the abstract that the cited number includes the SAM-augmented variant.","section":"Abstract and Section 4.2 (Tables 3 and 5)"},{"comment":"The comparison with the state of the art in semantic segmentation uses panoptic-trained models evaluated with semantic inference. The authors provide a supporting control (Supplementary Table 12) showing that retraining the same model on semantic labels yields roughly comparable mIoU. This control is valuable and largely mitigates concern about objective mismatch. However, the main text should point to this control more explicitly, as the current text in Section 4.2 only says the comparison is 'relevant and fair' based on the typical behavior of Mask2Former; the control is the actual evidence for that statement.","section":"Section 4.2 (Tables 3 and 5) and Supplementary Table 12"}],"minor_comments":[{"comment":"The heading 'Semantic segmentaiton ablations' contains a typo ('Segmentaiton'); please correct it.","section":"Supplementary Section A (heading)"},{"comment":"SemiVL [22] is cited as 'ECCV'24' in the tables and text, but the bibliography entry lists the year 2025; please make the year consistent.","section":"References and Tables 3, 5, 15, 16"},{"comment":"The sentence 'preliminary experiments show -4.8 p.p. PQ on 1/128' reports a result without any details or reference; please include the setup in the supplement or a footnote so the claim can be checked.","section":"Section 4.1"},{"comment":"The manuscript uses both 'pseudo-labels' and 'pseudolabels' (e.g., 'pseudo-labels generation' and 'SAM pseudo-labels'); please unify the spelling.","section":"Throughout"},{"comment":"The caption of Table 3 says improvements over SemiVL are in green, but the textual version of the paper cannot convey color; please ensure the published PDF uses accessible color coding and that the caption explains any color encoding used.","section":"Table 3 and Supplementary Figure captions"},{"comment":"The limitation discussion on the manually set alpha is good, but it appears only in the supplementary material; consider adding a brief version to the main paper, as the appropriate range of alpha is relevant to practical use.","section":"Appendix C (Limitations)"}],"recommendation":"major_revision","confidential_remarks":"This is a strong empirical paper with a clear ablation structure and an honest limitations discussion. The main reason for major revision is the unmeasured and internally inconsistent efficiency claim ('8x less GPU memory' vs. '8x fewer GPUs') that appears in the abstract and conclusion, and the need to qualify the semantic-segmentation comparison when the headline number comes from the SAM-augmented DEARLi variant. Both issues are fixable with textual corrections; the underlying method and the core experimental results appear sound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: DEARLi is a solid empirical paper on semi-supervised panoptic segmentation. The main claim — that you can get large gains in low-label regimes by using CLIP for recognition and SAM for localization, separately, inside a mask-transformer Mean Teacher loop — holds up well in the experiments. What does not hold up is the abstract's \"8x less GPU memory\" claim. I checked: the paper never reports a memory measurement. Section 4 says all frozen-backbone runs fit on one A100-40GB, and fine-tuning the backbone roughly doubles memory. The conclusion says \"8x fewer GPUs,\" which is a different quantity and also unmeasured. The stress-test note is right, and the fix is easy: measure peak GPU memory against SemiVL in a controlled setting, or drop the claim from the abstract.\n\nWhat is new: the combination of mask-pooled CLIP zero-shot posteriors geometrically ensembled with the mask transformer's posteriors for pseudo-label generation, plus a class-agnostic decoder warm-up on SAM masks. That specific pipeline is not in the cited literature. The paper is honest that ensembling and warm-up borrow from FC-CLIP and Mean Teacher, but the decoupling plus the SAM warm-up is a real recipe, not a trivial hookup.\n\nWhat is done well: the ablations are thorough and the story is coherent. The three-seed variance in the supplement is modest, which backs the headline numbers. They also ran SemiVL with a ConvNeXt backbone themselves to make the comparison less apples-to-oranges, and they flag that DEARLi's SAM warm-up makes the SemiVL comparison potentially unfair. DEAR without SAM still beats SemiVL, so the central empirical claim does not depend on that ambiguity. The limitation section admits alpha is hand-set and suggests adaptive tuning. No circularity: CLIP is an external signal and the ensembling weight is ablated.\n\nSoft spots, in proportion: the memory claim is the main one, and it is in the abstract, so it matters. There are single-run comparison tables, but the seed study and consistent trends across five data splits largely mitigate that. The mask-quality assumption behind ensembling is real, but the SAM warm-up is explicitly designed to strengthen masks, so I would not call it a weak point.\n\nFor whom: anyone working on SSL segmentation or on distilling foundation models into mask transformers will get value. It deserves a serious referee. I would send it to review, with a request to substantiate or remove the memory claim.","headline":"Strong empirical SSL segmentation paper whose central gains are credible, but the abstract's unmeasured '8x less GPU memory' claim should be fixed before this is taken at face value.","tokens_in":24614,"tokens_out":2297,"would_cite":true,"duration_ms":25387,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DEARLi claims a semi-supervised panoptic method that reaches 29.9 PQ and 38.9 mIoU on ADE20K with only 158 labeled images, beating both previous semi-supervised methods and supervised models trained on four times more data.","keywords":["semi-supervised panoptic segmentation","mask transformers","CLIP zero-shot recognition","SAM pseudo-labels","geometric ensembling","decoder warm-up","mean-teacher consistency","low-annotation segmentation"],"falsifier":"Train DEARLi on ADE20K 1/128 and degrade only the mask proposals used for CLIP mask pooling, for example by jittering their boundaries or randomly dropping low-recall masks while keeping CLIP features and the rest of the pipeline identical; if panoptic quality does not drop in step with the degradation, the claimed recognition mechanism is not what carries the gain.","tokens_in":23678,"feed_emoji":"🖼️","tokens_out":6862,"duration_ms":70942,"temperature":0.7,"pith_summary":"Panoptic segmentation normally needs dense pixel labels, which cost over an hour per image. This paper argues that two cheap, pretrained foundation models can supply most of that signal in semi-supervised training, provided each is aimed at a separate weakness: CLIP, a vision-language model, contributes recognition by zero-shot naming of mask regions, while SAM, a class-agnostic segmentation model, contributes localization by pre-training the decoder on region boundaries. The method wraps both signals around a mask transformer trained with mean-teacher consistency on labeled and unlabeled images. On ADE20K with only 158 labeled images it reports 29.9 PQ and 38.9 mIoU, exceeding the previous semi-supervised semantic segmentation state of the art by a wide margin while using 8 times less GPU memory and beating supervised models trained on four times more labels. The significance is that very low-label, high-taxonomy panoptic setups stop being annotation-bound.","feed_headline":"Panoptic segmentation hits 29.9 PQ with 158 labeled images","feed_subtitle":"A frozen CLIP model fixes recognition, SAM warms up the decoder, and the pair beats supervised models on 4x more data.","key_machinery":"The load-bearing mechanism is the weighted geometric ensembling of mask-transformer posteriors $P_{\\mathrm{MT}}$ with mask-pooled CLIP zero-shot posteriors $P_{\\mathrm{CLIP}}$ via $P_{\\mathrm{ENS}} = (P_{\\mathrm{MT}})^\\alpha \\odot (P_{\\mathrm{CLIP}})^{1-\\alpha}$ with $\\alpha = 0.6$, where mask pooling averages frozen CLIP features over each thresholded candidate mask. Around this sits the class-agnostic decoder warm-up (DeWa): before semi-supervised training, the mask-transformer decoder is optimized only on localization losses against SAM pseudo-labels, so the segmentation heads start with object-boundary knowledge that the few labeled images cannot provide. The two mechanisms are orthogonal, which is why their gains add.","core_discovery":"The paper's central claim is that recognition and localization should be enhanced separately and then recombined in a mask transformer. For recognition, the teacher's mask-wide class probabilities are combined with zero-shot CLIP predictions obtained by averaging frozen CLIP image features inside each candidate mask; this geometric ensembling biases pseudo-labels toward classes that are rare in the tiny labeled set. For localization, the decoder is first warmed up by training only its segmentation losses against class-agnostic SAM masks, avoiding the granularity mismatch between SAM regions and dataset taxonomy, and then fine-tuned in the semi-supervised stage. The authors report that this decoupled pipeline yields panoptic quality that no semi-supervised panoptic method has previously reached at these label budgets, and that it transfers to semantic evaluation without retraining.","pith_inferences":["Because the recognition signal is a zero-shot posterior over arbitrary class names, the trained decoder could probably be re-pointed at new taxonomies at inference time without retraining; the paper does not test this.","The paper's own ablation shows higher $\\alpha$ helps as labeled data grows, so an adaptive ensembling weight that rises during training would likely squeeze out additional gains; the authors list this as a limitation.","Decoder warm-up distills only objectness, not class identity, so the same recipe may transfer to other mask-based dense prediction tasks such as instance segmentation; this is an untested extension.","The qualitative failure where house doors and windows are split into separate segments suggests CLIP's labeling policy can diverge from the dataset's, so gains on long-tail classes may need dataset-specific prompt engineering."],"forward_implications":["Semi-supervised panoptic models can surpass fully supervised models trained on four times more labels at the lowest annotation budgets.","Freezing the CLIP backbone removes backpropagation through the feature extractor, which is what keeps training on a single GPU with roughly eight times less memory than the previous CLIP-based semi-supervised method.","The gains are concentrated in long-tail classes, so the method directly attacks the failure mode that makes large class taxonomies hard under label scarcity.","One panoptically trained model also serves as a semantic segmentation model, since the same masks can be scored with semantic inference without retraining.","Class-agnostic decoder warm-up improves the mask transformer even when the SAM pseudo-labels come from a different domain, indicating the localization pre-training generalizes beyond the target dataset."],"supporting_citations":[{"why":"supplies the Mask2Former decoder architecture that the method pre-trains and extends.","marker":"[9]"},{"why":"provides the frozen vision-language model whose mask-pooled zero-shot posteriors enter the ensembled pseudo-labels.","marker":"[52]"},{"why":"generates the class-agnostic masks used in the decoder warm-up stage.","marker":"[32]"},{"why":"is the previous CLIP-based semi-supervised segmentation method used as the state-of-the-art comparison.","marker":"[22]"},{"why":"supplies the mask-pooled CLIP classification and posterior ensembling recipe adapted here.","marker":"[76]"},{"why":"provides the mean-teacher consistency mechanism that turns weakly and strongly perturbed unlabeled images into training signal.","marker":"[61]"},{"why":"defines the weak-to-strong consistency training and the standard data partitions used in the experiments.","marker":"[74]"},{"why":"gives the mask-transformer panoptic inference used to convert ensembled posteriors into hard pseudo-labels.","marker":"[8]"}],"fun_headline_variants":["CLIP and SAM: decoupled enhancement for semi-supervised panoptic","29.9 PQ from only 158 labeled images with decoupled CLIP and SAM","Recognition and localization untangled: panoptic SOTA at low labels","Semi-supervised panoptic: 8x less memory, 29.9 PQ on ADE20K","CLIP zero-shot masks, SAM warm-up: panoptic beats supervised"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ensembling benefit assumes the mask transformer already produces candidate masks with reasonably accurate boundaries; if those masks are poorly localized or CLIP's text-image alignment is weak for the target class names, the ensembled pseudo-labels simply inherit that noise and the recognition gains collapse.","fun_headline_variants_meta":{"raw":{"variants":["CLIP and SAM: decoupled enhancement for semi-supervised panoptic","29.9 PQ from only 158 labeled images with decoupled CLIP and SAM","Recognition and localization untangled: panoptic SOTA at low labels","Semi-supervised panoptic: 8x less memory, 29.9 PQ on ADE20K","CLIP zero-shot masks, SAM warm-up: panoptic beats supervised"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000712,"raw_usage":{"total_tokens":3194,"prompt_tokens":924,"completion_tokens":2270,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":2162}},"tokens_in":540,"tokens_out":2270,"duration_ms":18380,"temperature":1.0,"reasoning_tokens":2162,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:38:36.810801+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train DEARLi on ADE20K 1/128 and degrade only the mask proposals used for CLIP mask pooling, for example by jittering their boundaries or randomly dropping low-recall masks while keeping CLIP features and the rest of the pipeline identical; if panoptic quality does not drop in step with the degradation, the claimed recognition mechanism is not what carries the gain.","supporting_citations":[{"cited_title":"Masked-attention mask transformer for universal image segmentation","cited_arxiv_id":null,"evidence_quote":"supplies the Mask2Former decoder architecture that the method pre-trains and extends."},{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"provides the frozen vision-language model whose mask-pooled zero-shot posteriors enter the ensembled pseudo-labels."},{"cited_title":"Segment any- thing","cited_arxiv_id":null,"evidence_quote":"generates the class-agnostic masks used in the decoder warm-up stage."},{"cited_title":"Semivl: Semi- supervised semantic segmentation with vision-language guidance","cited_arxiv_id":null,"evidence_quote":"is the previous CLIP-based semi-supervised segmentation method used as the state-of-the-art comparison."},{"cited_title":"Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results","cited_arxiv_id":null,"evidence_quote":"provides the mean-teacher consistency mechanism that turns weakly and strongly perturbed unlabeled images into training signal."},{"cited_title":"Revisiting weak-to-strong consistency in semi-supervised semantic segmentation","cited_arxiv_id":null,"evidence_quote":"defines the weak-to-strong consistency training and the standard data partitions used in the experiments."},{"cited_title":"Per- pixel classification is not all you need for semantic segmen- tation","cited_arxiv_id":null,"evidence_quote":"gives the mask-transformer panoptic inference used to convert ensembled posteriors into hard pseudo-labels."}],"review_version":1}