{"id":"31dba773-3d73-4e1d-a81d-217547524be5","arxiv_id":"2507.23134","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A carefully engineered pipeline of 2D grounding, 3D tracking, proposal merging, and Alpha-CLIP classification with a standardized similarity filter achieves state-of-the-art open-vocabulary 3D instance segmentation on major indoor benchmarks.","lead":"This paper combines and refines existing open-vocabulary 3D instance segmentation techniques, adding overlap removal, tracking-based proposal aggregation, iterative merging, and Alpha-CLIP classification with a standardized score filter. It reports state-of-the-art results on ScanNet200, S3DIS, and Replica, often with large margins over prior work.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Top-K budget mismatch in the ScanNet200 3D-only comparison (Ours K=300 vs OpenYOLO3D K=600) directly underpins a claimed SOTA margin, and the paper never shows the gap under a matched budget.","rationale":"The paper is a systems contribution with plausible components and a clean two-stage recipe, and the reader's conditional verdict is reasonable. I looked for a flaw that would invalidate the central SOTA claim rather than merely trim an overclaim. The strongest candidate is the Top-K budget asymmetry in Table 1's 3D-only block. This is explicit in Supplement B, and it is the only main-table comparison where the authors vary the budget across methods. The ablations reported on 'a subset' of the ScanNet200 validation set (Tables 5, 6, 10) are a real reporting gap, but they support component attribution rather than the headline SOTA, so I treat them as secondary. The abstract's closed-vocabulary sentence is also an overclaim, since Top-1 Mask3D outperforms Ours on ScanNet200, but it does not threaten the open-vocabulary recipe itself. I checked for circularity or hidden training in the method: none is apparent, and the use of frozen public models such as Grounded SAM, Alpha-CLIP, Mask3D, and ISBNet is standard for this line of work. The concrete test is straightforward and can be completed with the official OpenYOLO3D code, which the paper already used for the reported baseline. If the matched-budget margin disappears, the paper should re-report Table 1 or soften the 3D-only claim; if it survives, the condition can be lifted. My verdict therefore remains unchanged: conditional pending the matched-budget check.","tokens_in":1054,"tokens_out":1316,"duration_ms":108314,"concrete_test":"Run OpenYOLO3D's official code in the exact Table 1 3D-only setting on ScanNet200 validation (same Mask3D proposals, same query template, same Top-K selection) with K=300, and also rerun the proposed method with K=600, reporting mAP for both budgets. If OpenYOLO3D at K=300 is within 1 point of Ours at K=300, or if the ranking flips under K=600, the claimed classification advantage and the resulting SOTA statement for the 3D-only row should be re-qualified. If the margin is preserved under both matched budgets, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Supplement B the authors state: 'For Top-K evaluations, we use K = 300 for 2D-only and 3D-only experiments, and adopt K = 600 for 2D+3D experiments, following Open3DIS. OpenYOLO3D adopted K = 600 for the 3D-only experiment on the ScanNet200 dataset.' Table 1's Top-K 3D-only rows then compare Ours (K=300, 29.0 mAP) with OpenYOLO3D (K=600, 24.7 mAP), and Sec. 4.2 credits the 4.3-point gap to the classification module while asserting the proposals are the same. This is the load-bearing comparison for the 3D-only SOTA claim, and it is not controlled: Top-K truncation changes which low-confidence proposal-class pairs are retained, so the AP difference mixes classification quality with the budget mismatch. A matched-budget run is required before this row can support 'significantly outperforming other methods.' The 2D+3D headline (32.7 vs 23.7) and the S3DIS Top-1 results are not affected by this particular mismatch, so the core recipe may survive, but the reported Table 1 3D-only claim is unsecured. Separately, the abstract's claim of surpassing a closed-vocabulary method is contradicted by Top-1 Mask3D 26.9 vs Ours 25.8 in Table 1, but that is an overstatement rather than the main load-bearing issue.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage open-vocabulary 3D instance segmentation pipeline. In the proposal stage it combines image-based proposals from Grounded SAM with point-cloud proposals from Mask3D/ISBNet, introducing overlap removal in 2D, superpoint lifting with visibility filtering, frame-wise sIOU tracking, iterative merging/removal with multi-view refinement, and inclusion-based removal. In the classification stage it replaces CLIP with Alpha-CLIP and introduces a Standardized Maximum Similarity (SMS) score for false-positive filtering. Experiments on ScanNet200, S3DIS, and Replica report large gains over existing OV-3DIS methods (for example, 32.7 vs. 23.7 mAP on ScanNet200 under Top-K 2D+3D), and the abstract claims to surpass a closed-vocabulary method. The paper includes detailed pseudocode, a computation-time analysis, and sensitivity studies.","tokens_in":22493,"tokens_out":6589,"duration_ms":68627,"significance":"If the results hold under a matched comparison protocol, the paper is a substantial empirical contribution: it demonstrates that carefully combining existing concepts (2D grounding, tracking-based aggregation, iterative merging/removal, Alpha-CLIP) yields large and consistent gains over prior open-vocabulary 3D instance segmentation systems across multiple benchmarks. The ablations attribute specific gains to individual components, and the algorithm pseudocode and sensitivity analysis are useful for reproducibility. The main risks are that the 3D-only state-of-the-art claim rests on a Top-K budget mismatch, and that some ablations are reported on an unspecified subset of the validation set; both are addressable without changing the core method.","major_comments":[{"comment":"The Top-K 3D-only comparison on ScanNet200 uses K=300 for the proposed method and K=600 for OpenYOLO3D, as stated in Supplement B. Table 1 reports 29.0 vs. 24.7 mAP for this row, and Sec. 4.2 attributes the 4.3-point gap entirely to the classification module while noting that the proposals are identical. Because the Top-K budget changes which low-confidence proposal-class pairs are retained, the AP difference is confounded with the budget mismatch. Please rerun the comparison under a matched budget (both K=300 or both K=600) and report both outcomes, and do not claim a classification-only improvement until the budgets are matched.","section":"Table 1 / Supplement B / Sec. 4.2"},{"comment":"The abstract claims the framework 'even surpasses an end-to-end closed-vocabulary method,' but under the Top-1 protocol in Table 1 Mask3D reaches 26.9 mAP while Ours (2D+3D) reaches 25.8 mAP. The claim only holds under the Top-K protocol, which is a different evaluation setting from the one used for closed-vocabulary methods. Please qualify the claim to state the protocol, or remove the claim from the abstract.","section":"Abstract and Table 1"},{"comment":"The ablation studies for tracklet matching, iterative merging/removal with overlap removal and refinement, and hyperparameter sensitivity are reported on 'the subset of the ScanNet200 validation set,' but the subset is not specified (size, selection criterion, or scene list). Without this information, the component contributions are not reproducible and may not generalize to the full benchmark. Please specify the subset, or better, report these ablations on the full validation set; otherwise the corresponding conclusions should be limited accordingly.","section":"Tables 5, 6, and 10"}],"minor_comments":[{"comment":"The exact SMS threshold values (τ_SMS) used for each experiment are never listed; 'we adopt different SMS thresholds tailored to each experiment' is not sufficient for reproducibility. Please include the exact thresholds or state the selection rule.","section":"Supplement B"},{"comment":"The sIOU definition is undefined when the two instance mask sets have no co-visible superpoints; please specify the convention (for example, set the value to 0) for that case.","section":"Sec. 3.1, Eq. (2)"},{"comment":"The overlap-removal step is described in a single sentence, despite being a key component in the Table 6 ablation; a precise algorithmic description or pseudocode would improve reproducibility.","section":"Sec. 3.1"},{"comment":"The hyperparameter sensitivity table does not clearly indicate which column corresponds to the default configuration used in the main experiments; please add a marker or caption note.","section":"Supplement D.1, Table 10"},{"comment":"The word 'Exteneded' should be 'Extended' in the captions of Figures 11 and 12.","section":"Figures 11 and 12 captions"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical combination of known components with strong reported results; if the matched-budget 3D-only comparison holds, it would be a publishable contribution. The Top-K mismatch is the main obstacle, and the unspecified ablation subset undermines confidence in the component analysis. The abstract's closed-vocabulary claim should be corrected. The novelty is primarily in the recipe and refinements rather than in a new theoretical framework, which is appropriate for an empirical systems paper at this venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look for anyone working on OV-3DIS. The paper is a careful systems recipe, not a new paradigm. It takes the Open3DIS two-stage pipeline and tightens each screw: frame-wise sIOU tracking instead of tracklet-wise, 2D overlap removal before lifting, iterative merge/refine to kill duplicate and partial proposals, Alpha-CLIP for object-centric features, and a standardized maximum similarity score to filter false positives. The combination is credible and the reported gains are large: +9.0 mAP over Open3DIS on ScanNet200 2D+3D Top-K, +2.4 on S3DIS Top-1, with consistent improvements across AP and AR. The ablations attribute the gains reasonably, and the class-agnostic proposal evaluation is a nice touch. It also reports computation cost and acknowledges failures on small objects and ScanNet++. That honesty earns credit.\n\nThe soft spots are real but correctable. The most serious is the 3D-only Top-K comparison in Table 1: the paper compares its own method at K=300 against OpenYOLO3D at K=600, then credits the 4.3 mAP gap to the classification module. That gap is not controlled, because Top-K truncation changes which proposal-class pairs are scored. A matched-budget run (same K for both) is necessary before that row can support the 3D-only SOTA claim. The good news: the headline 2D+3D result and the S3DIS Top-1 results are not affected, so the core recipe probably survives.\n\nSecond, several ablations (Tables 5, 6, 10) are run on an unspecified 'subset' of the ScanNet200 validation set. It's likely a convenience subset, but without specifying which scenes or how many, the component contributions are not fully reproducible. Minor but should be fixed in revision. Third, the abstract claims to surpass an end-to-end closed-vocabulary method; under Top-1, Mask3D actually wins (26.9 vs 25.8). That only holds under the Top-K protocol, so the claim is overstated. Finally, no code release is mentioned, which is a shame for a systems paper.\n\nWho is this for? Anyone assembling an image-based 3D proposal pipeline will find the design choices useful. It is an incremental but solid empirical contribution, not a conceptual breakthrough. I would send it to review: the issues are fixable and the core claims likely hold, but the authors need to rerun the matched-budget comparison, specify the ablation subset, and soften the abstract.","headline":"Solid systems recipe with large, mostly credible gains; the 3D-only SOTA claim rests on an unmatched Top-K comparison and needs a rerun.","tokens_in":23059,"tokens_out":2610,"would_cite":true,"duration_ms":28287,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A paper argues that combining existing building blocks with careful engineering — not a new architecture — produces state-of-the-art open-vocabulary 3D instance segmentation on indoor benchmarks.","keywords":["open-vocabulary 3D instance segmentation","two-stage proposal generation","Alpha-CLIP","Standardized Maximum Similarity","frame-wise superpoint IoU tracking","iterative proposal merging and removal","ScanNet200","S3DIS"],"falsifier":"Re-run the ScanNet200 Top-K comparisons with every method capped at the same budget (for example K=300 for 3D-only systems) and re-run the ablations on the full 312-scene validation set; if a baseline then reaches or exceeds 32.7 mAP, the claimed margin disappears.","tokens_in":21958,"feed_emoji":"🏠","tokens_out":10224,"duration_ms":111302,"temperature":0.7,"pith_summary":"Open-vocabulary 3D instance segmentation asks a system to locate every object matching an arbitrary text query in a scanned indoor scene, including object categories it was never trained on. This paper's thesis is that the scattered ideas in existing methods — 2D foundation-model grounding, tracking-based 3D aggregation, and CLIP-style classification — are complementary, and that a carefully staged recipe combining them sets a new benchmark. Its pipeline generates proposals from both images and point clouds, removes overlapping and partial masks through iterative merging and refinement, and classifies with mask-aware Alpha-CLIP followed by a standardized maximum similarity filter. The reported result is state-of-the-art accuracy on ScanNet200 (32.7 mAP under the Top-K protocol, versus 23.7 for the prior best) and S3DIS (31.3 versus 28.9 mAP), with the largest gains on rare tail classes. The practical message is that fine-grained engineering details — overlap removal in 2D, frame-wise tracking, and score normalization — determine most of the performance.","feed_headline":"Open-vocabulary 3D segmentation hits 32.7 mAP on ScanNet200","feed_subtitle":"Mask-aware CLIP, tracking-based aggregation, and a normalized similarity filter beat prior open-vocabulary pipelines on indoor scenes.","key_machinery":"The argument runs on a chain of four interacting mechanisms. Frame-wise superpoint-level intersection over union (sIOU) tracking matches each new 2D observation against the per-frame instances already in a tracklet, considering only co-visible superpoints; this absorbs noisy detections that tracklet-wise matching would propagate. Iterative merging and removal then fuses tracklets whose proposal IoU exceeds the merge threshold, refines each merged mask with a multi-view consensus rate, and deletes smaller proposals whose inclusion rate in another proposal exceeds the inclusion threshold, curbing overlapped and partial masks. At classification, Alpha-CLIP injects the object mask as an alpha channel so visual features focus on the object rather than its surroundings, and the Standardized Maximum Similarity (SMS) score converts each query's raw CLIP similarities into scene-relative scores, rejecting proposals whose best match is not confident relative to the rest of the scene. Together these pieces do the work of turning noisy 2D foundation-model outputs into clean, classifiable 3D instances.","core_discovery":"The paper's central claim is that the two-stage paradigm — generate class-agnostic 3D proposals, then classify them with a vision-language model — is already the right architecture, and what separates a strong system from a weak one is the details at each stage. On the proposal side it combines image-based proposals (Grounded SAM masks lifted to 3D superpoints and aggregated by frame-wise superpoint-IoU tracking) with point-cloud proposals from pretrained 3D segmenters, then iteratively merges overlapping tracklets, refines them by multi-view consensus, and deletes proposals contained inside others. On the classification side it replaces CLIP with Alpha-CLIP, which consumes object masks as an extra alpha channel, and adds a Standardized Maximum Similarity (SMS) score computed per text query to reject low-confidence proposals. The authors report that this recipe reaches 32.7 mAP on ScanNet200 and 31.3 mAP on S3DIS under the Top-K protocol, outperforming previous open-vocabulary methods across all AP and AR metrics, and state that it even surpasses an end-to-end closed-vocabulary baseline in that setting.","pith_inferences":["The SMS score is a generic normalization: any retrieval pipeline whose scores are not calibrated across queries could apply per-query standardization without retraining, a transfer the paper does not explore.","The modular design means the recipe should keep improving as 2D grounding models and vision-language encoders improve; the reported numbers are tied to Grounded SAM and Alpha-CLIP, not to the overall architecture.","The paper's own ScanNet++ result shows the iterative merge/removal step helps less on small objects, so a size-adaptive merging threshold or a minimum-proposal-size guard is the natural follow-up.","At roughly 597 seconds per scene on a single GPU, most of it in 2D grounding and classification, the pipeline is a strong offline benchmark rather than a real-time system; speeding up or distilling those stages is an open test."],"forward_implications":["On ScanNet200, the combined pipeline raises mAP from 23.7 to 32.7 over the previous best and lifts tail-class mAP to 33.1, so rare object categories become substantially more findable.","On S3DIS, mAP rises from 28.9 to 31.3 and mAR from 44.1 to 48.2, showing that fusing image- and point-cloud-based proposals mainly buys recall.","The classification-stage changes alone add about 5.2 mAP on ScanNet200 (Alpha-CLIP +3.0, SMS filtering +2.2), indicating that open-vocabulary performance is not determined by proposal recall alone.","The same recipe transfers to synthetic Replica scenes, reaching 25.7 mAP and 48.8 mAR in the 2D+3D Top-K setting, so the method does not overfit one dataset.","Because image-only proposals match or beat previous full pipelines on several metrics, the 2D-grounding-to-3D-lifting chain is a reusable component for future systems."],"supporting_citations":[{"why":"provides the two-stage proposal-generation baseline and the visibility ratios used for lifting 2D masks to 3D superpoints.","marker":"[41]"},{"why":"introduces the tracking-based 3D aggregation and multi-view refinement that this work adapts with frame-wise sIOU matching.","marker":"[39]"},{"why":"supplies Grounded SAM, the 2D grounding model that produces the image-based instance masks.","marker":"[45]"},{"why":"is the mask-conditioned Alpha-CLIP model used in place of standard CLIP for object-centric classification.","marker":"[50]"},{"why":"provides the CLIP text and image encoders and the text template underlying the similarity matrix.","marker":"[44]"},{"why":"generates the point-cloud-based 3D proposals used on ScanNet200 and Replica.","marker":"[47]"},{"why":"generates the point-cloud-based 3D proposals used on S3DIS.","marker":"[40]"},{"why":"is the main 3D-only competitor and defines the Top-K evaluation protocol the paper compares against.","marker":"[2]"},{"why":"supplies the multi-scale visibility-weighted visual feature extraction scheme adopted for classification.","marker":"[51]"},{"why":"supplies the view-consensus mask refinement concept used after each merge.","marker":"[59]"}],"fun_headline_variants":["Mask-aware CLIP and SMS score boost open-vocab 3D segmentation","Open-vocab 3D segmentation SOTA via Alpha-CLIP and SMS filter","Details matter: open-vocab 3D segmentation reaches new SOTA","Indoor 3D segmentation: Alpha-CLIP and SMS outdo closed-vocab","Open-vocab 3D instance segmentation: mask-aware CLIP wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline margin rests on the assumption that every competitor is given the same number of allowed predictions per scene and that the ablation numbers from a subset of the validation set hold on the full benchmark; if either assumption fails, the reported gains may shrink.","fun_headline_variants_meta":{"raw":{"variants":["Mask-aware CLIP and SMS score boost open-vocab 3D segmentation","Open-vocab 3D segmentation SOTA via Alpha-CLIP and SMS filter","Details matter: open-vocab 3D segmentation reaches new SOTA","Indoor 3D segmentation: Alpha-CLIP and SMS outdo closed-vocab","Open-vocab 3D instance segmentation: mask-aware CLIP wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000698,"raw_usage":{"total_tokens":3191,"prompt_tokens":1023,"completion_tokens":2168,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":2059}},"tokens_in":639,"tokens_out":2168,"duration_ms":16949,"temperature":1.0,"reasoning_tokens":2059,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:01:06.845855+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the ScanNet200 Top-K comparisons with every method capped at the same budget (for example K=300 for 3D-only systems) and re-run the ablations on the full 312-scene validation set; if a baseline then reaches or exceeds 32.7 mAP, the claimed margin disappears.","supporting_citations":[{"cited_title":"Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance","cited_arxiv_id":null,"evidence_quote":"provides the two-stage proposal-generation baseline and the visibility ratios used for lifting 2D masks to 3D superpoints."},{"cited_title":"Ovir-3d: Open-vocabulary 3d in- stance retrieval without training on 3d data","cited_arxiv_id":null,"evidence_quote":"introduces the tracking-based 3D aggregation and multi-view refinement that this work adapts with frame-wise sIOU matching."},{"cited_title":"Alpha- clip: A clip model focusing on wherever you want","cited_arxiv_id":null,"evidence_quote":"is the mask-conditioned Alpha-CLIP model used in place of standard CLIP for object-centric classification."},{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"provides the CLIP text and image encoders and the text template underlying the similarity matrix."},{"cited_title":"Mask3d: Mask trans- former for 3d semantic instance segmentation","cited_arxiv_id":null,"evidence_quote":"generates the point-cloud-based 3D proposals used on ScanNet200 and Replica."},{"cited_title":"Isbnet: a 3d point cloud instance segmentation network with instance- aware sampling and box-aware dynamic convolution","cited_arxiv_id":null,"evidence_quote":"generates the point-cloud-based 3D proposals used on S3DIS."},{"cited_title":"Maskclus- tering: View consensus based mask graph clustering for open-vocabulary 3d instance segmentation","cited_arxiv_id":null,"evidence_quote":"supplies the view-consensus mask refinement concept used after each merge."}],"review_version":1}