{"id":"52fa459e-d970-4187-b328-bb0536d41e8d","arxiv_id":"2412.10972","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A decoupled 3D segmentation pipeline proposes class-agnostic masks from Gaussian Splatting features, then labels them with 2D open-vocabulary models.","lead":"DCSEG pairs 3D Gaussian Splatting reconstructions with off-the-shelf 2D segmentation models to label objects in 3D scenes without retraining. It separates object detection and labeling into independent stages, and reports gains over a NeRF-based baseline on indoor datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed superiority over OpenNeRF is not yet statistically established: the Replica margin is under 1 mIoU point with no variance, and the ScanNet comparison uses a sparse-view regime that may handicap the NeRF baseline; per-scene and repeated-run data are needed.","rationale":"The reader's weakest assumption is the 2D-model ceiling on assignment; I agree this is structural but the authors already flag it in Sec. 5.2, so it is a limitation rather than an unacknowledged flaw. The more actionable gap is that the quantitative headline—outperforming OpenNeRF—is supported by aggregates that are not shown to be robust. On Replica the margin is less than one mIoU point; on ScanNet the baseline is run by the authors in a sparse-view setting that likely disadvantages NeRF, and no per-scene variance is given. A concrete check with per-scene and repeated-run data would settle whether the advantage is real. This does not contradict the reader's conditional verdict; it strengthens the reasons for conditionality. Verdict remains conditional/unchanged.","tokens_in":10731,"tokens_out":8129,"duration_ms":75734,"concrete_test":"Request per-scene mIoU/mAcc for all 8 Replica scenes and all 4 ScanNet scenes, together with at least 3 independent runs (or bootstrap resampling over scenes) of both DCSEG and OpenNeRF. On ScanNet, rerun OpenNeRF with the full set of available images rather than 200, and also with 200 to isolate the effect of view sparsity. If the Replica total-mIoU advantage does not survive a paired test (e.g., Wilcoxon over scenes) or the ScanNet advantage disappears under full views, the outperformance claim should be downgraded. Also add SAGA with the same 200-image protocol as a 3DGS baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the central claim, DCSEG must beat OpenNeRF reliably, not just on one aggregate. On Replica (Table 1) the total mIoU gap is 19.9 vs 19.1 (+0.8) and the tail gap is 6.7 vs 6.6 (+0.1); across only 8 scenes with no error bars these margins are within run-to-run noise. On ScanNet (Table 3) OpenNeRF reports no numbers, so the authors run it themselves on 4 scenes with 200 images each; NeRF-based methods are known to degrade sharply with sparse views, so this protocol may systematically disadvantage the baseline. No per-scene table or confidence interval is given. The omission of SAGA, the 3DGS method whose contrastive feature stage DCSEG reuses, means the broader claim that decoupled 3DGS segmentation improves over 3DGS alternatives is untested. The acknowledged assignment ceiling from the 2D classifier (Sec. 5.2) is real but does not by itself decide the comparison; the empirical comparison is the load-bearing link.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DCSEG, a decoupled open-vocabulary 3D semantic segmentation pipeline built on 3D Gaussian Splatting. The method first reconstructs a scene with 3DGS, attaches class-agnostic affinity features to each Gaussian using a SAGA-style contrastive loss supervised by 2D SAM masks, and clusters these features with HDBSCAN to produce coarse instance- or part-level masks. A second stage projects these 3D clusters into 2D and assigns semantic labels by matching them to class-aware masks from a 2D open-vocabulary model (OVSeg or OpenSeg), using a bipartite assignment with duplicated vertices to handle multiple instances per class. The authors evaluate on Replica and ScanNet, reporting mIoU/mAcc against OpenNeRF, OpenScene, LERF, and MinkowskiNet, and include an ablation of the 2D backbone and matching versus assignment. The central claim is that the decoupled design outperforms NeRF-based OpenNeRF on total mIoU and mAcc while providing instance and part segmentation without retraining.","tokens_in":10976,"tokens_out":2257,"duration_ms":20478,"significance":"If the empirical claims are supported, the paper makes a useful contribution: it demonstrates a modular, representation-agnostic decomposition of 3D segmentation into mask proposal and mask classification, with the practical advantages of explicit geometry, fast rendering, and no retraining when swapping 2D foundation models. The ability to obtain instance- and part-level masks without a dedicated instance segmentation network is a genuine strength, and the ablation of OpenSeg versus OVSeg gives concrete evidence for the modularity claim. The paper is honest about the limitation that tail-class performance is bounded by the 2D classifier. However, the central comparison to OpenNeRF rests on margins that are small and unquantified, and the absence of the closest 3DGS baseline (SAGA) leaves the broader claim about 3DGS-based segmentation untested. The methodological skeleton is sound and the paper is clearly written, but the experimental evidence needs strengthening before the headline claim can be accepted.","major_comments":[{"comment":"The claimed superiority over OpenNeRF on Replica is not statistically established. The total mIoU margin is 19.9 versus 19.1 (+0.8), and the tail mIoU margin is 6.7 versus 6.6 (+0.1), averaged over only 8 scenes with no error bars or per-scene results. These differences are within run-to-run noise for stochastic training and evaluation. Please provide per-scene numbers, multiple seeds or confidence intervals, or a statistical test to show the improvement is reliable.","section":"§4.3, Table 1"},{"comment":"The ScanNet comparison uses only 200 images per scene for both DCSEG and OpenNeRF, but NeRF-based methods are known to degrade sharply with sparse views, so this protocol may systematically disadvantage the baseline. Since OpenNeRF does not report ScanNet numbers in its original paper, the authors run it themselves; please justify the 200-image protocol, report the original full-data numbers if available, or include a dense-view comparison to ensure the baseline is not handicapped.","section":"§4.4, Table 3"},{"comment":"Equation (1) is written with the inverse Jaccard formula: w(ma, mb) = sum_i sum_j |ma ∪ mb| / |ma ∩ mb|. As written, this is a pixel-summed inverse Jaccard index, and the standard Jaccard index should appear as the denominator of the cost or the cost should be 1 − Jaccard. This formula is load-bearing because it defines the assignment cost in the bipartite matching used for classification. Please correct the equation and define the notation clearly.","section":"§3, Eq. (1)"},{"comment":"The paper reuses SAGA's contrastive feature learning stage but never compares against SAGA, which is the closest 3DGS-based baseline. Without this comparison, the claim that the decoupled 3DGS pipeline improves over existing 3DGS segmentation approaches is untested. Please add SAGA results on the same scenes and protocol, or explicitly scope the claim to NeRF-based baselines only.","section":"§4, overall"},{"comment":"The acknowledged ceiling that tail-class performance is limited by the 2D foundation model is a structural property of the assignment stage: the 3D clustering has no independent semantic signal, so any object the 2D model fails to recognize cannot be labeled even if the 3D mask is perfect. This is not circular, but it means the reported gains over OpenNeRF depend on the specific 2D model used. The paper should state this dependency more prominently when interpreting the comparisons, and ideally include a per-class breakdown showing which classes drive the differences.","section":"§5.2, Limitations"}],"minor_comments":[{"comment":"There is a typo in Section 4.1: 'is is not bound to the evaluation classes' should read 'is not bound'.","section":"§3.1 / §4.1"},{"comment":"The caption of Table 3 says 'choosen' — should be 'chosen'.","section":"§4.4 / Table 3"},{"comment":"The contribution bullet 'event parts' appears to be a typo for 'even parts'.","section":"§1, Contributions"},{"comment":"Reference [31] cites SAM3D, but the text refers to SAM (Segment Anything Model). The attribution is likely incorrect; please cite the original SAM paper (Kirillov et al.) or clarify which model is actually used.","section":"References"},{"comment":"The caption 'e.g. the blanket/pillows or the wall behind the bed-lamps' is unclear; please specify which images show the claimed sharper boundaries.","section":"Figure 2"},{"comment":"The notation |ma,ij ∪ mb,ij| and |ma,ij ∩ mb,ij| is ambiguous because ma,ij and mb,ij are binary values at a pixel, not sets; please define the union and intersection in terms of binary masks.","section":"Equation (1)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable engineering contribution with a clean decoupling idea, but the central quantitative claim against OpenNeRF is currently supported by small margins with no variance. The missing SAGA comparison is a conspicuous gap given that the method directly builds on SAGA's contrastive loss. I would encourage the editor to send the paper back for a revision that includes per-scene results, error bars, a justification of the scan protocol, and at least a baseline SAGA experiment. The typo-level issues are minor and do not affect my recommendation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something genuinely useful: it takes the decoupled mask-proposal/classification idea from OpenMask3D and SAI3D, ports it to 3D Gaussian Splatting, and makes the modularity explicit. The duplicated-vertex bipartite assignment is a practical workaround for the cardinality mismatch, and the ability to get instance and part masks for free is a real bonus. The writing is clear, the ablations (OpenSeg vs OVSeg, matching vs assignment) are informative, and the limitations section is honest about the 2D foundation model ceiling and the outlier-cluster problem. No circularity: the pipeline uses external pretrained models and evaluates against ground truth, with no fitted parameters tuned to the metrics.\n\nThe soft spots are empirical rather than architectural. The central comparative claim—that DCSEG outperforms OpenNeRF—rests on margins that are too small to take at face value. On Replica the total mIoU gap is +0.8, and the tail gap is +0.1; with eight scenes and no per-scene or repeated-run variance, that is within noise. On ScanNet the authors run OpenNeRF themselves using only 200 images per scene, which is a protocol known to disadvantage NeRF-based methods; the comparison is not apples-to-apples with the Replica numbers. The missing SAGA baseline is the more serious omission, since DCSEG reuses SAGA's contrastive feature stage; without that comparison, the claim that decoupled 3DGS beats a 3DGS alternative is untested. The Jaccard formula as written is also wrong—it looks like a per-pixel ratio rather than a sum-of-intersection over sum-of-union—though this is likely a notation typo rather than an implementation error.\n\nWho gets value from this? Anyone working on 3DGS segmentation or modular open-vocabulary scene understanding. It is a solid baseline and a clear demonstration of decoupling, but not a breakthrough. I would send it to peer review, because the architecture and the problem are legitimate and the community needs more 3DGS segmentation comparisons. The authors should be asked to add SAGA (and ideally SAI3D/OpenMask3D), report per-scene results with variance, fix the Jaccard formula, and justify the 200-image protocol before the comparative claim is accepted.\n\nRecommendation: engage with the paper, but require the evaluation strengthening before publication.","headline":"DCSEG is a clean, modular 3DGS open-vocabulary segmentation pipeline that recombines known parts in a sensible way, but the empirical case against OpenNeRF is thinner than the headline suggests.","tokens_in":11535,"tokens_out":1844,"would_cite":true,"duration_ms":18327,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DCSEG decouples 3D mask proposal from 2D labeling to beat NeRF baselines in open-set segmentation.","keywords":["3D Gaussian Splatting","open-vocabulary segmentation","decoupled segmentation","class-agnostic mask proposal","contrastive learning","instance segmentation","bipartite assignment","ScanNet"],"falsifier":"Take a Replica scene and a tail class that OVSeg frequently misses, then rerun only the classification stage with that class's 2D masks artificially removed; if the 3D clusters for that class still exist but receive wrong or no labels despite unchanged geometry, the claimed dependence on the 2D foundation model is directly demonstrated, confirming the paper's own limitation statement.","tokens_in":1879,"feed_emoji":"🧊","tokens_out":4099,"duration_ms":52022,"temperature":0.7,"pith_summary":"This paper claims that 3D open-vocabulary segmentation can be built by decoupling the two core tasks: proposing class-agnostic object or part masks in 3D, and assigning semantic labels using a 2D open-vocabulary segmentation model. The authors construct this pipeline on 3D Gaussian Splatting, learning per-Gaussian features with contrastive supervision from SAM masks, clustering those features with HDBSCAN, and then matching the resulting 3D masks to class-aware 2D masks from OVSeg or OpenSeg. If this works, it would matter because it shows a modular alternative to NeRF-based segmentation that is faster to render, gives instance and part segmentation without extra training, and can adapt to new 2D foundation models without retraining. The paper reports improved mIoU and mAcc over OpenNeRF on Replica and ScanNet, while explicitly acknowledging that tail-class accuracy is limited by the 2D foundation model.","feed_headline":"Open-set 3D segmentation beats NeRF baseline using Gaussian splats","feed_subtitle":"Class-agnostic 3D clustering plus 2D open-vocab labels yields higher mIoU and free instance masks.","key_machinery":"The pipeline is carried by three coupled components: (1) per-Gaussian affinity features $f_{g_i}$ in $\\mathbb{R}^n$, learned by a scale-aware contrastive loss that pulls together features of pixels inside the same SAM mask and pushes apart those in different masks; (2) density-based hierarchical clustering (HDBSCAN) on those Gaussian features to produce class-agnostic clusters, which are rasterized into binary 2D masks $M_a$; and (3) a bipartite assignment between $M_a$ and the 2D class-aware masks $M_b$ from a foundation segmentation model, using the inverse Jaccard index as the cost and duplicating class vertices so multiple instances of the same class can each receive a label. The duplicated-vertex assignment, solvable by the Jonker-Volgenant variant of the Hungarian algorithm, is the workaround the authors adopt to avoid an NP-hard generalized assignment problem.","core_discovery":"The central claim is that decoupling class-agnostic 3D mask proposal from 2D class-aware mask classification yields a 3D open-set segmentation system that outperforms the NeRF-based OpenNeRF baseline in mean IoU and mean accuracy on both Replica and ScanNet scenes, while also producing instance-level and part-level masks without any retraining of an instance segmentation network. The 3D reconstruction uses explicit Gaussian primitives, and the class-agnostic masks come from clustering scale-aware contrastive features derived from SAM masks, while semantic labels come from matching these clusters to 2D open-vocabulary masks via a relaxed bipartite assignment. The authors further claim that this design is modular: swapping the 2D segmentation backbone (OpenSeg vs OVSeg) changes results but requires no retraining, and the same logic would extend to replacing the 3D representation.","pith_inferences":["The decoupling principle is a broader recipe: geometry determines what constitutes a coherent object or part, while a 2D language-conditioned model determines what that object is called; the bottleneck is the 2D classifier, so 3D segmentation quality will track 2D open-vocabulary progress.","The bipartite assignment with duplicated vertices is a heuristic that could fail when multiple instances of one class partially overlap in a given view; a testable extension would replace it with a globally consistent assignment across all views, or add a 3D language-feature disambiguation step, which the paper does not pursue.","Since the Gaussian representation is explicit and renderable, the same decoupled pipeline could be run iteratively on a live scene, enabling open-set segmentation in interactive or robotic settings without retraining whenever a novel object class is queried.","The paper's limitation on tail classes is not a flaw of the 3D clustering but a property of the 2D model; a straightforward stress test would feed a class that OVSeg cannot recognize and check that the 3D mask is still proposed, isolating the failure point."],"forward_implications":["If the central claim is correct, 3D open-set segmentation no longer requires training a dedicated 3D semantic head; it can be assembled from existing 2D open-vocabulary models plus a 3D clustering stage, meaning future improvements in 2D segmentation directly transfer to 3D.","Instance and part masks emerge as a byproduct of the class-agnostic clustering, so downstream tasks like robotic grasping or AR object manipulation get fine-grained 3D structure without an extra instance-segmentation network.","The explicit geometry of Gaussian splatting reduces noise on large uniform surfaces compared to NeRF-based pipelines, as the paper demonstrates with qualitative comparisons on walls and floors.","The plug-and-play modularity suggests that users can trade off speed versus tail-class accuracy simply by swapping the 2D backbone, as shown by the OVSeg versus OpenSeg comparison.","Because the method is zero-shot with respect to 3D labels, it can segment classes not present in the ground-truth annotations, such as posters on walls in ScanNet, potentially providing a cleaner signal for scene understanding than the provided labels."],"supporting_citations":[{"why":"Provides the SAGA approach of scale-aware contrastive learning on 3D Gaussians, which DCSEG adapts to learn per-Gaussian affinity features from SAM masks.","marker":"[3]"},{"why":"OpenNeRF is the primary NeRF-based baseline whose total mIoU and mAcc DCSEG claims to outperform on Replica and ScanNet.","marker":"[10]"},{"why":"OpenSeg serves as one of the two 2D open-vocabulary segmentation backbones used to generate class-aware masks for the classification stage.","marker":"[13]"},{"why":"OVSeg is the other 2D backbone, and the paper's best results come from OVSeg combined with the bipartite assignment.","marker":"[23]"},{"why":"3D Gaussian Splatting is the underlying explicit scene representation that replaces NeRF and provides the geometry for clustering.","marker":"[19]"},{"why":"HDBSCAN is the density-based hierarchical clustering algorithm that turns Gaussian affinity features into class-agnostic 3D masks.","marker":"[1]"},{"why":"ScanNet v2 provides the real-world indoor evaluation scenes and the 20-class NYUv2 label set used for quantitative comparisons.","marker":"[8]"},{"why":"Replica provides the synthetic indoor scenes with 51 class labels and the head/common/tail split used to report mIoU and mAcc.","marker":"[29]"}],"fun_headline_variants":["Decoupled Gaussian splatting tops NeRF in open-set 3D segmentation","Gaussian splats decouple 3D masks from labels, beating NeRF","Modular open-set 3D segmentation: Gaussian splats outclass NeRF","Gaussian splatting yields open-set 3D masks, beats NeRF pipeline"],"cache_read_input_tokens":13696,"weakest_assumption_plain":"The assignment stage assumes that class-agnostic 3D clusters, when projected to 2D, overlap with class-aware masks from OVSeg or OpenSeg well enough for the weighted bipartite match with duplicated vertices to assign correct labels to every cluster, including instances the 2D model might misrecognize.","fun_headline_variants_meta":{"raw":{"variants":["Decoupled Gaussian splatting tops NeRF in open-set 3D segmentation","Gaussian splats decouple 3D masks from labels, beating NeRF","Modular open-set 3D segmentation: Gaussian splats outclass NeRF","Gaussian splatting yields open-set 3D masks, beats NeRF pipeline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000633,"raw_usage":{"total_tokens":2940,"prompt_tokens":985,"completion_tokens":1955,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":1867}},"tokens_in":601,"tokens_out":1955,"duration_ms":11594,"temperature":1.0,"reasoning_tokens":1867,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:25:34.741928+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a Replica scene and a tail class that OVSeg frequently misses, then rerun only the classification stage with that class's 2D masks artificially removed; if the 3D clusters for that class still exist but receive wrong or no labels despite unchanged geometry, the claimed dependence on the 2D foundation model is directly demonstrated, confirming the paper's own limitation statement.","supporting_citations":[{"cited_title":"OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views","cited_arxiv_id":null,"evidence_quote":"OpenNeRF is the primary NeRF-based baseline whose total mIoU and mAcc DCSEG claims to outperform on Replica and ScanNet."},{"cited_title":"Scal- ing open-vocabulary image segmentation with image-level labels","cited_arxiv_id":null,"evidence_quote":"OpenSeg serves as one of the two 2D open-vocabulary segmentation backbones used to generate class-aware masks for the classification stage."},{"cited_title":"Open-vocabulary semantic segmentation with mask-adapted clip","cited_arxiv_id":null,"evidence_quote":"OVSeg is the other 2D backbone, and the paper's best results come from OVSeg combined with the bipartite assignment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"HDBSCAN is the density-based hierarchical clustering algorithm that turns Gaussian affinity features into class-agnostic 3D masks."}],"review_version":1}