{"id":"15c6a5f1-9d6f-40ab-a4ad-fc41d9b7ec44","arxiv_id":"2506.22817","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"MVOV3D corrects noise in multi-view vision-language features via region-level CLIP encoding, caption-based text features, and geometric pooling, achieving 14.7% mIoU on ScanNet200 and 16.2% on Matterport160 without training 3D networks.","lead":"This paper presents MVOV3D, a training-free pipeline that improves open-vocabulary 3D scene understanding by cleaning up noisy 2D image features before fusing them into 3D. It reports top results on challenging ScanNet200 and Matterport160 benchmarks, suggesting 2D foundation models alone can beat trained 3D networks on diverse categories.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set-tuned confidence threshold and component choices make the reported zero-shot SOTA optimistically biased; fixed-configuration evaluation is needed before the headline claim is supported.","rationale":"The reader's weakest assumption is the same one I would stress: the evaluation protocol is presented as zero-shot, but the supplementary material explicitly documents tuning the confidence threshold and component selection on the test benchmarks. This is a validity problem for the central claim, not a flaw in the fusion equations. The method itself has no internal inconsistency that I found: Equations 1–5 are well-defined, the ablations show contributions from each component, and the experiments cover multiple datasets. The criticism is therefore not that the approach is unsound, but that the reported numbers do not yet demonstrate the claimed zero-shot superiority. A holdout-based re-evaluation is a cheap, decisive fix. I also note that the paper does not release code or specify the prompts used for Grounding-DINO; if GD were prompted with the benchmark class names, that would be a second, more severe leakage, but the text describes the VFM use as class-agnostic and I do not have evidence to assert that. I therefore retain the reader's CONDITIONAL verdict: the core idea is worth keeping, but the headline SOTA requires a corrected evaluation protocol.","tokens_in":16621,"tokens_out":10960,"duration_ms":120023,"concrete_test":"Re-run Table 2 with a pre-registered configuration: choose δ, VFM (SAM/GD), and ICM once on a validation split (e.g., a held-out subset of ScanNet200/Matterport3D scenes not used for final test), then evaluate on the official test splits under those fixed choices. Report all four benchmarks with the same δ, and explicitly report Replica at δ=0.24, the value used for the other datasets. If Replica falls to about 18.2 or the ScanNet200/Matterport160 margins shrink materially, the zero-shot SOTA claim is not supported; if the fixed-configuration numbers remain close to 14.7/16.2/20.1, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MVOV3D, a training-free pipeline, achieves state-of-the-art open-vocabulary 3D segmentation by 'unleashing' multi-view fusion with noise correction. For that claim to hold as a zero-shot result, the evaluation must use a configuration fixed without access to the test benchmarks. The paper reports that the confidence threshold is set separately per dataset—δ=0.24 for ScanNet200 and Matterport160, δ=0.26 for Replica (Supp. Fig. 8)—and that the auxiliary component choices (SAM & GD vs. SAM or GD alone; RAM++ vs. DeCap/BLIP) are validated on Matterport160 test mIoU (Table 5). These are exactly the benchmarks used for the headline numbers. The bias is material: the supplementary sweep shows Replica's best δ differs from the value used on the other two datasets, so if a single δ were fixed across benchmarks, Replica would drop from 20.1 toward roughly 18.2, close to OpenScene-2D's 18.0; on ScanNet200 and Matterport160 the reported gains could also shrink. Thus the 'significant margin' over trained 3D networks is partly a product of test-set selection. The ablations in §4.3 are internally consistent and the pipeline is plausible, but the evidence as reported does not establish a true zero-shot SOTA.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MVOV3D, a training-free pipeline for open-vocabulary 3D semantic segmentation. Starting from per-pixel OpenSeg features on multi-view RGB-D frames, it refines the features in three stages: (i) it replaces noisy pixel features with CLIP region features computed from high-confidence SAM/Grounding-DINO masks; (ii) it fuses CLIP text embeddings of confident tags produced by image-caption models such as RAM++; and (iii) it smooths the fused point features within 3D superpoints obtained by graph-cut on geometric normals. The paper reports state-of-the-art results on ScanNet200 (14.7 mIoU), Matterport160 (16.2), Matterport80 (23.9), and Replica (20.1), and it extends the features to open-vocabulary 3D instance segmentation by feeding them into SOLE, achieving 20.7 AP on ScanNet200. The central claim is that a training-free 2D multi-view fusion approach with noise correction outperforms trained 3D networks on large-vocabulary open-vocabulary benchmarks.","tokens_in":16879,"tokens_out":7322,"duration_ms":67834,"significance":"If the reported results are reproducible under a genuinely fixed zero-shot configuration, the contribution is significant: it provides a coherent argument that training 3D networks on limited 3D data restricts open-vocabulary generalization, and it demonstrates that a modular, training-free combination of 2D foundation models plus geometric post-processing can beat trained 3D models on large-vocabulary benchmarks. The paper's strengths include a wide set of benchmarks, component ablations, long-tail category analysis, a distillation extension, and an explicit parameter-sensitivity study in the supplementary material. The main caveat is that the evaluation protocol currently selects the confidence threshold and the auxiliary model choices on the test benchmarks used for the headline numbers, and the tag-selection loop shares the same CLIP text encoder between training-free refinement and inference. These issues are fixable with additional control experiments, but they are load-bearing for the zero-shot SOTA claim. I therefore cannot recommend acceptance in the current form.","major_comments":[{"comment":"The headline zero-shot SOTA claim is undermined by test-set-based selection of free hyperparameters. Supp. Fig. 8 reports that the confidence threshold δ is set to 0.24 for ScanNet200 and Matterport160 but to 0.26 for Replica, and Table 5 selects the SAM & GD combination and the RAM++ caption model by comparing Matterport160 test mIoU. These are the same benchmarks whose numbers appear in Table 2. The per-dataset tuning is visible in the Replica curve, which peaks at a different δ from the other two datasets; if a single δ were fixed across all benchmarks, the Replica number would decline from 20.1 toward roughly 18.2, close to OpenScene-2D's 18.0, and the ScanNet200 and Matterport160 margins could also shrink. To support the central claim, the authors need to report results with a configuration fixed a priori, or at least show that the reported conclusions are unchanged when the same δ and the same component choices are used on every dataset. Without this, the reported significant margin over prior methods is partly an artifact of test-set selection.","section":"§4.2, Table 2; Supp. Fig. 8; Table 5"},{"comment":"The text-tag selection and the inference share the same CLIP text encoder, creating a self-reinforcing loop that can inflate the reported gains. In Eq. (2), the tag T_{i,s,select} is chosen as the text proposal with the highest CLIP cosine similarity to the CLIP region feature; Eqs. (3)-(4) then flood-fill the region with that tag's embedding and average it into the point feature. At inference, the same CLIP text encoder is used to compare the resulting point feature against the benchmark label names. Selection and evaluation are therefore not independent: a tag that is cosmetically similar to the region in CLIP space may become more similar to the region at inference by construction. The authors should provide a control in which tag selection is done with an independently trained text encoder (or with text features held out), and should report performance separately on categories that never appear in the selected tag vocabulary. Without such a control, the noise-correction story is confounded with self-confirmation.","section":"§3.2, Eqs. (2)-(4)"},{"comment":"Several ablation differences are within 0.2-0.3 mIoU (e.g., w/o 2D region features: 16.0 vs 16.2 on Matterport160; w/o 3D priors: 15.9 vs 16.2), and no error bars or significance tests are reported. Since the pipeline is deterministic, the relevant variability is across scenes; reporting per-scene standard deviations or paired bootstrap confidence intervals would clarify whether these components actually contribute. The same applies to the component choices in Table 5, where SAM & GD and RAM++ are selected on margins of only 0.2-0.3 mIoU. This issue is secondary to the headline SOTA claim, but it is load-bearing for the paper's decomposition of the gains into 1D, 2D, and 3D corrections.","section":"§4.3, Table 4"}],"minor_comments":[{"comment":"The phrase 'limited amount of 3D data upbound training strong open-vocabulary 3d models' is grammatically incomplete; 'upbound' appears to be a typo for 'limits' or 'bounds'.","section":"Abstract and §1"},{"comment":"The parameter-analysis figure is hard to read because the x-axis labels and the mapping of the three series are not explained in the caption, and the sentence beginning 'as illustrated in Fig. 8, when δ is set to 0.24... ' is not a complete sentence.","section":"Supp. Fig. 8 and Supp. Sec. D"},{"comment":"The notation for the image region features is inconsistent: f_{R_{i,s}}^{2D} is introduced as an element of R^{H×W×C}, but Eq. (1) then writes f_{R_{i,s}}^{2D}(h,w,c) as a scalar; the dimensional bookkeeping should be clarified.","section":"§3.2, Eq. (1)"},{"comment":"No code or pre-trained feature release is mentioned; given the modularity of the pipeline, releasing the evaluation scripts would materially improve reproducibility and make the per-dataset configuration choices auditable.","section":"§4.2 and §4.3"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the evaluation protocol: δ and the auxiliary model choices are selected on the test benchmarks used for the headline numbers, so the claimed zero-shot SOTA is not yet supported. I would ask the editor to require the fixed-configuration control experiments described in Major Comment 1, together with the CLIP-circularity control in Major Comment 2. The paper's core idea is plausible and the experiments are extensive, so I see this as fixable within the manuscript's scope rather than a fundamental flaw."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: the paper's core claim—that a training-free 2D fusion pipeline with region-level CLIP, caption-model text, and superpoint smoothing beats trained 3D networks on ScanNet200 and Matterport160—is plausible and likely correct in direction. But the reported zero-shot SOTA numbers are not trustworthy as they stand because delta and component choices were selected using the test benchmarks.\n\nWhat's genuinely new here: the specific combination of SAM+Grounding-DINO region features, RAM++ text features, confidence-threshold filtering, and geometric superpoint pooling applied to multi-view fusion. Each part is known, but the assembled pipeline and the evaluation on 200- and 160-class vocabularies are new. The paper does a good job documenting why trained 3D networks struggle on large vocabularies, and the long-tail breakdown in the supplement supports that story. The ablations are consistent, even if some gains are modest.\n\nThe soft spots. The biggest one is the evaluation protocol. The confidence threshold delta is set per dataset (0.24 for ScanNet200 and Matterport160, 0.26 for Replica) and the VFM/ICM choices are validated on Matterport160 test mIoU. That is test-set tuning, and it directly inflates the margin over OpenScene-2D and the trained baselines. The supplementary shows Replica would drop from 20.1 to about 18.2 if delta were fixed at 0.24—basically tying OpenScene-2D. The paper offers no error bars, so small ablation differences (0.2–0.3 mIoU) may be noise. And there is no code release, which makes it hard to verify the method.\n\nThe circularity concern is real but not fatal: using the same CLIP text space to select the text tag and to score at inference can inflate agreement with that tag. I would want the authors to report how often the selected tag matches the ground-truth label. It's not cheating, but it's a bias that should be discussed.\n\nBottom line: this is a solid engineering paper with a useful empirical message, and it deserves peer review. But the zero-shot claim needs a re-run with delta and component choices fixed on a validation split, multiple seeds, and code release. I would send it out with these requirements rather than desk-reject.","headline":"Training-free 2D fusion pipeline likely beats trained 3D nets on large-vocab benchmarks, but the zero-shot SOTA claim is compromised by test-set-tuned hyperparameters.","tokens_in":17453,"tokens_out":5026,"would_cite":true,"duration_ms":48597,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free pipeline that corrects noise in 2D vision-language features sets a new state of the art for open-vocabulary 3D semantic segmentation, reporting 14.7% mIoU on ScanNet200 and 16.2% on Matterport160.","keywords":["open-vocabulary 3D scene understanding","multi-view fusion","vision-language models","noise correction","zero-shot semantic segmentation","foundation models","3D semantic segmentation","image captioning"],"falsifier":"A concrete check: run MVOV3D on ScanNet200 and Matterport160 with $\\delta$ chosen on a held-out validation split rather than the test labels, with the VFM and ICM components fixed to the reported ones, and compare mIoU to the reported 14.7% and 16.2%. If the gap over the leading trained baselines (8.2% RegionPLC, 8.7% OV3D, 9.8% DMA on ScanNet200) narrows to near zero, the claim that training-free corrected fusion is intrinsically superior would collapse.","tokens_in":16397,"feed_emoji":"🧩","tokens_out":10734,"duration_ms":102214,"temperature":0.7,"pith_summary":"This paper tries to establish that the ceiling on open-vocabulary 3D scene understanding, recognizing object classes that never appeared in any 3D training set, is set by noise in 2D vision-language features rather than by the lack of 3D training data. It proposes MVOV3D, a pipeline that performs no 3D training at all: it fuses multi-view 2D features onto 3D points, but first corrects those features by replacing unreliable pixel-level predictions with region-level CLIP image features, adding text features from high-confidence caption-model tags, and finally pooling each point's features within geometrically consistent superpoints. Against strong trained baselines, the paper reports 14.7% mIoU on ScanNet200, 16.2% on Matterport160, 23.9% on Matterport80, and 20.1% on Replica, all without training a 3D network. A sympathetic reader would take the contribution to be a demonstration that 2D foundation-model features, once denoised, transfer to 3D better than learned 3D features do on large-vocabulary benchmarks.","feed_headline":"Training-free fusion beats trained 3D networks","feed_subtitle":"Fixing vision-language noise pushes ScanNet200 to 14.7% mIoU and Matterport160 to 16.2%.","key_machinery":"The load-bearing object is the corrected per-pixel 2D feature map $f^{2D'}_i$, computed at each pixel as the average of the original VLM feature, the region-level CLIP vision feature $f^{2D}_{R_{i,s}}$ propagated from a VFM mask, and the CLIP text feature $f^{1D}_{T_{i,s,select}}$ flood-filled from a caption-model tag, with an indicator function $I(\\cdot)$ in the denominator so only available terms count. This map is fused to 3D points by the standard point-pixel alignment of OpenScene, and then geometry enters through superpoints $U_q$ produced by a graph-cut over normal similarity: the final per-point feature $F^{3D''}(p_k)$ is the average of the initially fused features over all points in $p_k$'s superpoint. Each of the three corrections targets a specific failure mode identified in the paper: region isolation fixes co-occurrence confusion, confident text supplies vocabulary beyond what the VLM can name, and superpoint pooling suppresses residual fusion noise.","core_discovery":"The paper's central discovery is that a training-free combination of off-the-shelf foundation models, orchestrated to cancel each other's noise, can beat 3D networks that are explicitly trained for open-vocabulary segmentation. The pipeline starts from pixel-aligned VLM features (OpenSeg), then corrects each image from three directions: high-confidence masks from vision foundation models isolate objects so the CLIP vision encoder can give clean region features; image caption models supply text tags whose CLIP text embeddings are accepted only above a similarity threshold $\\delta$; and after back-projecting these corrected 2D features to points, the features are averaged within superpoints built from normal similarity. The paper reports that this corrected fusion reaches a new state of the art on ScanNet200 and Matterport160 and outperforms current leading trained 3D networks by a significant margin, with the biggest gains on rare, long-tail categories.","pith_inferences":["If the reported margins survive a protocol where the confidence threshold $\\delta$ and the component choices are fixed without any test-set feedback, the result implies that the open-vocabulary 3D bottleneck is currently VLM noise rather than 3D data scarcity, so 2D foundation-model progress would transfer nearly one-for-one to 3D.","The ablation order suggests the 1D text correction gives the largest single gain, so the breadth of image-tagging vocabularies may be the binding constraint for recognizing rare classes in 3D.","A testable extension would be to replace the global threshold $\\delta$ with an adaptive per-region criterion, for example accepting a text tag only when multiple views agree on it; this could reduce mislabeling without retraining.","The superpoint pooling assumes geometrically similar points belong to the same class, which may oversmooth scenes where visually similar objects are semantically distinct; combining the geometric pooling with mask boundaries from the VFM could preserve the gain while reducing that risk."],"forward_implications":["On large-vocabulary benchmarks, ensembling distilled 3D features with 2D multi-view features (OpenScene-2D3D) is worse than using the 2D features alone, which contradicts the pattern reported on 20-class and 21-class benchmarks.","A 3D network distilled from the corrected features inherits part of the gain, improving OpenScene-3D by +0.6 mIoU on ScanNet200 and +1.0 on Matterport160, so the correction is useful even when only a 3D model can be deployed.","The long-tail breakdown shows the method's advantage is concentrated in rare categories: tail-class mIoU is 18.5 on Replica and 10.8 on Matterport160, whereas distilled 3D features nearly collapse to 1.1 on Matterport160 tail classes.","Because no training is involved, swapping in stronger versions of the VFM, ICM, or VLM components should translate directly into higher 3D open-vocabulary accuracy."],"supporting_citations":[{"why":"supplies the pixel-aligned VLM (OpenSeg) whose per-pixel features are the initial multi-view features and the subject of the noise analysis.","marker":"[10]"},{"why":"provides the shared CLIP vision and text encoders used for region features, text embeddings, and final open-vocabulary similarity scoring.","marker":"[36]"},{"why":"with Grounding-DINO, generates the high-confidence class-agnostic masks that isolate objects for region-level feature extraction.","marker":"[19]"},{"why":"produces the object proposals that, together with SAM, define the 2D masks used in the 2D refinement step.","marker":"[26]"},{"why":"is the default image caption model (RAM++) whose text tags supply the 1D refinement features; ablations also use [22] and [24].","marker":"[13]"},{"why":"defines the OpenScene 2D multi-view fusion baseline, the point-pixel alignment used in fusion, and the OpenScene-2D/2D3D comparisons that motivate the method.","marker":"[32]"},{"why":"supplies the supervoxel/superpoint algorithm used to build the geometric regions for the 3D guided fusion.","marker":"[31]"},{"why":"is the leading 3D-2D distillation baseline (DMA) whose Matterport80/160 and ScanNet200 scores are the primary numbers MVOV3D surpasses.","marker":"[38]"},{"why":"is the geometry-guided self-distillation baseline (GGSD) that MVOV3D beats on Matterport160.","marker":"[44]"},{"why":"is the point-language contrastive baseline (RegionPLC) whose ScanNet200 score of 8.2% is the comparison behind the claimed margin.","marker":"[52]"}],"fun_headline_variants":["Noise-corrected VLM fusion beats trained 3D segmentation","Training-free multi-view fusion sets new 3D SOTA","Fixing VLM noise unlocks open-vocab 3D gains","Zero-shot fusion topples trained 3D networks","Multi-view noise correction tops learned 3D models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the confidence cutoff and the off-the-shelf component choices keep working on new scenes without retuning, and that those components make fewer mistakes than the vision-language noise they are correcting.","fun_headline_variants_meta":{"raw":{"variants":["Noise-corrected VLM fusion beats trained 3D segmentation","Training-free multi-view fusion sets new 3D SOTA","Fixing VLM noise unlocks open-vocab 3D gains","Zero-shot fusion topples trained 3D networks","Multi-view noise correction tops learned 3D models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1554,"prompt_tokens":1014,"completion_tokens":540,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":454}},"tokens_in":630,"tokens_out":540,"duration_ms":5676,"temperature":1.0,"reasoning_tokens":454,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:56:53.029643+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: run MVOV3D on ScanNet200 and Matterport160 with $\\delta$ chosen on a held-out validation split rather than the test labels, with the VFM and ICM components fixed to the reported ones, and compare mIoU to the reported 14.7% and 16.2%. If the gap over the leading trained baselines (8.2% RegionPLC, 8.7% OV3D, 9.8% DMA on ScanNet200) narrows to near zero, the claim that training-free corrected fusion is intrinsically superior would collapse.","supporting_citations":[{"cited_title":"Scal- ing open-vocabulary image segmentation with image-level labels","cited_arxiv_id":null,"evidence_quote":"supplies the pixel-aligned VLM (OpenSeg) whose per-pixel features are the initial multi-view features and the subject of the noise analysis."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"provides the shared CLIP vision and text encoders used for region features, text embeddings, and final open-vocabulary similarity scoring."},{"cited_title":"Segment any- thing","cited_arxiv_id":null,"evidence_quote":"with Grounding-DINO, generates the high-confidence class-agnostic masks that isolate objects for region-level feature extraction."},{"cited_title":"Open-Set Image Tagging with Multi-Grained Text Supervision","cited_arxiv_id":"2310.15200","evidence_quote":"is the default image caption model (RAM++) whose text tags supply the 1D refinement features; ablations also use [22] and [24]."},{"cited_title":"Openscene: 3d scene understanding with open vocabularies","cited_arxiv_id":null,"evidence_quote":"defines the OpenScene 2D multi-view fusion baseline, the point-pixel alignment used in fusion, and the OpenScene-2D/2D3D comparisons that motivate the method."},{"cited_title":"V oxel cloud connectivity segmentation- supervoxels for point clouds","cited_arxiv_id":null,"evidence_quote":"supplies the supervoxel/superpoint algorithm used to build the geometric regions for the 3D guided fusion."},{"cited_title":"Dense multimodal alignment for open-vocabulary 3d scene understanding","cited_arxiv_id":null,"evidence_quote":"is the leading 3D-2D distillation baseline (DMA) whose Matterport80/160 and ScanNet200 scores are the primary numbers MVOV3D surpasses."},{"cited_title":"Open vocabulary 3d scene under- standing via geometry guided self-distillation","cited_arxiv_id":null,"evidence_quote":"is the geometry-guided self-distillation baseline (GGSD) that MVOV3D beats on Matterport160."},{"cited_title":"Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding","cited_arxiv_id":null,"evidence_quote":"is the point-language contrastive baseline (RegionPLC) whose ScanNet200 score of 8.2% is the comparison behind the claimed margin."}],"review_version":1}