{"id":"7d6d67ef-1f7e-4635-840a-0703c6a5f16a","arxiv_id":"2508.20063","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A single-stage image-based detector that, trained with pseudo boxes from SAM segments and CLIP features, detects and classifies arbitrary indoor objects in 3D at 0.3 seconds per scene.","lead":"OpenM3D is a 3D object detector that finds and names objects in indoor scenes from ordinary color photos, trained without any human-labeled 3D boxes. It runs in 0.3 seconds per scene instead of seconds to minutes for current open-vocabulary methods, which matters for robots and augmented reality that need fast, label-free 3D understanding.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No direct end-to-end comparison with existing OV-3D detectors on the main benchmark; the 'superior accuracy' claim rests on self-constructed baselines and an asymmetric ScanNetv2 table.","rationale":"The reader's conditional verdict is appropriate, but I do not think the mesh dependence is the single most load-bearing concern. The paper indeed uses the ground-truth mesh in Coordinates Standardization (App. B.1) and Mesh Segmentation Refinement (Sec. 3), and the 'w/o MSR' ablation does not isolate coordinate standardization, so the mesh effect is not fully measured. However, 'without human annotations' may tolerate a reconstructed mesh because the mesh is not human-annotated class or box labels. The more direct threat to the strongest claim is the absence of a clean, end-to-end comparison to existing OV-3D detection methods on the primary benchmark. Table 3 includes OpenM3D trained with OV-3DET/SAM3D pseudo boxes, not the actual detection results of those methods. Table 4, the only table with published methods, is on ScanNetv2 and has an asymmetric-evaluation footnote. If a reviewer ran the official OV-3DET on ScanNet200 and it matched or exceeded 4.23 mAP@25, the headline would be false. The concrete test above settles this. Since this is an evidentiary gap rather than a demonstrated error, conditional acceptance remains appropriate; I recommend no change to the reader's verdict.","tokens_in":20854,"tokens_out":8202,"duration_ms":84509,"concrete_test":"Use the official OV-3DET codebase (or CoDA/ImOV3D if available) to run their released/retrained models end-to-end on the ScanNet200 validation split with the same 20-view RGB-D inputs and the same class prompts, and report mAP@25/mAR@25. Then compare directly with OpenM3D's 4.23/15.12. If a published baseline reaches or exceeds OpenM3D's mAP@25, the 'superior accuracy' claim fails; if OpenM3D remains on top, the claim is supported. Also rerun Table 4 with identical candidate-box/evaluation conditions to remove the ambiguity in the footnote.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that OpenM3D achieves 'superior accuracy and speed ... compared to existing methods' on ScanNet200 and ARKitScenes. The speed part is well supported (0.3 s/scene vs 5 s for OV-3DET and 81 s for the depth-estimation baseline). The accuracy part is not: Table 3, the main OV result on ScanNet200, compares OpenM3D only against S2D, a baseline built from the authors' own class-agnostic detector with a CLIP classifier, and against OpenM3D retrained on OV-3DET/SAM3D pseudo boxes. It never reports the end-to-end mAP of OV-3DET, SAM3D, CoDA, or ImOV3D on ScanNet200. The only table listing published methods, Table 4, is on ScanNetv2 and carries a footnote ('† indicates methods evaluated with OV-3DET’s pseudo-boxes, while our evaluation uses ground-truth 3D boxes from ScanNetv2 in our multi-view setting') that indicates asymmetric evaluation. Consequently, the headline 'outperforms existing methods' is not yet empirically established; the paper establishes that its pseudo boxes are better than OV-3DET/SAM3D boxes and that its single-stage design is faster, but not that the full method beats existing end-to-end OV detectors on the claimed benchmarks. This is load-bearing because the strongest claim explicitly relies on that comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"OpenM3D proposes a single-stage, multi-view image-based 3D object detector for open-vocabulary indoor detection. Training uses no human box/class annotations: SAM 2D segments are lifted to partial 3D segments, merged across viewpoints through a graph-embedding/K-means pipeline, optionally refined with mesh-based graph-cut segments, and converted to axis-aligned 3D pseudo boxes. The detector is trained with a class-agnostic localization loss on these boxes and a voxel-semantic alignment loss that matches voxel features to CLIP embeddings from the 2D segments. At inference only multi-view RGB images and camera poses are needed; classification is performed by matching average voxel features in each box to CLIP text embeddings. Experiments on ScanNet200 and ARKitScenes report higher pseudo-box precision/recall than OV-3DET and SAM3D, better class-agnostic AP, and high speed (0.3 s per scene).","tokens_in":21236,"tokens_out":8069,"duration_ms":88452,"significance":"If the results hold, the paper makes a useful contribution by removing 3D sensors at inference and human box annotations at training, while retaining a fast single-stage pipeline. The pseudo-box generation study is the strongest part: it shows clear precision gains (e.g., 32.07% vs 14.48% for SAM3D at IoU@0.25 on ScanNet200) and consistent class-agnostic detector improvements. The paper is also transparent in treating pseudo-box evaluation on the training set as a diagnostic rather than as evidence of test-set circularity. However, the headline accuracy claim against existing OV detectors is not yet empirically established: the main ScanNet200 benchmark lacks end-to-end comparisons with published OV detectors, and the ScanNetv2 comparison in Table 4 is asymmetric. The training-time dependence on ground-truth meshes is also not disclosed and ablated. These are fixable with additional experiments or a more guarded claim, but they are load-bearing for the paper's central message.","major_comments":[{"comment":"The claim that OpenM3D is 'comparable' to or better than prior point-cloud-based OV detectors is not supported by the evidence as presented. The footnote says † methods were evaluated with OV-3DET's pseudo-boxes, while OpenM3D's evaluation uses ground-truth 3D boxes from ScanNetv2 in its multi-view setting. That is an asymmetric protocol: the prior numbers reflect pseudo-box errors, while OpenM3D's numbers reflect GT-box supervision at evaluation. Moreover, Table 3 on the main ScanNet200 benchmark only includes OpenM3D trained on different pseudo boxes and the self-constructed S2D baselines. Please add end-to-end results for OV-3DET, CoDA, and ImOV3D on the same ScanNet200 split and with the same GT-box evaluation, or substantially soften the 'superior accuracy' claim.","section":"§5.4, Table 4"},{"comment":"The training protocol uses the ground-truth mesh of each training scene in two places: (i) partial 3D segments are fused to mesh vertices by voxelization + KNN, and (ii) mesh-based graph-cut segments are used to re-label and refine the image-derived complete 3D segments. This is an extra dataset input beyond the announced 'posed RGB-D images are given' setting and likely contributes to the pseudo-box quality. The weaker MSR gains on ARKitScenes suggest mesh-quality sensitivity. Please ablate the pipeline without mesh inputs and state this dependence explicitly in the abstract/contributions; otherwise the 'without human annotations' framing overstates the method's transferability to mesh-free RGB-D scans.","section":"§3 'Mesh Segmentation Refinement' and App. B.1 'Coordinates Standardization'"},{"comment":"The ScanNet200 open-vocabulary comparison is limited to (i) OpenM3D retrained on OV-3DET/SAM3D pseudo boxes and (ii) S2D, which reuses OpenM3D's class-agnostic detector. These are useful ablations, but they do not establish superiority over existing end-to-end OV 3D detectors. The margins among the main rows are small (4.23 vs 4.04 vs 3.92 mAP@25), and with no error bars it is unclear whether these differences are meaningful. Please add published OV detectors to the same benchmark or narrow the paper's claims to pseudo-box quality, speed, and comparisons against the constructed baselines.","section":"§5.4, Tables 2-3"},{"comment":"All reported numbers are single-run point estimates. Given the stochastic components (DeepWalk, K-means, detector training) and the small margins in the main tables, the significance of the improvements cannot be assessed. Please report mean±std over at least 3 seeds for the key comparisons, or provide a statistical test.","section":"Tables 1-4, 8-10"}],"minor_comments":[{"comment":"The text says S2D shows a drop in mAR@25 'from 0.15 to 0.10', but Table 3 reports 15.12 and 10.05 (percentages). Please fix the units/notation.","section":"§5.4"},{"comment":"S2D is described as '7 times slower' in §5.1 and 'sixfold' in §5.4. Make the factor consistent (Table 7 implies 7×).","section":"§5.1 vs §5.4"},{"comment":"The overlap measure uses the smaller segment in the denominator (i.e., IoU with min rather than union). Please clarify whether θ was tuned and report sensitivity to θ and to the K-means cluster count K=100.","section":"Eq. (2)-(3)"},{"comment":"The 'first multi-view open-vocabulary 3D object detector' claim needs qualification: related work includes LeRF (image-only, but requiring per-scene reconstruction at inference) and ImOV3D (trained from images, point cloud at inference). Define the comparison boundary explicitly.","section":"Abstract/Introduction"},{"comment":"The limitation section only discusses CLIP classification. The ground-truth mesh dependence and the absence of end-to-end comparisons with existing OV detectors should be acknowledged there as well.","section":"Limitation (§C)"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising and the pseudo-box analysis is solid, but the main comparison gap and the undeclared use of ground-truth meshes at training must be addressed before publication. Please also ask the authors to clarify the 'first' claim against LeRF/ImOV3D and to add variance estimates, since the margins in the main tables are small. If the authors can provide the requested end-to-end comparisons or re-scope the claims, this would be a strong contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper: it has a genuinely new idea for generating 3D pseudo-boxes from multi-view images, and the speed numbers are credible. But when the abstract says 'superior accuracy ... compared to existing methods,' that part is not actually shown.\n\nWhat's good: The graph-embedding approach to merging SAM segments across views is a real step forward. Treating each partial 3D segment as a node, connecting overlapping segments, running DeepWalk, and clustering with K-means is a clean way to get complete 3D segments, and the pseudo-box precision numbers in Table 1 beat OV-3DET and SAM3D by a wide margin. The voxel-semantic alignment loss that pulls 3D voxel features toward CLIP embeddings is a sensible way to make a single-stage detector open-vocabulary. And the inference-time story is strong: only multi-view RGB, no CLIP or depth at inference, 0.3 s per scene. That's a real practical advantage.\n\nThe soft spots: The accuracy comparison is not apples-to-apples. The main ScanNet200 table (Table 3) only compares OpenM3D against S2D (their own two-stage) and against retraining with other pseudo-box sources. It never reports the end-to-end mAP of OV-3DET, SAM3D, CoDA, or ImOV3D on ScanNet200. Table 4, which does list those methods, is on ScanNetv2 and carries a footnote that the published numbers used OV-3DET's pseudo-boxes while OpenM3D uses ground-truth boxes. That asymmetry makes the 'superior accuracy' headline impossible to verify. Second, the 'without human annotations' framing still leans on the dataset's ground-truth mesh for coordinate standardization and mesh segmentation refinement. If you're in a setting without a high-quality mesh, the pipeline starts with noisier targets and the reported gains could shrink. Third, all results are single-run, no error bars, and the absolute mAP on ScanNet200 is around 4%, so the practical headroom is modest.\n\nNet: this is a solid paper for the method and the efficiency claim, and the pseudo-box generation is worth knowing about. But the accuracy claim needs a clean comparison against existing OV detectors on the same benchmark and same evaluation protocol before I would take it at face value. I'd send it to peer review, but the authors need to address the missing comparison and the mesh dependency.","headline":"The pseudo-box pipeline and efficiency are real; the accuracy-over-SoTA claim is not yet supported by the comparisons in the paper.","tokens_in":21739,"tokens_out":2973,"would_cite":true,"duration_ms":32144,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OpenM3D is the first multi-view open-vocabulary 3D object detector trained without human annotations, using graph-embedding pseudo boxes and CLIP-aligned voxel features to detect and classify objects from RGB images alone at 0.3 seconds per","keywords":["open-vocabulary 3D object detection","multi-view images","pseudo label generation","graph embedding","CLIP alignment","single-stage detector","indoor scenes","ScanNet200"],"falsifier":"Run the full training on ScanNet200 with the mesh-based refinement (MSR and coordinates standardization) removed or replaced by a noisy reconstructed mesh, and compare AP@25 against the reported 26.92 and against the 19.53 obtained when training on OV-3DET boxes. If the gap largely disappears, the result depends on ground-truth mesh access rather than on the graph-embedding clustering itself.","tokens_in":20793,"feed_emoji":"🎯","tokens_out":5027,"duration_ms":47706,"temperature":0.7,"pith_summary":"The paper aims to show that open-vocabulary 3D object detection in indoor scenes can be done without human annotations and without needing depth or point clouds at inference. It introduces OpenM3D, a single-stage detector built on image-induced voxel features, trained with class-agnostic 3D boxes and a voxel-semantic alignment loss. The 3D boxes come from a new pseudo-label pipeline that clusters per-view 2D segments across all viewpoints using graph embeddings, achieving higher precision and recall than existing pseudo boxes from OV-3DET and SAM3D. At inference, OpenM3D takes only multi-view RGB images and matches voxel features against text embeddings, reaching 0.3 seconds per scene and beating two-stage baselines on ScanNet200 and ARKitScenes.","feed_headline":"OpenM3D detects open-vocabulary 3D objects from images alone","feed_subtitle":"Trained on pseudo boxes and CLIP-aligned voxel features, it runs in 0.3 seconds per scene.","key_machinery":"The graph embedding-based 3D pseudo box generation: each 2D segment (from SAM) is a node, lifted to a partial 3D segment; edges mark overlapping segments in the same voxel; DeepWalk embeddings are clustered by K-means into complete 3D segments, and mesh segmentation refinement fuses these with ground-truth mesh segments. This supplies the class-agnostic training targets. The voxel-semantic alignment loss then pulls each voxel feature toward the CLIP embedding of the 2D segment that projects into it, enabling open-vocabulary classification at inference by cosine similarity with text embeddings.","core_discovery":"The central claim is that a single-stage multi-view image-based detector can perform open-vocabulary 3D detection without human annotations and without 3D sensors at test time, and that it beats stronger-looking two-stage pipelines in both accuracy and speed. The paper constructs this detector by (1) generating class-agnostic 3D pseudo boxes through graph-embedding-based clustering of SAM segments lifted into 3D, refined by ground-truth mesh segmentation, and (2) aligning the detector's voxel features with CLIP segment features during training, so that the same features localize objects and classify them against arbitrary text prompts. On ScanNet200 and ARKitScenes, the resulting pseudo boxe","pith_inferences":["The training pipeline still depends on high-quality ground-truth meshes (for mesh segmentation refinement and coordinate standardization), so the 'no human annotations' claim really means 'no human 3D boxes or class labels' given dataset meshes; removing this mesh dependence is a natural next step.","The method's gains over SAM3D come partly from considering all viewpoints jointly; extending the graph-embedding idea to temporal or multi-session data could help with long-horizon 3D aggregation.","Because the detector's open-vocabulary performance tracks CLIP's semantic granularity, better text-image alignment or a different alignment target could yield larger improvements than better pseudo boxes.","A direct testable extension is replacing ground-truth meshes with an off-the-shelf multi-view reconstruction during training, which would separate the contribution of the mesh from the contribution of the graph clustering."],"forward_implications":["Indoor 3D detection no longer requires depth sensors or point-cloud processing at inference; only multi-view RGB images and camera poses are needed.","Open-vocabulary classification can be folded into a single pass of the detector, eliminating the costly second-stage CLIP image encoder used by two-stage baselines.","The pseudo-box pipeline provides a general recipe for turning noisy per-view segmentation into coherent 3D object proposals without human labels.","The method's speed (0.3 seconds per scene) makes real-time open-vocabulary 3D perception feasible for robotics and augmented reality."],"supporting_citations":[{"why":"supplies the image-induced voxel feature volume and geometry-shaping backbone that OpenM3D adapts.","marker":"[50]"},{"why":"supplies the class-agnostic 2D segments used to build partial 3D segments for pseudo boxes and to sample CLIP features.","marker":"[19]"},{"why":"provides the pretrained image-text embedding space used for voxel-semantic alignment and open-vocabulary classification.","marker":"[38]"},{"why":"defines the posed-RGB-D training setting with no human 3D annotations and serves as the main pseudo-box and detector baseline.","marker":"[29]"},{"why":"provides the multi-view 3D segment baseline whose pseudo boxes are compared against and beaten.","marker":"[58]"},{"why":"supplies the graph embedding method whose node embeddings are clustered to form complete 3D segments.","marker":"[34]"},{"why":"supplies the graph-cut segmentation used on ground-truth meshes in the mesh segmentation refinement step.","marker":"[8]"},{"why":"provides the ScanNet scenes, meshes, and evaluation setup used for the main benchmarks.","marker":"[5]"}],"fun_headline_variants":["OpenM3D: open-vocabulary 3D detection from images alone","No labels, no depth sensor: OpenM3D detects 3D objects in 0.3s","OpenM3D: image-only open-vocabulary 3D detection, no annotations","Single-stage, image-only, open-vocabulary 3D detection in 0.3s","OpenM3D: using only RGB, detects any 3D object without human labels"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The training pipeline uses ground-truth scene meshes to refine pseudo-box boundaries and to standardize 3D coordinates; if such meshes are unavailable, the pseudo boxes get noisier and the reported accuracy gains could shrink.","fun_headline_variants_meta":{"raw":{"variants":["OpenM3D: open-vocabulary 3D detection from images alone","No labels, no depth sensor: OpenM3D detects 3D objects in 0.3s","OpenM3D: image-only open-vocabulary 3D detection, no annotations","Single-stage, image-only, open-vocabulary 3D detection in 0.3s","OpenM3D: using only RGB, detects any 3D object without human labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000448,"raw_usage":{"total_tokens":2164,"prompt_tokens":876,"completion_tokens":1288,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":1182}},"tokens_in":620,"tokens_out":1288,"duration_ms":11957,"temperature":1.0,"reasoning_tokens":1182,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:13:16.990394+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full training on ScanNet200 with the mesh-based refinement (MSR and coordinates standardization) removed or replaced by a noisy reconstructed mesh, and compare AP@25 against the reported 26.92 and against the 19.53 obtained when training on OV-3DET boxes. If the gap largely disappears, the result depends on ground-truth mesh access rather than on the graph-embedding clustering itself.","supporting_citations":[{"cited_title":"Imgeonet: Image-induced geometry-aware voxel repre- sentation for multi-view 3d object detection","cited_arxiv_id":null,"evidence_quote":"supplies the image-induced voxel feature volume and geometry-shaping backbone that OpenM3D adapts."},{"cited_title":"Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick","cited_arxiv_id":null,"evidence_quote":"supplies the class-agnostic 2D segments used to build partial 3D segments for pseudo boxes and to sample CLIP features."},{"cited_title":"Learning transferable vi- sual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"provides the pretrained image-text embedding space used for voxel-semantic alignment and open-vocabulary classification."},{"cited_title":"Open-vocabulary point-cloud object detection without 3d annotation","cited_arxiv_id":null,"evidence_quote":"defines the posed-RGB-D training setting with no human 3D annotations and serves as the main pseudo-box and detector baseline."},{"cited_title":"Deepwalk: Online learning of social representations","cited_arxiv_id":null,"evidence_quote":"supplies the graph embedding method whose node embeddings are clustered to form complete 3D segments."},{"cited_title":"Effi- cient graph-based image segmentation","cited_arxiv_id":null,"evidence_quote":"supplies the graph-cut segmentation used on ground-truth meshes in the mesh segmentation refinement step."},{"cited_title":"Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner","cited_arxiv_id":null,"evidence_quote":"provides the ScanNet scenes, meshes, and evaluation setup used for the main benchmarks."}],"review_version":1}