{"id":"f674ddae-85de-4539-8160-8f80a88efa6c","arxiv_id":"2507.13363","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The paper builds a training-free, open-vocabulary 3D detector by lifting GroundingDINO and SAM 2D proposals into LiDAR or pseudo-depth point clouds, reaching about 30% mAP on nuScenes mini.","lead":"This paper shows that 3D object boxes can be produced for arbitrary text queries without training any 3D model. A 2D vision-language detector finds objects, their pixels are lifted into point clouds, and simple geometry fits the boxes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core geometric assumption that visible points within each SAM mask cover the full object extent is unverified and likely fails for occluded or truncated objects, which would invalidate the claim of general open-vocabulary 3D detection.","rationale":"I agree with the reader's weakest_assumption. The most load-bearing concern is the unverified assumption that the visible LiDAR points inside each SAM mask span the full object extent after DBSCAN. This is central because the medoid and calipers produce a box from those points; occlusion/truncation make the visible set partial, biasing center and size. The paper's experiments on two nuScenes sequences do not stratify by visibility, so the failure is invisible in the aggregate mAP. I considered the reader's other criticism about dataset-specific priors; however, Table 1 row 5 (Medoid + Rotating Calipers) achieves 29.30% mAP without lane geometry or shape priors, so the no-priors claim can be supported by a prior-free variant. Thus the occlusion/truncation assumption is the decisive risk. The appropriate remedy is a conditional acceptance requiring a visibility-stratified evaluation and an explicit statement of this limitation; hence the verdict remains CONDITIONAL (UNCHANGED).","tokens_in":8157,"tokens_out":7637,"duration_ms":75995,"concrete_test":"On the nuScenes mini validation split, stratify the evaluation by the official per-annotation visibility attribute (nuScenes provides visibility codes 1 to 4). Compute mAP and mASE separately for fully visible objects (visibility 4) and for occluded objects (visibility 1-3) using the prior-free configuration (Medoid + Rotating Calipers + DBSCAN, Table 1, row 5). If the occluded-object mAP is substantially below the fully-visible mAP (e.g., below half), or if mASE increases significantly, the full-extent assumption is violated and the central claim needs to be restricted to unoccluded objects.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The pipeline in Section 3.4 assumes that after back-projecting the SAM mask and DBSCAN filtering, the surviving 3D points capture the full geometric extent of the target object. The medoid center and the minimum-area rotating-calipers rectangle in the ground plane describe the visible point set, not the hidden portion of the object. Under occlusion (common in driving scenes, e.g., a pedestrian partially behind a car) or truncation, the visible point cloud is a strict subset of the object's surface; the resulting box is systematically biased toward the visible side, producing centers shifted away from the true center and sizes smaller than the ground-truth box. The paper's nuScenes evaluation does not stratify results by per-object visibility, so it cannot detect or quantify this failure. The central claim of training-free open-vocabulary 3D detection depends on this assumption holding broadly; without it, the method is at best a visible-surface estimator, not an object detector.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a training-free, open-vocabulary 3D object detection pipeline that combines off-the-shelf 2D open-vocabulary detectors (GroundingDINO), SAM instance segmentation, back-projection of LiDAR or monocular pseudo-depth into 3D, DBSCAN noise filtering, and classical geometric fitting (medoid centering and rotating calipers) to produce 3D bounding boxes without any 3D-annotated training data. The authors also introduce Pseudo-nuScenes, a fog-augmented RGB-D variant of nuScenes mini, and report 29.94% mAP on nuScenes mini and 12–16% mAP on Pseudo-nuScenes, finding that their best variant is competitive with the older 3D For Free baseline while remaining training-free.","tokens_in":8337,"tokens_out":5285,"duration_ms":57895,"significance":"If validated, the core idea is attractive and practically relevant: it shows that mature 2D vision-language models plus geometric post-processing can produce open-vocabulary 3D boxes without 3D supervision, which is a meaningful step beyond trained 3D open-vocabulary detectors. The pipeline is modular, the components are off-the-shelf, and the authors state they release code. The evaluation is not circular, since the 3D boxes are produced by external 2D models and geometry rather than test-label supervision. However, the current evidence is too thin to support the headline claim of 'competitive' open-vocabulary 3D detection: the evaluation uses only 10 validation scenes and 5 classes, has no error bars or per-class breakdown, does not compare with modern open-vocabulary 3D baselines, and the best-performing variant relies on lane geometry and shape priors that contradict the paper's stated absence of dataset-specific priors. The paper is an interesting position piece, but the central claim needs substantially more support.","major_comments":[{"comment":"The geometric inflation step assumes that the back-projected points inside each SAM mask, after DBSCAN keeps only the densest cluster, capture the full geometric extent of the object. The medoid center and the rotating-calipers box are computed from the visible point set, so under occlusion or truncation the box is biased toward the visible side and is systematically smaller than the ground-truth box. Driving scenes frequently contain such cases, and nuScenes provides visibility attributes that would allow a stratified evaluation. The manuscript does not provide any such stratification, so the reported 29.94% mAP cannot be interpreted as evidence that the method handles occluded or truncated objects. Please add an experiment that reports performance separately for visible, occluded, and truncated objects, or clearly state and quantify this limitation.","section":"Section 3.4 and Section 5.2"},{"comment":"The paper repeatedly claims that the method uses 'no dataset-specific priors' (e.g., Section 2, Section 6.1), but the best-performing row in Table 1 is 'Medoid + Lane geometry + shape priors' (29.94%). Lane geometry and per-class shape priors are dataset-specific priors by construction, and their definitions or values are not provided anywhere in the manuscript. This makes the headline result unreproducible and weakens the 'training-free and prior-free' claim. Please either report the fully prior-free variant (e.g., medoid + rotating calipers without lane geometry and without class-specific shape priors) as the main claim, or explicitly list and justify all priors used.","section":"Section 5.3, Table 1, Section 6.1"},{"comment":"The claim that 'our method featuring the 3D Rotating Calipers strategy yields improved orientation estimates, with a lower mAOE (1.045 vs. 1.144), surpassing the baseline inflation methods [21]' is contradicted by Table 1. The value 1.045 appears in the row 'Medoid + Lane geometry + shape priors', not in a rotating-calipers row, and the 3D For Free baseline with HD maps reports mAOE 0.90, which is lower than 1.045. This comparison should be corrected, and the conclusion about orientation accuracy should be re-evaluated.","section":"Section 6.1"},{"comment":"The entire evaluation uses only two nuScenes validation sequences (10 scenes total) and 5 classes, with no error bars, no per-class results, and no comparison with the open-vocabulary 3D detection methods discussed in Section 2 (e.g., CLIP-FO3D, Lu et al., FSD). mAP computed on 10 scenes is high-variance, and the absence of any modern open-vocabulary baseline makes the 'competitive localization performance' claim unsupported. Please enlarge the evaluation (at least the full nuScenes validation set), provide error bars or per-sequence results, and compare with at least one recent open-vocabulary 3D detector.","section":"Section 5.1, Tables 1 and 2"}],"minor_comments":[{"comment":"The fog model is defined with parameters beta and A, but the values of beta are never specified and A is only described as 'set to white'. Without these values, Pseudo-nuScenes is not reproducible.","section":"Section 4.1"},{"comment":"The row labels are inconsistent and ambiguous: 'Medoid + Lane geometry + shape priors', 'shape priors No', and 'Medoid + Rotating Calipers for orientation, shape' do not clearly indicate which components are active and which are ablated.","section":"Table 1"},{"comment":"References [10] and [11] are duplicate entries for GroundingDINO; the reference list should be de-duplicated and all author names should be checked for typos (e.g., 'V ora' in [1,2]).","section":"References"},{"comment":"The paper states that Pseudo-nuScenes is a 'benchmark', but it is presented as a processed version of the nuScenes mini split with generated pseudo-depth and fog; details about how the dataset will be released (format, license, exact generation script) are missing.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The central idea is plausible and the pipeline is clearly described, but the evidence is not yet sufficient for the claimed contributions. The most load-bearing issues are the contradiction between the best variant using lane geometry and shape priors and the stated prior-free claim, the misreported mAOE comparison, and the very small evaluation set. I would encourage the editor to ask the authors for a corrected comparison, a larger or stratified evaluation, and an explicit statement of all priors used."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me save you the read. This paper shows that GroundingDINO + SAM + DBSCAN + rotating calipers can produce 3D boxes for natural-language queries without any training, and it introduces Pseudo-nuScenes, a fog-augmented RGB-only benchmark. The core idea is alive: it works well enough on nuScenes mini to be a useful proof-of-concept, and the code and benchmark could be genuinely useful to people studying 2D-to-3D lifting.\n\nThe good parts are real. The DBSCAN ablation (1.3% to 21.9% mAP) is a nice illustration of why noise filtering matters. The pseudo-depth experiments at 12-16% mAP honestly show how much recall you lose when you drop LiDAR. The writing is clear and the pipeline is easy to follow.\n\nThe soft spots are in proportion to the claims. The evaluation is 10 validation scenes and 5 classes, with no error bars. The comparison is only with 3D For Free, and their best numbers are 5-14 points lower, so \"competitive\" is a stretch. There is no comparison with any open-vocabulary 3D baseline, trained or otherwise. More importantly, the paper's Section 2 summary says \"no dataset-specific priors,\" but the best table row uses lane geometry and handcrafted shape priors. That contradiction needs to be resolved. The rotating-calipers-only version drops to 21.94%, so the \"no priors\" version is significantly weaker. The geometric assumption that the visible points inside a SAM mask cover the object's full extent is unverified; occlusion and truncation will systematically bias centers and sizes. The paper doesn't stratify by visibility or report per-class results, so it can't quantify this. That's a real limitation, but not a fatal one: the method is best described as a visible-surface box estimator, and the claim of general open-vocabulary 3D detection should be softened. References have duplicates and a couple of author lists look wrong; a referee should flag that.\n\nThis deserves a serious referee. It's a solid proof-of-concept with a benchmark others might use. I would send it to peer review, but expect heavy revision: fix the priors contradiction, add error bars and per-visibility breakdown, compare to at least one actual open-vocab 3D detector, and clean up the references. The central idea holds; it just needs to be positioned honestly.","headline":"A plausible training-free 2D-to-3D lifting recipe with a real benchmark, but the evaluation is thin and the best row quietly uses the dataset-specific priors the paper says it avoids.","tokens_in":8927,"tokens_out":3801,"would_cite":true,"duration_ms":40804,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that arbitrary natural-language classes can be localized in 3D with off-the-shelf 2D foundation models and pure geometry, no 3D labels or training needed.","keywords":["open-vocabulary 3D object detection","training-free detection","2D foundation models","geometric box inflation","DBSCAN outlier filtering","rotating calipers","nuScenes","monocular pseudo-depth"],"falsifier":"On nuScenes frames where an object is partially occluded or truncated, compare the predicted box against the full annotated box and plot center and size error against the fraction of the object visible in the mask; if errors spike sharply when the visible fraction is small, the coverage assumption fails.","tokens_in":7920,"feed_emoji":"🚗","tokens_out":8897,"duration_ms":85512,"temperature":0.7,"pith_summary":"Open-vocabulary 3D detection usually needs 3D training data or a learned 3D feature extractor. This paper claims to sidestep both: a 2D vision-language detector proposes boxes for arbitrary text prompts, a segmentation model refines them, and camera geometry lifts the resulting masks into 3D point clouds from which an untrained geometric routine recovers 3D bounding boxes. The system is entirely training-free, needs no 3D annotations, and works with either LiDAR or monocular pseudo-depth. On nuScenes mini it reports 29.94% mAP with a medoid-plus-shape-priors variant, close to a prior HD-map-based baseline, and on a new fog-augmented RGB-only benchmark, Pseudo-nuScenes, it reports 12–16% mAP. The paper positions this as the first fully training-free, open-vocabulary 3D detection pipeline built purely from 2D foundation models and geometry.","feed_headline":"No training, no 3D labels: geometry turns 2D detections into 3D boxes","feed_subtitle":"A training-free pipeline hits 29.94% mAP on nuScenes mini and 12-16% using only RGB pseudo-depth.","key_machinery":"The load-bearing mechanism is the 2D-to-3D lifting and inflation loop. GroundingDINO turns a natural-language prompt into 2D boxes; SAM turns each box into an instance mask; known camera intrinsics and extrinsics back-project the pixels covered by the mask onto either LiDAR points or UniDepth pseudo-depth points; DBSCAN keeps the densest cluster to remove projection noise; and a medoid center plus a ground-plane oriented bounding box from Rotating Calipers, with height taken from vertical point extent, yields the final 3D box. The optional shape priors and lane geometry are handcrafted anchors used only in the best-performing variant. What this machinery does is replace learned 3D feature extraction with pure geometry: every stage is deterministic and training-free.","core_discovery":"On its own terms, the paper's central claim is that an off-the-shelf 2D open-vocabulary detector, combined with a segmentation mask and a classical 3D inflation step, is sufficient to localize arbitrary natural-language classes in 3D without any 3D supervision. The discovery is an application result: mature 2D vision-language models already encode enough object knowledge that their 2D outputs can be turned into usable 3D boxes by projection and geometry alone. The strongest reported evidence is 29.94% mAP on nuScenes mini using the medoid of back-projected LiDAR points as center, with handcrafted lane geometry and shape priors for box size, compared with 34.31% for the rotating-calipers variant of the HD-map-based baseline; replacing LiDAR with UniDepth pseudo-depth gives 12–16% mAP under synthetic fog. The paper does not claim to beat trained 3D detectors; it claims that the ceiling of 2D foundation models transfers to 3D with no additional learning.","pith_inferences":["A testable extension follows from the pipeline's construction: improving 2D open-vocabulary recall should improve 3D mAP nearly one-for-one, because a 3D proposal can only exist where the 2D detector first found the object.","A control experiment on the new benchmark would isolate error sources: running the same RGB-only pipeline on clear-weather nuScenes images versus Pseudo-nuScenes fog images would separate depth-estimation noise from fog degradation.","Because labels and confidence scores are inherited from 2D detections, the system's semantics are only as good as the 2D detector; a spurious 2D box appears as a spurious 3D box with no learned module to correct it."],"forward_implications":["Any class expressible in a 2D open-vocabulary prompt becomes a 3D detection target, so novel categories can be queried at inference time with no 3D class list.","The gap to the HD-map baseline is modest (29.94% vs 34.31% mAP) despite using no maps or dataset priors, so pure geometric inflation is a viable substitute for map-based priors in LiDAR settings.","Pseudo-depth makes the pipeline fully RGB-only, and even under synthetic fog the method retains 12–16% mAP, indicating that monocular depth can stand in for LiDAR in constrained settings.","DBSCAN outlier removal is not a detail: without it the rotating-calipers-only variant drops from 21.94% to 1.30% mAP, so density filtering is what keeps back-projected noise from destroying boxes."],"supporting_citations":[{"why":"Supplies the text-conditioned 2D detections that seed the entire 3D pipeline.","marker":"[10]"},{"why":"Supplies the instance masks whose back-projected points define each candidate object's 3D extent.","marker":"[6]"},{"why":"Provides the zero-shot metric depth maps used in the RGB-only pseudo-depth variant.","marker":"[15]"},{"why":"Provides the nuScenes mini split images, LiDAR, calibrations, annotations, and evaluation protocol.","marker":"[1]"},{"why":"Provides the HD-map-based pseudo-annotation baseline whose mAP the paper's LiDAR result is compared against.","marker":"[21]"}],"fun_headline_variants":["2D foundation models unlock open-vocabulary 3D detection","No labels, no training: geometry projects 2D detections into 3D","Training-free 3D boxes from 2D vision-language detectors","Geometry only: 2D detections become open-vocab 3D boxes","From 2D to 3D without training or human annotations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The visible 3D points inside a mask, after the densest cluster is kept, cover the object's full extent, so the medoid center and calipers box can match the true annotated box.","fun_headline_variants_meta":{"raw":{"variants":["2D foundation models unlock open-vocabulary 3D detection","No labels, no training: geometry projects 2D detections into 3D","Training-free 3D boxes from 2D vision-language detectors","Geometry only: 2D detections become open-vocab 3D boxes","From 2D to 3D without training or human annotations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1536,"prompt_tokens":1037,"completion_tokens":499,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":400}},"tokens_in":653,"tokens_out":499,"duration_ms":5002,"temperature":1.0,"reasoning_tokens":400,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:47:53.090590+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On nuScenes frames where an object is partially occluded or truncated, compare the predicted box against the full annotated box and plot center and size error against the fraction of the object visible in the mask; if errors spike sharply when the visible fraction is small, the coverage assumption fails.","supporting_citations":[{"cited_title":"Unidepth: Universal monocular metric depth estimation","cited_arxiv_id":null,"evidence_quote":"Provides the zero-shot metric depth maps used in the RGB-only pseudo-depth variant."},{"cited_title":"Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom","cited_arxiv_id":null,"evidence_quote":"Provides the nuScenes mini split images, LiDAR, calibrations, annotations, and evaluation protocol."}],"review_version":1}