{"id":"f3877865-909c-4b7b-b2c9-bdbf9ad180ad","arxiv_id":"2411.15657","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"OVM3D-Det automatically labels 3D boxes from RGB images and depth estimates, then trains a monocular 3D detector that beats naive baseline matching on novel classes in indoor and outdoor datasets.","lead":"The paper trains a 3D object detector that can recognize new object types using only ordinary camera images, without LiDAR or 3D annotations. It generates its own 3D training labels by combining an open-vocabulary 2D detector, a depth-estimation model, and size priors from GPT-4.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pseudo-label quality hinges on unvalidated zero-shot metric depth from Unidepth; Sec. 3.1 lacks per-instance depth or box-quality measurement, and Appendix F admits distance-dependent error.","rationale":"The reader's weakest_assumption already targets the depth model, and I concur. The reason this is the single most load-bearing issue is upstream dependency: Eq. (1) is the only source of metric 3D structure in the entire pipeline. If the depth is wrong, the erosion and box-search refinements are cosmetic; the paper provides no direct evidence that Unidepth produces metric depth on these four datasets. The no-overlap assertion is one piece of evidence for generalization, but even if it is true, zero-shot metric accuracy is empirically uncertain. Appendix F's admission of quadratic error growth is a red flag for outdoor novel classes that are often far away. Table 12's depth swap shows sensitivity, and Table 5's ablations all assume the pseudo-LiDAR geometry is a reasonable starting point. The baseline and metric issues raised by the reader are real but secondary: the baseline is weak by design in a new task, and the lenient AP3D metric is applied uniformly. A direct pseudo-label quality measurement on KITTI val would settle the depth question decisively and is feasible because KITTI provides LiDAR. Thus I propose no change to the reader's conditional verdict; the condition is exactly this missing measurement.","tokens_in":19148,"tokens_out":10199,"duration_ms":95400,"concrete_test":"Run the complete Sec. 3.1-3.2 pipeline on the KITTI validation split (which has LiDAR ground truth) with Unidepth, and compare each pseudo-box to its matched GT box: report median and 90th-percentile depth-center error, median 3D IoU, and the fraction of pseudo-boxes with IoU > 0.1, separately for distance bins (<20m, 20-50m, >50m). If median depth error at >20m exceeds roughly 1m or median 3D IoU falls below roughly 0.1, the auto-labels are too noisy to support the claim that the detector learns genuine 3D geometry, and the reported +6.7 AP on novel KITTI classes would need to be reinterpreted as fitting noise. If the numbers are healthy, the depth assumption is validated and the conditional can be lifted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that OVM3D-Det trains a detector from RGB images only—rests on the auto-labeling chain in Sec. 3.1: Eq. (1) unprojects pixel depth D(u,v) into pseudo-LiDAR, and every later component (adaptive erosion, ground-plane fit, PCA orientation, box search in Sec. 3.2) consumes this geometry. If Unidepth's zero-shot metric depth is biased or scale-off on any target dataset, all downstream boxes inherit the error; no amount of erosion or box search can correct a systematic depth shift. The paper states that KITTI, nuScenes, SUN RGB-D, and ARKitScenes are absent from Unidepth's training data, but gives no quantitative depth error or pseudo-box quality on these datasets. Appendix F concedes that depth errors grow quadratically with distance, which is exactly the regime where novel outdoor instances live. The only depth-related ablation (Table 12) swaps Metric3D for Unidepth and shifts AP from 17.0 to 18.5, confirming sensitivity to depth quality, but it does not test for per-instance scale errors. Since the 'without 3D data' claim is only as strong as the pseudo-labels, the missing measurement is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OVM3D-Det, a pipeline that trains an open-vocabulary monocular 3D object detector using only RGB images, without target-domain LiDAR or 3D annotations. The method uses Grounded-SAM for open-vocabulary 2D detection and segmentation, Unidepth for zero-shot metric depth estimation, back-projects instance masks into pseudo-LiDAR, applies adaptive erosion, estimates orientation via PCA, and refines boxes using LLM-provided class size priors and a box-search loss. The resulting pseudo 3D labels supervise a Cube R-CNN detector with a text-alignment head. Experiments on KITTI, nuScenes, SUN RGB-D, and ARKitScenes compare against an oracle, a hand-built Cube R-CNN + Grounding DINO baseline, and point-cloud open-vocabulary methods, reporting novel-category improvements of +6.7 to +16.8 AP. Ablation studies isolate the contributions of each component.","tokens_in":19424,"tokens_out":7033,"duration_ms":66498,"significance":"If the central claim holds, this is a useful contribution: it demonstrates a practical way to bootstrap open-vocabulary monocular 3D detection without target-domain 3D annotations, which could enable scaling to internet-scale RGB data. The paper has clear strengths: controlled ablations in Table 5 support each design choice; the evaluation spans four indoor and outdoor datasets; Appendix B shows LLM priors are comparable to dataset statistics; and the limitations section is honest about distance-dependent depth errors. The main uncertainties are the lenient evaluation protocol, the unquantified reliance on zero-shot metric depth, and the unclear amount of validation supervision used to tune hyperparameters. These are fixable with additional experiments and disclosure, so the contribution is defensible but needs revision.","major_comments":[{"comment":"The main results report only mAP3D averaged over IoU thresholds from 0.05 to 0.50. At an IoU threshold of 0.05, almost any coarsely placed box counts as a true positive, and this protocol is not standard for KITTI or nuScenes, which typically use AP at 0.25/0.5 or official difficulty-based metrics. Because the central claim is that the auto-labeled pipeline 'significantly outperforms' baselines, the authors should report AP at standard thresholds (e.g., AP3D at IoU 0.25 and 0.50) and per-class AP for the novel categories. This is particularly important because the reported gaps may shrink considerably at stricter IoU thresholds.","section":"Sec. 4.1, Tables 1-2"},{"comment":"The entire pseudo-label chain assumes Unidepth's zero-shot depth estimates are metric and scale-correct on all target datasets, yet the paper never reports per-instance depth error or pseudo-box quality on KITTI, nuScenes, SUN RGB-D, or ARKitScenes. Appendix F and Appendix H concede that depth errors grow quadratically with distance and that distant objects are missed, which is precisely the regime where many novel outdoor categories live. Table 12 swaps Metric3D for Unidepth (17.0 vs 18.5 AP), confirming sensitivity to depth quality, but it does not measure per-instance scale errors. Please add depth error metrics (e.g., AbsRel and delta-1 against available depth sensors) and pseudo-label quality metrics (recall, precision, and IoU against ground-truth boxes). Without these measurements, the claim of training 'using only RGB images' is not fully supported.","section":"Sec. 3.1, Eq. (1), Appendix F/H"},{"comment":"The hyperparameters tau1, tau2, lambda, the number of erosion iterations, and the mask-size threshold are free parameters, and the paper does not disclose how they were selected. If any of them were chosen by maximizing mAP3D on KITTI validation using ground-truth 3D boxes, then the method is not strictly 'without 3D data,' because validation supervision leaks into the auto-labeling configuration. Please disclose the exact selection procedure, report sensitivity on at least one additional dataset, and, ideally, provide a fully unsupervised selection rule for the main variant. The KITTI-only sensitivity analysis in Table 5(f-g) is not sufficient to establish that the method is robust across the four datasets.","section":"Sec. 3.2, Eq. (2), Appendix E, Table 5(f-g)"},{"comment":"The baselines compared in the main tables are an oracle and a hand-built Cube R-CNN + Grounding DINO baseline. There are no comparisons to existing weakly supervised monocular 3D detectors that use LiDAR at training time (e.g., WeakM3D) or to fully supervised monocular detectors at the same thresholds. As a result, the reader cannot judge the practical cost-performance trade-off of removing 3D annotations. Please add at least one reference comparison using the same metrics, or explicitly state that no directly comparable open-vocabulary monocular method exists and position the results as a first baseline rather than as superiority over the broader detection literature.","section":"Sec. 4.2, Tables 1-2"}],"minor_comments":[{"comment":"The title contains a stray space in 'Open-V ocabulary,' and Section 4.1 contains the typo 'state-of-state-art'; these should be corrected.","section":"Title, Sec. 4.1"},{"comment":"The claim that LLM priors and dataset-statistics priors perform similarly is demonstrated only on KITTI in Table 5(d). Appendix B lists the priors for all datasets but does not report the corresponding mAP values; please clarify the transferability claim with per-dataset results.","section":"Appendix B, Table 5(d)"},{"comment":"Lambda is set to 5 for indoor and 10 for outdoor, but no sensitivity analysis for lambda is reported. Please add one, or explicitly justify the chosen values as robust.","section":"Appendix E, Table 5(e)"},{"comment":"The self-training experiment should state the exact pseudo-label filtering rule and confirm that no ground-truth boxes are used to select or refine the self-training labels; otherwise the improvement could partly arise from label drift and should be interpreted with caution.","section":"Appendix F, Table 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution with well-designed ablations, and I would not reject it. However, the headline claim of training 'without 3D data' is currently stronger than what the evaluation demonstrates: the lenient IoU range and the unquantified reliance on zero-shot metric depth both need direct evidence. The depth-quality concern raised in the stress-test is real and should be addressed with per-dataset depth error and pseudo-label quality numbers. I recommend major revision rather than minor because the missing measurements are load-bearing for the central claim, not merely presentational."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is the first RGB-only training pipeline for open-vocabulary monocular 3D detection, and the design is sensible; but the headline gains are measured on an unusually lenient IoU range, and the zero-shot depth assumption that drives the whole pipeline is never validated at box level on the target datasets.\n\nThe new thing is the task setup, not the components. Grounded-SAM, Unidepth, Cube R-CNN, and WeakM3D-style losses all exist; the contribution is assembling them into an auto-labeling loop and showing the refinements matter. The ablations are the paper's best evidence: adaptive erosion beats fixed (18.5 vs 16.4), PCA beats histogram (18.5 vs 16.7), LLM priors match dataset statistics (18.5 vs 18.4), and the combined box-search loss beats either term alone. That is real, controlled work, and the consistent gains across four indoor/outdoor datasets make the story credible.\n\nThe soft spots are in the protocol and the load-bearing assumption. The 3D AP is averaged over IoU 0.05-0.50, which is far more forgiving than standard KITTI (0.5/0.7) or nuScenes metrics; the paper does not report those standard thresholds. Against the oracle, the method reaches roughly 30-60% of novel-class AP, so 'significantly outperforms' the hand-built baseline is fair but should be read as a weak baseline. On depth: the paper claims no dataset overlap with Unidepth training data but gives no per-instance depth error or pseudo-box quality on the four datasets. Table 12 does report AbsRel on KITTI (4.21 vs 5.33) and shows the detector is sensitive to the depth model (18.5 vs 17.0 AP), so the stress-test concern is partly addressed for KITTI but not for nuScenes/SUN/ARKit, and never at box level. Appendix F admits quadratic error growth with distance, and the self-training far-range AP is only 4.5 to 6.0, so the limitation is real and acknowledged.\n\nOne more thing: tau1, tau2, lambda, and erosion iterations are ablated on KITTI; the paper should state clearly that these were chosen on the validation split, not the test split, before the absolute numbers are trusted. The baseline (Cube R-CNN + Grounding DINO) is also weak, and a comparison to the weakly supervised monocular method of Jiang et al. would help position the result, even if that method is not open-vocabulary.\n\nBottom line: this is a paper worth a serious referee. The setup is novel, the ablations are honest, and the limitations section is candid. A referee should push for standard metrics, per-dataset depth and box-quality numbers, and threshold-tuning disclosure, but this should go to review rather than desk rejection.","headline":"First RGB-only training pipeline for open-vocabulary monocular 3D detection, with solid ablations, but the headline numbers rest on a lenient metric and an unvalidated depth assumption that needs referee scrutiny.","tokens_in":19987,"tokens_out":3296,"would_cite":true,"duration_ms":28123,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-vocabulary monocular 3D detectors can be trained from RGB images alone by auto-labeling pseudo-LiDAR boxes.","keywords":["open-vocabulary 3D object detection","monocular 3D detection","pseudo-LiDAR","auto-labeling","large language model priors","zero-shot depth estimation","RGB-only training"],"falsifier":"Measure the per-instance depth error of Unidepth on KITTI, nuScenes, SUN RGB-D, or ARKitScenes against ground-truth depth; if the median relative error at typical object distances exceeds the level at which pseudo-box IoU with ground-truth boxes drops below roughly 0.25, the RGB-only training claim fails at its first step. A simpler variant: rerun the pipeline with ground-truth depth replacing Unidepth and check whether AP rises sharply.","tokens_in":18933,"feed_emoji":"📷","tokens_out":5117,"duration_ms":44667,"temperature":0.7,"pith_summary":"This paper tries to establish that an open-vocabulary monocular 3D object detector can be trained using only RGB images, with no LiDAR and no 3D bounding-box annotations. The proposed framework, OVM3D-Det, produces training boxes by unprojecting monocular depth into per-object pseudo-LiDAR, then calibrating those boxes with adaptive erosion and category-size priors from a large language model. If true, this removes the main cost barrier to scaling open-vocabulary 3D detection: any camera image becomes a possible training example. The reported results show gains over a 2D-open-vocabulary baseline on novel categories in four datasets, indoor and outdoor.","feed_headline":"RGB images alone can train open-vocabulary 3D detectors","feed_subtitle":"Auto-labeled pseudo-LiDAR boxes beat 2D baselines by up to 16.8 AP on novel classes.","key_machinery":"The load-bearing mechanism is an automatic labeling pipeline whose three parts fix distinct failure modes: adaptive pseudo-LiDAR erosion removes projection noise at mask boundaries by applying morphological erosion with an iteration count that scales with mask size; a bounding-box search proposes eight candidate boxes anchored at the coarse box's corners and scores them with a ray-tracing loss plus a point-ratio loss; and GPT-4-supplied category dimensions act as priors to reject or fix boxes that are too small or too large. The search is what makes occlusion survivable, because a partly seen object does not yield a tight box directly.","core_discovery":"On the paper's own terms, the central discovery is that noisy pseudo-LiDAR can be turned into usable 3D pseudo-labels by two targeted corrections: eroding object masks adaptively so depth-unprojection edge artifacts disappear, and refining candidate boxes with object priors plus a search loss that combines ray tracing and point-ratio terms. With these labels, a Cube R-CNN detector with a text-alignment head learns to localize and recognize novel categories without ever seeing a point cloud or a manual 3D box. The paper quantifies the effect on KITTI, nuScenes, SUN RGB-D, and ARKitScenes, reporting AP improvements of +6.7, +9.7, +8.5, and +16.8 over the strongest RGB-based baseline on novel classes.","pith_inferences":["Beyond the paper, the quadratic growth of depth error with distance implies this auto-labeling approach is strongest for close-range perception such as robotics and augmented reality, and weakest for far-range driving; evaluating on near-only subsets should show the gap narrowing.","Although not explored here, the same pipeline could use temporal or multi-view consistency from video frames to correct depth drift and occlusion before box search.","The success of LLM priors suggests that category-level shape priors could also be drawn from text embeddings or parametric CAD models, removing the need to query a specific LLM API."],"forward_implications":["Because training needs only RGB images, the approach can scale to internet-scale image collections, and the paper shows detection AP keeps rising as training data volume grows.","The category priors can come from an LLM or from dataset statistics with similar results, so commonsense textual knowledge is enough to anchor box dimensions.","Detection quality tracks depth-estimation quality: swapping Unidepth for Metric3D on KITTI moves AP from 17.0 to 18.5, so better depth models will transfer directly into this pipeline.","Self-training on the model's own confident outputs further improves AP, especially for far objects, which means the label noise can be iteratively reduced.","The open-vocabulary setup needs no base/novel split for training; splits are used only for evaluation."],"supporting_citations":[{"why":"Supplies zero-shot metric depth maps that are unprojected into pseudo-LiDAR for every image.","marker":"[40]"},{"why":"Detects and segments arbitrary text-prompted classes, producing the masks and class labels that seed pseudo boxes.","marker":"[47]"},{"why":"Provides the monocular 3D detector architecture trained on the auto-generated pseudo labels.","marker":"[3]"},{"why":"Supplies category-level typical object dimensions used as priors to judge and correct candidate boxes.","marker":"[1]"},{"why":"Introduces the depth-to-point-cloud unprojection that converts estimated depth into pseudo-LiDAR.","marker":"[57]"},{"why":"Source of the orientation histogram and ray-tracing loss adapted for box search.","marker":"[38]"},{"why":"Point-cloud open-vocabulary detector used as the comparison baseline for pseudo-LiDAR-trained models.","marker":"[30]"},{"why":"Open-set 2D detector that powers the baseline pipeline and the Grounding DINO component of the 2D model.","marker":"[28]"}],"fun_headline_variants":["No 3D data needed: RGB-only training for open-vocabulary 3D detection","Open-vocabulary 3D detectors trained from 2D images alone","Pseudo-LiDAR labels get fixed: RGB-only 3D detection works","Erase depth artifacts, refine boxes: RGB-only 3D detection","Improve 3D detection without LiDAR: +16.8 AP on novel classes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire label-generation chain assumes the pretrained depth model predicts metric, scale-correct distances on the target cameras and scenes even though those datasets were not part of its training data, and the paper's own appendix notes depth errors grow quadratically with distance.","fun_headline_variants_meta":{"raw":{"variants":["No 3D data needed: RGB-only training for open-vocabulary 3D detection","Open-vocabulary 3D detectors trained from 2D images alone","Pseudo-LiDAR labels get fixed: RGB-only 3D detection works","Erase depth artifacts, refine boxes: RGB-only 3D detection","Improve 3D detection without LiDAR: +16.8 AP on novel classes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000666,"raw_usage":{"total_tokens":3062,"prompt_tokens":988,"completion_tokens":2074,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":1966}},"tokens_in":604,"tokens_out":2074,"duration_ms":12308,"temperature":1.0,"reasoning_tokens":1966,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:03:34.261832+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the per-instance depth error of Unidepth on KITTI, nuScenes, SUN RGB-D, or ARKitScenes against ground-truth depth; if the median relative error at typical object distances exceeds the level at which pseudo-box IoU with ground-truth boxes drops below roughly 0.25, the RGB-only training claim fails at its first step. A simpler variant: rerun the pipeline with ground-truth depth replacing Unidepth and check whether AP rises sharply.","supporting_citations":[{"cited_title":"Unidepth: Universal monocular metric depth estimation","cited_arxiv_id":null,"evidence_quote":"Supplies zero-shot metric depth maps that are unprojected into pseudo-LiDAR for every image."},{"cited_title":"Omni3d: A large benchmark and model for 3d object detection in the wild","cited_arxiv_id":null,"evidence_quote":"Provides the monocular 3D detector architecture trained on the auto-generated pseudo labels."},{"cited_title":"Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving","cited_arxiv_id":null,"evidence_quote":"Introduces the depth-to-point-cloud unprojection that converts estimated depth into pseudo-LiDAR."},{"cited_title":"Weakm3d: Towards weakly supervised monocular 3d object detection","cited_arxiv_id":null,"evidence_quote":"Source of the orientation histogram and ray-tracing loss adapted for box search."},{"cited_title":"Open-vocabulary point-cloud object detection without 3d annotation","cited_arxiv_id":null,"evidence_quote":"Point-cloud open-vocabulary detector used as the comparison baseline for pseudo-LiDAR-trained models."},{"cited_title":"Grounding dino: Marrying dino with grounded pre-training for open-set object detection","cited_arxiv_id":null,"evidence_quote":"Open-set 2D detector that powers the baseline pipeline and the Grounding DINO component of the 2D model."}],"review_version":1}