Pith. sign in

REVIEW 4 major objections 4 minor 76 references

Training an Open-Vocabulary Monocular 3D Object Detection Model without 3D Data

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Open-vocabulary monocular 3D detectors can be trained from RGB images alone by auto-labeling pseudo-LiDAR boxes.

desk verdict First RGB-only training pipeline for open-vocabulary monocular 3D detection, with solid ablations, but the headline numbers rest on a lenient metric and an unvalidated depth assumption that needs referee scrutiny. read the letter →

arxiv 2411.15657 v1 pith:WL4W4CJP submitted 2024-11-23 cs.CV

Rui Huang , Henry Zheng , Yan Wang , Zhuofan Xia , Marco Pavone , Gao Huang This is my paper · ORCID
classification cs.CV
keywords open-vocabulary3Dobjectdetectionmonocularpseudo-LiDARauto-labelinglargelanguagemodelpriorszero-shotdepthestimationRGB-onlytraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that an open-vocabulary monocular 3D object detector can be trained using only RGB images, with no LiDAR and no 3D bounding-box annotations. The proposed framework, OVM3D-Det, produces training boxes by unprojecting monocular depth into per-object pseudo-LiDAR, then calibrating those boxes with adaptive erosion and category-size priors from a large language model. If true, this removes the main cost barrier to scaling open-vocabulary 3D detection: any camera image becomes a possible training example. The reported results show gains over a 2D-open-vocabulary baseline on novel categories in four datasets, indoor and outdoor.

What carries the argument

The load-bearing mechanism is an automatic labeling pipeline whose three parts fix distinct failure modes: adaptive pseudo-LiDAR erosion removes projection noise at mask boundaries by applying morphological erosion with an iteration count that scales with mask size; a bounding-box search proposes eight candidate boxes anchored at the coarse box's corners and scores them with a ray-tracing loss plus a point-ratio loss; and GPT-4-supplied category dimensions act as priors to reject or fix boxes that are too small or too large. The search is what makes occlusion survivable, because a partly seen object does not yield a tight box directly.

What would settle it

Measure the per-instance depth error of Unidepth on KITTI, nuScenes, SUN RGB-D, or ARKitScenes against ground-truth depth; if the median relative error at typical object distances exceeds the level at which pseudo-box IoU with ground-truth boxes drops below roughly 0.25, the RGB-only training claim fails at its first step. A simpler variant: rerun the pipeline with ground-truth depth replacing Unidepth and check whether AP rises sharply.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that noisy pseudo-LiDAR can be turned into usable 3D pseudo-labels by two targeted corrections: eroding object masks adaptively so depth-unprojection edge artifacts disappear, and refining candidate boxes with object priors plus a search loss that combines ray tracing and point-ratio terms. With these labels, a Cube R-CNN detector with a text-alignment head learns to localize and recognize novel categories without ever seeing a point cloud or a manual 3D box. The paper quantifies the effect on KITTI, nuScenes, SUN RGB-D, and ARKitScenes, reporting AP improvements of +6.7, +9.7, +8.5, and +16.8 over the strongest RGB-based baseline on novel classes.

Load-bearing premise

The entire label-generation chain assumes the pretrained depth model predicts metric, scale-correct distances on the target cameras and scenes even though those datasets were not part of its training data, and the paper's own appendix notes depth errors grow quadratically with distance.

Editorial extensions

If this is right

  • Because training needs only RGB images, the approach can scale to internet-scale image collections, and the paper shows detection AP keeps rising as training data volume grows.
  • The category priors can come from an LLM or from dataset statistics with similar results, so commonsense textual knowledge is enough to anchor box dimensions.
  • Detection quality tracks depth-estimation quality: swapping Unidepth for Metric3D on KITTI moves AP from 17.0 to 18.5, so better depth models will transfer directly into this pipeline.
  • Self-training on the model's own confident outputs further improves AP, especially for far objects, which means the label noise can be iteratively reduced.
  • The open-vocabulary setup needs no base/novel split for training; splits are used only for evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the quadratic growth of depth error with distance implies this auto-labeling approach is strongest for close-range perception such as robotics and augmented reality, and weakest for far-range driving; evaluating on near-only subsets should show the gap narrowing.
  • Although not explored here, the same pipeline could use temporal or multi-view consistency from video frames to correct depth drift and occlusion before box search.
  • The success of LLM priors suggests that category-level shape priors could also be drawn from text embeddings or parametric CAD models, removing the need to query a specific LLM API.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes OVM3D-Det, a pipeline that trains an open-vocabulary monocular 3D object detector using only RGB images, without target-domain LiDAR or 3D annotations. The method uses Grounded-SAM for open-vocabulary 2D detection and segmentation, Unidepth for zero-shot metric depth estimation, back-projects instance masks into pseudo-LiDAR, applies adaptive erosion, estimates orientation via PCA, and refines boxes using LLM-provided class size priors and a box-search loss. The resulting pseudo 3D labels supervise a Cube R-CNN detector with a text-alignment head. Experiments on KITTI, nuScenes, SUN RGB-D, and ARKitScenes compare against an oracle, a hand-built Cube R-CNN + Grounding DINO baseline, and point-cloud open-vocabulary methods, reporting novel-category improvements of +6.7 to +16.8 AP. Ablation studies isolate the contributions of each component.

Significance. If the central claim holds, this is a useful contribution: it demonstrates a practical way to bootstrap open-vocabulary monocular 3D detection without target-domain 3D annotations, which could enable scaling to internet-scale RGB data. The paper has clear strengths: controlled ablations in Table 5 support each design choice; the evaluation spans four indoor and outdoor datasets; Appendix B shows LLM priors are comparable to dataset statistics; and the limitations section is honest about distance-dependent depth errors. The main uncertainties are the lenient evaluation protocol, the unquantified reliance on zero-shot metric depth, and the unclear amount of validation supervision used to tune hyperparameters. These are fixable with additional experiments and disclosure, so the contribution is defensible but needs revision.

major comments (4)
  1. [Sec. 4.1, Tables 1-2] The main results report only mAP3D averaged over IoU thresholds from 0.05 to 0.50. At an IoU threshold of 0.05, almost any coarsely placed box counts as a true positive, and this protocol is not standard for KITTI or nuScenes, which typically use AP at 0.25/0.5 or official difficulty-based metrics. Because the central claim is that the auto-labeled pipeline 'significantly outperforms' baselines, the authors should report AP at standard thresholds (e.g., AP3D at IoU 0.25 and 0.50) and per-class AP for the novel categories. This is particularly important because the reported gaps may shrink considerably at stricter IoU thresholds.
  2. [Sec. 3.1, Eq. (1), Appendix F/H] The entire pseudo-label chain assumes Unidepth's zero-shot depth estimates are metric and scale-correct on all target datasets, yet the paper never reports per-instance depth error or pseudo-box quality on KITTI, nuScenes, SUN RGB-D, or ARKitScenes. Appendix F and Appendix H concede that depth errors grow quadratically with distance and that distant objects are missed, which is precisely the regime where many novel outdoor categories live. Table 12 swaps Metric3D for Unidepth (17.0 vs 18.5 AP), confirming sensitivity to depth quality, but it does not measure per-instance scale errors. Please add depth error metrics (e.g., AbsRel and delta-1 against available depth sensors) and pseudo-label quality metrics (recall, precision, and IoU against ground-truth boxes). Without these measurements, the claim of training 'using only RGB images' is not fully supported.
  3. [Sec. 3.2, Eq. (2), Appendix E, Table 5(f-g)] The hyperparameters tau1, tau2, lambda, the number of erosion iterations, and the mask-size threshold are free parameters, and the paper does not disclose how they were selected. If any of them were chosen by maximizing mAP3D on KITTI validation using ground-truth 3D boxes, then the method is not strictly 'without 3D data,' because validation supervision leaks into the auto-labeling configuration. Please disclose the exact selection procedure, report sensitivity on at least one additional dataset, and, ideally, provide a fully unsupervised selection rule for the main variant. The KITTI-only sensitivity analysis in Table 5(f-g) is not sufficient to establish that the method is robust across the four datasets.
  4. [Sec. 4.2, Tables 1-2] The baselines compared in the main tables are an oracle and a hand-built Cube R-CNN + Grounding DINO baseline. There are no comparisons to existing weakly supervised monocular 3D detectors that use LiDAR at training time (e.g., WeakM3D) or to fully supervised monocular detectors at the same thresholds. As a result, the reader cannot judge the practical cost-performance trade-off of removing 3D annotations. Please add at least one reference comparison using the same metrics, or explicitly state that no directly comparable open-vocabulary monocular method exists and position the results as a first baseline rather than as superiority over the broader detection literature.
minor comments (4)
  1. [Title, Sec. 4.1] The title contains a stray space in 'Open-V ocabulary,' and Section 4.1 contains the typo 'state-of-state-art'; these should be corrected.
  2. [Appendix B, Table 5(d)] The claim that LLM priors and dataset-statistics priors perform similarly is demonstrated only on KITTI in Table 5(d). Appendix B lists the priors for all datasets but does not report the corresponding mAP values; please clarify the transferability claim with per-dataset results.
  3. [Appendix E, Table 5(e)] Lambda is set to 5 for indoor and 10 for outdoor, but no sensitivity analysis for lambda is reported. Please add one, or explicitly justify the chosen values as robust.
  4. [Appendix F, Table 6] The self-training experiment should state the exact pseudo-label filtering rule and confirm that no ground-truth boxes are used to select or refine the self-training labels; otherwise the improvement could partly arise from label drift and should be interpreted with caution.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the pseudo-label pipeline is an external empirical process evaluated against held-out ground truth, and the depth-quality and tuning concerns are correctness risks rather than circularity.

full rationale

The paper's contribution is an empirical auto-labeling pipeline, not an analytical derivation, and no load-bearing step reduces by construction to its own inputs. Equation (1) is the standard pinhole unprojection z = D(u,v), x = (u-cU)z/fU, y = (v-cV)z/fV, cited to Pseudo-LiDAR [57]; although a co-author overlaps with [57], the formula is elementary, parameter-free, and externally established, so the citation is not circular. The downstream components—adaptive erosion, ground-plane fitting, PCA orientation, LLM-prior box search, and ray-tracing/point-ratio losses—operate on pseudo-LiDAR and are ultimately evaluated against held-out ground truth on KITTI, nuScenes, SUN RGB-D, and ARKitScenes. The default LLM dimension priors come from GPT-4 and are independent of the test ground-truth dimensions; Table 5(d) shows dataset-statistic priors give similar results, but those are not the default and are used only as sanity checks. Hyperparameters such as tau1, tau2, lambda, and erosion iterations are selected via ablations on KITTI AP, which is standard tuning rather than a fitted-parameter-renamed-as-prediction, and the reported detection outputs are not equal to the tuning signal by construction. The main vulnerability—zero-shot metric depth from Unidepth on datasets absent from its training set—is a genuine correctness risk acknowledged in Sec. 3.1 and Appendix F, where the paper states that depth errors increase quadratically with distance; this affects validity but is not circularity, because pseudo-labels are produced from external depth predictions and then compared against external ground truth. Self-citations in related work ([19], [35], [57], [64], [68]) are not load-bearing arguments that presuppose the present claim. Therefore the derivation chain is self-contained in the sense relevant to circularity.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The method relies on two strong off-the-shelf models (Grounded-SAM, Unidepth), a calibrated pinhole model, ground-parallel boxes, and class-size priors injected by GPT-4. No new physical entities are introduced. The main burden is the five hand-set or benchmark-tuned parameters (priors, tau1, tau2, lambda, erosion settings) that directly affect the quality of the pseudo labels.

free parameters (5)
  • dimension priors from LLM = Per class, e.g., car [1.8, 1.5, 4.5] for KITTI (Tab. 8)
    Used to decide if a coarse box is reasonable and to search for the optimal box. The paper shows dataset-statistic priors give similar AP, so these priors are effectively class-specific inputs to the label generator.
  • lower threshold tau1 = 0.8
    Ablated in Tab. 5(f); chosen by KITTI AP maximization. The model depends on this threshold to decide when a coarse box is valid.
  • upper threshold tau2 = 1.2
    Ablated in Tab. 5(g); chosen on KITTI. Together with tau1 it defines the reasonableness range for box dimensions.
  • box-search weight lambda = 5 for indoor, 10 for outdoor
    Set per scene type in Sec. E to balance ray-tracing loss and point-ratio loss; not ablated across values.
  • erosion iterations and size threshold = Outdoor: 4 iterations if mask width >10 px else 2; indoor: 12 and 2
    Hand-set in Sec. E. The adaptive erosion module depends on these numbers and on the pixel-width cutoff, which are chosen by the authors.
assumptions (6)
  • domain assumption Grounded-SAM produces correct object masks for arbitrary text prompts
    Step 1 of the method; the open-vocabulary 2D detector and segmentor must find and mask target objects. Failure cases in Fig. 8, such as detecting a reflection as a vehicle, show this assumption can break.
  • domain assumption Unidepth provides zero-shot metric depth on target datasets with no overlap
    Depth estimation is the backbone of pseudo-LiDAR. The paper states no overlap, but relies entirely on the pre-trained model's generalization to unseen scenes.
  • standard math Pinhole camera model with known intrinsics holds (Eq. 1)
    Unprojection of pixels via focal lengths and center is standard for calibrated cameras; the method assumes these intrinsics are available.
  • domain assumption All objects' 3D boxes are parallel to the ground plane
    Sec. 3.2 assumes ground-parallel boxes to estimate orientation and project points. This may fail for wall-mounted objects like shelves, mirrors, or pictures, which are common in indoor scenes.
  • ad hoc to paper Boxes whose dimensions lie within tau1 to tau2 times class priors are valid; otherwise search yields a better box
    The box search relies on this reasonableness criterion, and the thresholds are fixed by KITTI ablation rather than derived from theory.
  • domain assumption Ray-tracing plus point-ratio loss selects the box that best explains visible pseudo-LiDAR
    Borrowed from WeakM3D and augmented with point-ratio; assumes visible surface points concentrate on the camera-near side of the true box, which need not hold for heavily occluded or transparent objects.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training an Open-Vocabulary Monocular 3D Object Detection Model without 3D Data." pith.science (2026). https://pith.science/paper/WL4W4CJP

@misc{pith2026241115657,
  author       = {Pith},
  title        = {Pith review of: Training an Open-Vocabulary Monocular 3D Object Detection Model without 3D Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WL4W4CJP}},
  note         = {Machine review of arXiv:2411.15657}
}
read the original abstract

Open-vocabulary 3D object detection has recently attracted considerable attention due to its broad applications in autonomous driving and robotics, which aims to effectively recognize novel classes in previously unseen domains. However, existing point cloud-based open-vocabulary 3D detection models are limited by their high deployment costs. In this work, we propose a novel open-vocabulary monocular 3D object detection framework, dubbed OVM3D-Det, which trains detectors using only RGB images, making it both cost-effective and scalable to publicly available data. Unlike traditional methods, OVM3D-Det does not require high-precision LiDAR or 3D sensor data for either input or generating 3D bounding boxes. Instead, it employs open-vocabulary 2D models and pseudo-LiDAR to automatically label 3D objects in RGB images, fostering the learning of open-vocabulary monocular 3D detectors. However, training 3D models with labels directly derived from pseudo-LiDAR is inadequate due to imprecise boxes estimated from noisy point clouds and severely occluded objects. To address these issues, we introduce two innovative designs: adaptive pseudo-LiDAR erosion and bounding box refinement with prior knowledge from large language models. These techniques effectively calibrate the 3D labels and enable RGB-only training for 3D detectors. Extensive experiments demonstrate the superiority of OVM3D-Det over baselines in both indoor and outdoor scenarios. The code will be released.

Figures

Figures reproduced from arXiv: 2411.15657 by the authors.

Figure 1
Figure 1. Comparison between point cloud-based and image-based open-vocabulary 3D object [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison between LiDAR data and pseudo-LiDAR. Although pseudo-LiDAR is much [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The overall framework of OVM3D-Det. Step ①: Generate per-instance pseudo-LiDAR. Step ②: Apply an adaptive erosion process to remove artifacts and noises. Step ③: Estimate the orientation. Step ④: Tightly fit a box and utilize object priors to assess the estimated box; if deemed unreasonable, search for the optimal box. Step ⑤: Train the model with pseudo labels. 3 Method Our goal is to train an open-vocabulary monoc… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Ray tracing loss and point ratio loss. In our experiments, we find that GPT-4 can of￾fer reliable information regarding the usual sizes of objects within specific categories, including their width, length, and height. When we re￾place the object sizes predicted by GPT-…
Figure 6
Figure 6. Figure 6: Effect of training data. As the volume of training data grows, we consistently see performance im￾provements. potential of our approach to be transferred to a wide range of data. We also test the sensitivity of the lower threshold τ1 and the upper threshold τ2 used for…
Figure 7
Figure 7. Figure 7: Diagram of the erosion process of Mi . To perform the erosion operation Mi ⊖ B, first place the structuring element B over each pixel of Mi . If every foreground pixel of B aligns with a foreground pixel of Mi , then the central pixel of B in Mi will retain the value o…
Figure 8
Figure 8. Figure 8: Failure case. We present some failure cases of OVM3D-Det in [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 67 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 2, 6, 18

    arXiv 2023

  2. [2]

    Arkitscenes: A diverse real- world dataset for 3d indoor scene understanding using mobile rgb-d data

    Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, et al. Arkitscenes: A diverse real- world dataset for 3d indoor scene understanding using mobile rgb-d data. arXiv preprint arXiv:2111.08897, 2021. 7, 17, 18

    arXiv 2021

  3. [3]

    Omni3d: A large benchmark and model for 3d object detection in the wild

    Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3d: A large benchmark and model for 3d object detection in the wild. In CVPR,

    work page

  4. [4]

    M3d-rpn: Monocular 3d region proposal network for object detection

    Garrick Brazil and Xiaoming Liu. M3d-rpn: Monocular 3d region proposal network for object detection. In ICCV, 2019. 2, 3

    work page 2019

  5. [5]

    nuscenes: A multimodal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020. 7, 17, 18

    work page 2020

  6. [6]

    Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection

    Yang Cao, Zeng Yihan, Hang Xu, and Dan Xu. Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection. In NeurIPS, 2023. 1, 3, 8

    work page 2023

  7. [7]

    Seal: Self-supervised embodied active learning using exploration and 3d consistency

    Devendra Singh Chaplot, Murtaza Dalal, Saurabh Gupta, Jitendra Malik, and Russ R Salakhut- dinov. Seal: Self-supervised embodied active learning using exploration and 3d consistency. In NeurIPS, 2021. 1

    work page 2021

  8. [8]

    Clip2scene: Towards label-efficient 3d scene understanding by clip

    Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. In CVPR, 2023. 3

    work page 2023

Show all 76 references
  1. [9]

    Monocular 3d object detection for autonomous driving

    Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, and Raquel Urtasun. Monocular 3d object detection for autonomous driving. In CVPR, 2016. 2, 3

    2016

  2. [10]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017. 1

    2017

  3. [11]

    Pla: Language-driven open-vocabulary 3d scene understanding

    Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Pla: Language-driven open-vocabulary 3d scene understanding. In CVPR, 2023. 3

    2023

  4. [12]

    Find n’propagate: Open- vocabulary 3d object detection in urban environments

    Djamahl Etchegaray, Zi Huang, Tatsuya Harada, and Yadan Luo. Find n’propagate: Open- vocabulary 3d object detection in urban environments. arXiv preprint arXiv:2403.13556, 2024. 3

    2024 arXiv

  5. [13]

    Rangedet: In defense of range view for lidar-based 3d object detection

    Lue Fan, Xuan Xiong, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Rangedet: In defense of range view for lidar-based 3d object detection. In ICCV, 2021. 1

    2021

  6. [14]

    Are we ready for autonomous driving? the kitti vision benchmark suite

    Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012. 1, 7, 17, 18

    2012

  7. [15]

    Deepmix: mobility-aware, lightweight, and hybrid 3d object detection for headsets

    Yongjie Guan, Xueyu Hou, Nan Wu, Bo Han, and Tao Han. Deepmix: mobility-aware, lightweight, and hybrid 3d object detection for headsets. In MobiSys, 2022. 1

    2022

  8. [16]

    Towards zero-shot scale-aware monocular depth estimation

    Vitor Guizilini, Igor Vasiljevic, Dian Chen, Rare s, Ambrus, , and Adrien Gaidon. Towards zero-shot scale-aware monocular depth estimation. In ICCV, 2023. 2, 3

    2023

  9. [17]

    Live semantic 3d perception for immersive augmented reality

    Lei Han, Tian Zheng, Yinheng Zhu, Lan Xu, and Lu Fang. Live semantic 3d perception for immersive augmented reality. TVCG, 2020. 1

    2020

  10. [18]

    Weakly supervised 3d object detection with multi-stage generalization

    Jiawei He, Yuqi Wang, Yuntao Chen, and Zhaoxiang Zhang. Weakly supervised 3d object detection with multi-stage generalization. arXiv preprint arXiv:2306.05418, 2023. 3

    2023 arXiv

  11. [19]

    Segment3d: Learning fine-grained class-agnostic 3d segmentation without manual labels

    Rui Huang, Songyou Peng, Ayca Takmaz, Federico Tombari, Marc Pollefeys, Shiji Song, Gao Huang, and Francis Engelmann. Segment3d: Learning fine-grained class-agnostic 3d segmentation without manual labels. In ECCV, 2024. 3

    2024

  12. [20]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021. 3 11

    2021

  13. [21]

    Weakly supervised monocular 3d detection with a single-view image

    Xueying Jiang, Sheng Jin, Lewei Lu, Xiaoqin Zhang, and Shijian Lu. Weakly supervised monocular 3d detection with a single-view image. In CVPR, 2024. 3

    2024

  14. [22]

    A 3d-deep-learning-based aug- mented reality calibration method for robotic environments using depth sensor data

    Linh Kästner, Vlad Catalin Frasineanu, and Jens Lambrecht. A 3d-deep-learning-based aug- mented reality calibration method for robotic environments using depth sensor data. In ICRA,

  15. [23]

    Lerf: Language embedded radiance fields

    Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In ICCV, 2023. 3

    2023

  16. [24]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In ICCV, 2023. 3, 4, 18

    2023

  17. [25]

    Lidar r-cnn: An efficient and universal 3d object detector

    Zhichao Li, Feng Wang, and Naiyan Wang. Lidar r-cnn: An efficient and universal 3d object detector. In CVPR, 2021. 1

    2021

  18. [26]

    Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers

    Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. In ECCV, 2022. 1

    2022

  19. [27]

    Feature pyramid networks for object detection

    Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017. 17

    2017

  20. [28]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ECCV, 2024. 4, 8, 18

    2024

  21. [29]

    Ground-aware monocular 3d object detection for autonomous driving

    Yuxuan Liu, Yuan Yixuan, and Ming Liu. Ground-aware monocular 3d object detection for autonomous driving. RA-L, 2021. 2, 3

    2021

  22. [30]

    Open-vocabulary point-cloud object detection without 3d annotation

    Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. Open-vocabulary point-cloud object detection without 3d annotation. In CVPR, 2023. 1, 3, 8, 9

    2023

  23. [31]

    Driveworld: 4d pre-trained scene understanding via world models for autonomous driving

    Chen Min, Dawei Zhao, Liang Xiao, Jian Zhao, Xinli Xu, Zheng Zhu, Lei Jin, Jianshu Li, Yulan Guo, Junliang Xing, et al. Driveworld: 4d pre-trained scene understanding via world models for autonomous driving. In CVPR, 2024. 1

    2024

  24. [32]

    An end-to-end transformer model for 3d object detection

    Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. In ICCV, 2021. 1

    2021

  25. [33]

    3d bounding box estimation using deep learning and geometry

    Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka. 3d bounding box estimation using deep learning and geometry. In CVPR, 2017. 2, 3

    2017

  26. [34]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. TMLR, 2023. 3

    2023

  27. [35]

    3d object detection with pointformer

    Xuran Pan, Zhuofan Xia, Shiji Song, Li Erran Li, and Gao Huang. 3d object detection with pointformer. In CVPR, 2021. 1

    2021

  28. [36]

    Is pseudo-lidar needed for monocular 3d object detection? In ICCV, 2021

    Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon. Is pseudo-lidar needed for monocular 3d object detection? In ICCV, 2021. 3, 8

    2021

  29. [37]

    Learning occupancy for monocular 3d object detection

    Liang Peng, Junkai Xu, Haoran Cheng, Zheng Yang, Xiaopei Wu, Wei Qian, Wenxiao Wang, Boxi Wu, and Deng Cai. Learning occupancy for monocular 3d object detection. In CVPR,

  30. [38]

    Weakm3d: Towards weakly supervised monocular 3d object detection

    Liang Peng, Senbo Yan, Boxi Wu, Zheng Yang, Xiaofei He, and Deng Cai. Weakm3d: Towards weakly supervised monocular 3d object detection. In ICLR, 2022. 3, 5, 6

    2022

  31. [39]

    Openscene: 3d scene understanding with open vocabularies

    Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In CVPR, 2023. 3

    2023

  32. [40]

    Unidepth: Universal monocular metric depth estimation

    Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In CVPR, 2024. 2, 3, 4, 7, 18

    2024

  33. [41]

    Embodied language grounding with 3d visual feature representations

    Mihir Prabhudesai, Hsiao-Yu Fish Tung, Syed Ashar Javed, Maximilian Sieb, Adam W Harley, and Katerina Fragkiadaki. Embodied language grounding with 3d visual feature representations. In CVPR, 2020. 1 12

    2020

  34. [42]

    Deep hough voting for 3d object detection in point clouds

    Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In ICCV, 2019. 1

    2019

  35. [43]

    End-to-end pseudo-lidar for image-based 3d object detection

    Rui Qian, Divyansh Garg, Yan Wang, Yurong You, Serge Belongie, Bharath Hariharan, Mark Campbell, Kilian Q Weinberger, and Wei-Lun Chao. End-to-end pseudo-lidar for image-based 3d object detection. In CVPR, 2020. 3

    2020

  36. [44]

    Monoground: Detecting monocular 3d objects from the ground

    Zequn Qin and Xi Li. Monoground: Detecting monocular 3d objects from the ground. In CVPR,

  37. [45]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 3

    2021

  38. [46]

    Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer

    René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. TPAMI,

  39. [47]

    Grounded sam: Assembling open-world models for diverse visual tasks

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159, 2024. 4, 5, 7, 18

    2024 arXiv

  40. [48]

    Imvoxelnet: Image to voxels projection for monocular and multi-view general-purpose 3d object detection

    Danila Rukhovich, Anna V orontsova, and Anton Konushin. Imvoxelnet: Image to voxels projection for monocular and multi-view general-purpose 3d object detection. In WACV, 2022. 1

    2022

  41. [49]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015. 17

    2015

  42. [50]

    V-DETR: DETR with vertex relative position encoding for 3d object detection

    Yichao Shen, Zigang Geng, Yuhui Yuan, Yutong Lin, Ze Liu, Chunyu Wang, Han Hu, Nanning Zheng, and Baining Guo. V-DETR: DETR with vertex relative position encoding for 3d object detection. In ICLR, 2024. 1

    2024

  43. [51]

    Pv-rcnn: Point-voxel feature set abstraction for 3d object detection

    Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-rcnn: Point-voxel feature set abstraction for 3d object detection. In CVPR, 2020. 1

    2020

  44. [52]

    Sun rgb-d: A rgb-d scene understand- ing benchmark suite

    Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understand- ing benchmark suite. In CVPR, 2015. 1, 7, 17, 18

    2015

  45. [53]

    Openmask3d: Open-vocabulary 3d instance segmentation

    Ayça Takmaz, Elisabetta Fedele, Robert W Sumner, Marc Pollefeys, Federico Tombari, and Francis Engelmann. Openmask3d: Open-vocabulary 3d instance segmentation. In NeurIPS,

  46. [54]

    Weakly supervised monocular 3d object detection using multi-view projection and direction consistency

    Runzhou Tao, Wencheng Han, Zhongying Qiu, Cheng-Zhong Xu, and Jianbing Shen. Weakly supervised monocular 3d object detection using multi-view projection and direction consistency. In CVPR, 2023. 3

    2023

  47. [55]

    Exploring object-centric temporal modeling for efficient multi-view 3d object detection

    Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. In ICCV, 2023. 1

    2023

  48. [56]

    Dust3r: Geometric 3d vision made easy

    Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In CVPR, 2023. 2, 3

    2023

  49. [57]

    Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving

    Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In CVPR, 2019. 2, 3, 4

    2019

  50. [58]

    Plumenet: Efficient 3d object detection from stereo images

    Yan Wang, Bin Yang, Rui Hu, Ming Liang, and Raquel Urtasun. Plumenet: Efficient 3d object detection from stereo images. In IROS, 2021. 3

    2021

  51. [59]

    Ov-uni3detr: Towards unified open-vocabulary 3d object detection via cycle-modality propagation

    Zhenyu Wang, Yali Li, Taichi Liu, Hengshuang Zhao, and Shengjin Wang. Ov-uni3detr: Towards unified open-vocabulary 3d object detection via cycle-modality propagation. In ECCV,

  52. [60]

    Monocular 3d object detection with pseudo-lidar point cloud

    Xinshuo Weng and Kris Kitani. Monocular 3d object detection with pseudo-lidar point cloud. In ICCVW, 2019. 8

    2019

  53. [61]

    Embodied task planning with large language models

    Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, and Haibin Yan. Embodied task planning with large language models. arXiv preprint arXiv:2307.01848, 2023. 1

    2023 arXiv

  54. [62]

    Mlcvnet: Multi-level context votenet for 3d object detection

    Qian Xie, Yu-Kun Lai, Jing Wu, Zhoutao Wang, Yiming Zhang, Kai Xu, and Jun Wang. Mlcvnet: Multi-level context votenet for 3d object detection. In CVPR, 2020. 1 13

    2020

  55. [63]

    Monocd: Monocular 3d object detection with complementary depths

    Longfei Yan, Pei Yan, Shengzhou Xiong, Xuanyu Xiang, and Yihua Tan. Monocd: Monocular 3d object detection with complementary depths. In CVPR, 2024. 3

    2024

  56. [64]

    Bevformer v2: Adapting modern image backbones to bird’s-eye-view recognition via perspective supervision

    Chenyu Yang, Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, et al. Bevformer v2: Adapting modern image backbones to bird’s-eye-view recognition via perspective supervision. In CVPR, 2023. 1

    2023

  57. [65]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024. 2, 3

    2024

  58. [66]

    Monocular 3d object detection via feature domain adaptation

    Xiaoqing Ye, Liang Du, Yifeng Shi, Yingying Li, Xiao Tan, Jianfeng Feng, Errui Ding, and Shilei Wen. Monocular 3d object detection via feature domain adaptation. In ECCV, 2020. 8

    2020

  59. [67]

    Metric3d: Towards zero-shot metric 3d prediction from a single image

    Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, and Chunhua Shen. Metric3d: Towards zero-shot metric 3d prediction from a single image. In ICCV, 2023. 2, 3, 17, 18

    2023

  60. [68]

    Pseudo-lidar++: Accurate depth for 3d object detection in autonomous driving

    Yurong You, Yan Wang, Wei-Lun Chao, Divyansh Garg, Geoff Pleiss, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Pseudo-lidar++: Accurate depth for 3d object detection in autonomous driving. In ICLR, 2020. 3

    2020

  61. [69]

    Deep layer aggregation

    Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In CVPR, 2018. 17

    2018

  62. [70]

    Clip2: Contrastive language-image-point pretraining from real-world point cloud data

    Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit-Yan Yeung, Zhen Yang, Xiaodan Liang, and Hang Xu. Clip2: Contrastive language-image-point pretraining from real-world point cloud data. In CVPR, 2023. 3

    2023

  63. [71]

    Opensight: A simple open-vocabulary framework for lidar-based object detection

    Hu Zhang, Jianhua Xu, Tao Tang, Haiyang Sun, Xin Yu, Zi Huang, and Kaicheng Yu. Opensight: A simple open-vocabulary framework for lidar-based object detection. In ECCV, 2024. 1, 3

    2024

  64. [72]

    Decoupled pseudo-labeling for semi-supervised monocular 3d object detection

    Jiacheng Zhang, Jiaming Li, Xiangru Lin, Wei Zhang, Xiao Tan, Junyu Han, Errui Ding, Jingdong Wang, and Guanbin Li. Decoupled pseudo-labeling for semi-supervised monocular 3d object detection. In CVPR, 2024. 3

    2024

  65. [73]

    A real 3d embodied dataset for robotic active visual learning

    Qianfan Zhao, Lu Zhang, Lingxi Wu, Hong Qiao, and Zhiyong Liu. A real 3d embodied dataset for robotic active visual learning. RA-L, 2022. 1

    2022

  66. [74]

    Mogde: Boosting mobile monocular 3d object detection with ground depth estimation

    Yunsong Zhou, Quan Liu, Hongzi Zhu, Yunzhe Li, Shan Chang, and Minyi Guo. Mogde: Boosting mobile monocular 3d object detection with ground depth estimation. In NeurIPS,

  67. [75]

    Object2scene: Putting objects in context for open-vocabulary 3d detection

    Chenming Zhu, Wenwei Zhang, Tai Wang, Xihui Liu, and Kai Chen. Object2scene: Putting objects in context for open-vocabulary 3d detection. arXiv preprint arXiv:2309.09456, 2023. 1, 3

    2023 arXiv

  68. [76]

    Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning

    Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng, Zipeng Qin, Shanghang Zhang, and Peng Gao. Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning. In ICCV, 2023. 1 14 Appendix A Category Splits The category splits for all datasets are shown in Ta...

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.