Pith. sign in

REVIEW 4 major objections 6 minor 56 references

Towards Open-Vocabulary Multimodal 3D Object Detection with Attributes

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OVODA enables open-vocabulary 3D object and attribute detection without knowing novel class anchor sizes, outperforming prior baselines on nuScenes and Argoverse 2.

desk verdict The OVAD dataset and the joint object-plus-attribute task are the real contributions; the 'outperforms SOTA on Argoverse 2' claim is not backed by any external baseline in the paper. read the letter →

arxiv 2508.16812 v1 pith:DOXAEYRL submitted 2025-08-22 cs.CV

Xinhao Xiang , Kuan-Chuan Peng , Suhas Lohit , Michael J. Jones , Jiawei Zhang This is my paper
classification cs.CV
keywords detectionobjectattributesnovelovodaattributeopen-vocabularyanchor
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a 3D object detector can recognize object classes and attributes it has never seen during training, without being told the physical size (anchor box) of those unseen classes. The authors argue this is possible by training a class-agnostic object proposer and then using a frozen multimodal foundation model to align image, point-cloud, and text features for every proposal and for pairs of nearby proposals. If correct, autonomous systems could detect arbitrary road users and their spatial or motion attributes directly from language descriptions, rather than from a fixed list of classes and box templates. The paper also introduces a new dataset, OVAD, with 84,384 attribute annotations on nuScenes, including spatial relations absent from existing benchmarks.

What carries the argument

The approach hinges on four components: (1) a class-agnostic 3D object proposer trained only on objectness and box regression, avoiding class-specific anchors; (2) OneLLM, a frozen foundation model providing text, image, and point-cloud encoders, used to compute softmax alignment scores between proposal features and text embeddings; (3) a Complex Event Generation (CEG) module that creates temporal proposals for motion attributes and pairwise proposals within a 15-meter threshold for spatial attributes, with perspective-specified prompts of the form 'From the perspective of C_j, C_i is in front of C_j'; and (4) four losses — two class-agnostic L1 alignment losses and two cross-entropy losses

What would settle it

Evaluate OVODA on novel classes deliberately absent from OneLLM's pretraining (e.g., 'tuk-tuk' or 'wheelchair') and on attribute phrases that are paraphrases of the templates (e.g., replacing 'behind' with 'to the rear of'). If novel-object APN drops to near zero or attribute success rate is no better than a text-only CLIP baseline, the semantic-alignment claim is falsified. Alternatively, remove the point-cloud modality (zero points) and check whether novel attribute detection degrades catastrophically, which would indicate no genuine cross-modal understanding.

Watch

Extended reading notes

Core claim

The central claim is that open-vocabulary 3D object detection in outdoor driving scenes does not require per-class anchor sizes for novel classes, provided the detector is trained class-agnostically and semantic classification is performed by aligning proposal features with text embeddings from a frozen foundation model. The paper further claims that the same alignment mechanism, applied to temporally stacked single-object proposals and to pairs of nearby object proposals, enables open-vocabulary attribute detection, including spatial relationships and motion states. This is demonstrated by OVODA, which builds on 3DETR for proposal generation, uses OneLLM as the frozen multimodal encoder, an

Load-bearing premise

OneLLM's frozen embeddings provide a reliable semantic space for aligning image, point-cloud, and text features of object proposals and pairs of proposals, including classes and attributes never seen in driving data; if this alignment is weak for rare outdoor classes or for two-object clips, the novel-object and novel-attribute detection claims collapse.

Editorial extensions

If this is right

  • If the claim holds, the requirement to know anchor sizes for novel classes is lifted, making open-vocabulary 3D detection practical for arbitrary new road-user categories without retraining the box head.
  • The same text-alignment pipeline extends to attribute detection, so a detector can report zero-shot events such as 'moving person' or 'car behind cyclist' as long as the foundation model's text encoder can embed the phrase.
  • The OVAD dataset provides the first outdoor benchmark with spatial-relation, motion-state, and interaction annotations, enabling future work on open-vocabulary attribute detection.
  • The CEG pairing and perspective-specified prompts offer a general recipe for injecting spatial and temporal context into text-visual alignment, possibly transferable to other 3D perception and scene-understanding tasks.
  • Real-time operation suggests the framework could be embedded in autonomous-driving stacks without a separate attribute-detection stage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves unexamined the possibility that novel classes or attributes falling outside OneLLM's learned semantic space (e.g., rare local vehicle types or fine-grained material attributes) would be missed; a reader might test this by probing OVODA with truly out-of-distribution class names.
  • The spatial attributes are defined by relative coordinates and only for pairs within 15 meters; extending to longer-range or occluded relations would likely require a different proposal mechanism, such as learned pair sampling or attention over all detections.
  • The 'no anchor size' advantage may be partially attributable to the class-agnostic proposer; an interesting extension would be to combine OVODA's alignment head with a modern sparse anchor-free 3D detector to see if proposal quality further improves.
  • A strong test of semantic understanding rather than template matching would be to paraphrase the attribute phrases (e.g., 'to the left of' vs 'left of') and check whether attribute detection success rate remains stable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes OVODA, an open-vocabulary multimodal 3D object and attribute detection framework that aims to detect novel object classes and attributes without requiring anchor sizes for novel classes. The method builds on a 3DETR-style class-agnostic proposer, augments its features with frozen OneLLM embeddings, adds prompt tuning and perspective-specified text prompts, and introduces a complex-event generation module that pairs nearby object proposals to recognize spatial and motion attributes. The authors also introduce OVAD, a nuScenes-derived dataset with 84,384 attribute annotations across 11 attribute classes. Evaluations are reported on nuScenes (Tables 4, 6–8) and Argoverse 2 (Table 5). The central claim is that OVODA outperforms state-of-the-art open-vocabulary 3D object detection under the no-anchor-size condition on both datasets while also detecting attributes.

Significance. If the claims are substantiated, the paper would make a useful contribution to an underexplored problem: joint open-vocabulary 3D object and attribute detection. The released OVAD dataset is a concrete asset for the community, and the design choice of using a frozen foundation model (OneLLM) avoids a fitted-parameter loop. However, the current evidence is incomplete. The Argoverse 2 comparison is only against an ablated self-variant, some loss definitions appear internally inconsistent, and the reported attribute success rates are low and not contextualized. These issues need to be resolved before the headline claims are supportable.

major comments (4)
  1. [§5.1, Table 5; Abstract and Conclusion] The abstract and conclusion claim that OVODA outperforms state-of-the-art open-vocabulary 3D object detection on both nuScenes and Argoverse 2. For Argoverse 2, Table 5 compares OVODA only with its own ablated variant (with and without CFM+prompt tuning); no external method is run on that dataset. The statement that the degraded OVODA already outperforms CoDA v2 on nuScenes is a cross-dataset inference and does not constitute a comparison on Argoverse 2. Either add external baselines on Argoverse 2 (e.g., an adapted CoDA v2 and, if possible, another anchor-free method) or revise the two-dataset claim to reflect the evidence. Additionally, no error bars or multiple-seed results are reported; given the small margins in the Nb0n10 row (Table 4, mAP 4.70 vs 4.32), run-to-run variation cannot be excluded.
  2. [§3.4, Eq. (1) and Eq. (5)] The positive-set definitions for the two classification losses appear vacuous as written. Eq. (1) defines O_disc as proposals with IoU3D < θ_b with every base proposal and with predicted class c*_j ∉ C_b. The loss L_oc is then summed over proposals B_disc_j from O_disc and weighted by f(B_disc_j, B_b), which is described as checking whether B_disc_j is within B_b. Under the stated definitions, f is always 0 for all elements of O_disc, so L_oc never contributes. The same contradiction occurs for A_disc and L_ac with g(A_disc_j, B_ba) in Eq. (5). This is load-bearing because the claimed benefits of the object and attribute classification losses rest on these terms being active. Please define the intended positive/negative proposal sets and show that the losses are non-empty during training, or correct the equations if this is a notational error.
  3. [§5.1, Table 6] The attribute-detection claim is supported by a 25.90% AD-only success rate and a 6.77% full-pipeline success rate. These numbers are presented without comparison to a baseline (e.g., random chance, a closed-set attribute classifier, or an existing attribute-detection method) and without per-class or base/novel breakdowns. As reported, they do not establish that attributes are 'successfully recognized' in a meaningful sense. The paper should add contextualization, per-attribute results, and ideally confidence intervals or significance tests, especially because several reported gains are small (e.g., SR 5.56→6.77).
  4. [Table 4] The comparison with Find n’ Propagate is not made under the same condition as the paper’s headline claim: that method requires predefined anchor sizes for novel classes, which the paper explicitly argues is an unfair advantage. As a result, the only anchor-free external baseline on nuScenes is the adapted CoDA v2. The paper should state clearly that 'state-of-the-art' in the abstract refers to anchor-free methods; otherwise the comparison with Find n’ Propagate is misleading.
minor comments (6)
  1. [§5.1, Nb0n10 discussion] The text claims 'an 11.4% performance improvement' in the Nb0n10 setting. This number is not directly reproducible from Table 4; specify the metric and the baseline used for the percentage.
  2. [Supplementary Table 10] The attribute vocabulary lists 'moving' twice. Also, the text colors referred to in the table are not usable in a monochrome print version; use explicit labels or symbols.
  3. [§3.4 and §5.1] The prompt-tuning mechanism is described only as 'learnable visual prompts at the FM’s input layer'; the prompt length, initialization, and insertion point are not specified, which makes the method difficult to reproduce.
  4. [§5.1, Table 8] The ablation for PSP and HFA is performed only with CLIP; the OneLLM row uses both components together, so the individual contribution of PSP and HFA in the final system is not isolated.
  5. [§4] The OVAD spatial attributes are generated from ground-truth boxes using the same geometric definitions that are later used to build the text prompts. This is not circular, but it means the spatial-attribute benchmark measures geometric relational classification rather than open-vocabulary semantics; the paper should state this limitation explicitly.
  6. [§5.1, Table 7] The foundation-model comparison uses CLIP and CogVLM, which provide no point-cloud encoder, while OneLLM does. The reported gains may therefore reflect the availability of a point-cloud modality as much as the choice of foundation model; this confound should be acknowledged.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the Argoverse 2 baseline gap is an evidence/support issue, not a derivation loop.

full rationale

The derivation chain in OVODA is not circular by the definitions used here. Novel object and attribute scores are computed as dot products between visual features and text embeddings produced by a frozen foundation model (OneLLM), with P^O_j = Softmax(V^O_j · F^{T_O}) and P^A_j = Softmax(V^A_j · F^{T_A}). No novel-class or novel-attribute parameter is fitted to the novel data; the thresholds θ_b, θ_o, θ_s, θ_a are fixed hyperparameters, and the training losses (L_od, L_oc, L_ad, L_ac) use only base-class supervision. The phrase “following [5]” refers to an external method (CoDA v2), not to the authors' own prior work, and the self-citations in the paper are contextual examples of closed-set or fusion detectors rather than load-bearing support for the open-vocabulary mechanism. The OVAD spatial-attribute labels are generated from ground-truth boxes using geometric definitions, and the method's spatial text templates use the same relational vocabulary; however, at inference the model must infer that geometry from visual inputs, so this is a benchmark-construction concern rather than a reduction of the prediction to its input. The main weakness of the paper is that Table 5 on Argoverse 2 compares OVODA only against its own ablated variant and not against an external state-of-the-art method, so the abstract's two-dataset superiority claim is not fully evidenced. That is a missing-baseline/support issue, not a circularity: no equation, fitted parameter, or self-citation makes the claimed result equivalent to its inputs by construction.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The framework rests on an external foundation model rather than on a self-contained derivation. It also introduces several hand-set thresholds and an unspecified temporal window and loss weights. These are not fitted constants in the scientific sense, but they influence what is counted as a novel detection and thus affect the reported numbers.

free parameters (7)
  • theta_b (IoU threshold) = 0.2
    Hand-set threshold in Eq. 1 and Eq. 5 that determines whether a proposal is considered novel.
  • theta_o (objectness threshold) = 0.8
    Hand-set threshold in Eq. 1 for novel object discovery.
  • theta_s (semantic confidence threshold) = 0.5
    Hand-set threshold in Eq. 1 for novel object semantic score.
  • theta_a (attribute semantic threshold) = 0.5
    Hand-set threshold in Eq. 5 for novel attribute discovery.
  • theta_d (spatial distance threshold) = 15 m
    Hand-set threshold for pairing object proposals in Eq. 2 and for OVAD annotation; affects which spatial relations can be detected.
  • T (temporal window) = not stated
    Number of past timestamps concatenated for non-spatial attribute proposals in Sec. 3.3; value omitted from paper.
  • loss weights w_od, w_oc, w_ad, w_ac = not stated
    Weights balancing the four losses in Sec. 3.4; described only as 'set to ensure comparable ranges'.
assumptions (5)
  • domain assumption OneLLM provides a reliable shared semantic space for images, point clouds, and text in outdoor driving scenes.
    Novel class and attribute prediction is based on Softmax(V dot F_text) with frozen OneLLM (Sec. 3.2, 3.3); if this alignment is unreliable, the open-vocabulary claim fails.
  • domain assumption A class-agnostic 3DETR-style proposer trained on base class boxes transfers to novel classes without anchor size priors.
    OVODA's objectness and box regression are trained only on base boxes (Sec. 3.2); novel object discovery depends on proposer recall for unseen classes.
  • domain assumption Spatial relation labels can be generated automatically from ground-truth boxes within 15 m using mathematical definitions.
    OVAD spatial annotations and the text proposal template share the same coordinate-based rule (Sec. 4, Eq. 3, Eq. 4), making evaluation self-consistent but unverified against human judgments.
  • domain assumption Including novel class names in the training vocabulary is a valid open-vocabulary protocol and does not leak supervision.
    Supplementary Sec. 1 includes novel and extra class names in the text vocabulary during training, following [5,8,44]; the correctness of this protocol is assumed.
  • domain assumption L1 distillation between OneLLM visual features and 3D backbone features transfers semantic knowledge and improves novel class discriminability.
    The L_od and L_ad losses (Sec. 3.4) assume that aligning 3D features to OneLLM features is beneficial; no independent analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Open-Vocabulary Multimodal 3D Object Detection with Attributes." pith.science (2026). https://pith.science/paper/DOXAEYRL

@misc{pith2026250816812,
  author       = {Pith},
  title        = {Pith review of: Towards Open-Vocabulary Multimodal 3D Object Detection with Attributes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DOXAEYRL}},
  note         = {Machine review of arXiv:2508.16812}
}
read the original abstract

3D object detection plays a crucial role in autonomous systems, yet existing methods are limited by closed-set assumptions and struggle to recognize novel objects and their attributes in real-world scenarios. We propose OVODA, a novel framework enabling both open-vocabulary 3D object and attribute detection with no need to know the novel class anchor size. OVODA uses foundation models to bridge the semantic gap between 3D features and texts while jointly detecting attributes, e.g., spatial relationships, motion states, etc. To facilitate such research direction, we propose OVAD, a new dataset that supplements existing 3D object detection benchmarks with comprehensive attribute annotations. OVODA incorporates several key innovations, including foundation model feature concatenation, prompt tuning strategies, and specialized techniques for attribute detection, including perspective-specified prompts and horizontal flip augmentation. Our results on both the nuScenes and Argoverse 2 datasets show that under the condition of no given anchor sizes of novel classes, OVODA outperforms the state-of-the-art methods in open-vocabulary 3D object detection while successfully recognizing object attributes. Our OVAD dataset is released here: https://doi.org/10.5281/zenodo.16904069 .

Figures

Figures reproduced from arXiv: 2508.16812 by the authors.

Figure 1
Figure 1. Relaxing the constraints of (a) single-modal [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Our proposed OVODA framework combines information from multi-view images and point clouds, align object and attribute text with vision features using a common foundation model encoder in order to discover and localize complex open-vocabulary events that include multiple objects and attributes. We show the figure using information from a single time instant for simplicity. In our method, we aggregate information over… view at source ↗
Figure 6
Figure 6. Qualitative comparison of OVODA (middle) versus CoDAv2 [5] (right) with the ground truth (left) [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figures from the paper (3 more)
Figure 7
Figure 7. Figure 7: The qualitative comparison of OVODA (left) versus CoDAv2 [5] (right) for 3D single object detection in nuScenes dataset. The ground truth annotations are rendered in light blue/light red/light green for the class car/pedestrian/others, the predicted bounding boxes are …
Figure 8
Figure 8. Figure 8: More qualitative comparison of OVODA (left) versus CoDAv2 [5] (right) for 3D single object detection in nuScenes dataset. The ground truth annotations are rendered in light blue/light red/light green for the class car/pedestrian/others, the predicted bounding boxes are…
Figure 9
Figure 9. Figure 9: More qualitative results of OVODA for 3D complex event detection in nuScenes dataset. All ground truth annotations of single object are rendered in light green. The ground truth annotations are rendered in light purple/yellow/light blue for the car-car/pedestrain-pedes…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 43 canonical work pages

  1. [1]

    TransFusion: Robust LiDAR-camera fusion for 3D object detection with transformers

    Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan T ai. TransFusion: Robust LiDAR-camera fusion for 3D object detection with transformers. In IEEE/CVF Conference on Computer V ision and P attern Recognition (CVPR), 2022

    work page 2022

  2. [2]

    Is space-time attention all you need for video understanding? InInternational Conference on Machine Learning (ICML), 2021

    Gedas Bertasius, Heng W ang, and Lorenzo T orresani. Is space-time attention all you need for video understanding? InInternational Conference on Machine Learning (ICML), 2021. URL https://arxiv.org/abs/2102.05095

    arXiv 2021

  3. [3]

    Lang, Sourabh V ora, V enice Erin Liong, Qiang Xu, Anush Krishnan, Y u Pan, Giancarlo Baldan, and Oscar Beijbom

    Holger Caesar, V arun Bankiti, Alex H. Lang, Sourabh V ora, V enice Erin Liong, Qiang Xu, Anush Krishnan, Y u Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset for autonomous driving. InIEEE/CVF Conference on Computer V ision and P attern Recognition (CVPR), 2020

    work page 2020

  4. [4]

    CoDA: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3D object detection

    Y ang Cao, Yihan Zeng, Hang Xu, and Dan Xu. CoDA: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3D object detection. InNeural Information Processing Systems (NeurIPS), 2023

    work page 2023

  5. [5]

    Collaborative Novel Object Discovery and Box-Guided Cross-Modal Alignment for Open-Vocabulary 3D Object Detection

    Y ang Cao, Yihan Zeng, Hang Xu, and Dan Xu. Collaborative novel object discovery and box-guided cross-modal alignment for open-vocabulary 3D object detection.arXiv preprint arXiv:2406.00830, 2024. URL https://arxiv.org/abs/2406.00830

    work page Pith review arXiv 2024

  6. [6]

    Quo vadis, action recognition? A new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. InIEEE/CVF Conference on Computer V ision and P attern Recognition (CVPR), 2017. URL https://arxiv.org/abs/1705.07750

    arXiv 2017

  7. [7]

    Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model

    Y u Du, Fangyun W ei, Zihe Zhang, Miaojing Shi, Y ue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. InIEEE/CVF XIANG ET AL.: OPEN-VOCABULARY OBJECT DETECTION WITH A TTRIBUTES 11 Conference on Computer V ision and P attern Recognition (CVPR), 2022. URL https: //arxiv.org/abs/2203.14940

    work page Pith review arXiv 2022

  8. [8]

    Find n' Propagate: Open-Vocabulary 3D Object Detection in Urban Environments

    Djamahl Etchegaray, Zi Huang, T atsuya Harada, and Y adan Luo. Find n’ propagate: Open- vocabulary 3D object detection in urban environments. InEuropean Conference on Computer V ision (ECCV), 2024. URL https://arxiv.org/abs/2403.13556

    work page Pith review arXiv 2024

Show all 56 references
  1. [9]

    Fully sparse 3D object detection

    Lue Fan, Feng W ang, Naiyan W ang, and Zhaoxiang Zhang. Fully sparse 3D object detection. In Neural Information Processing Systems (NeurIPS), 2022

    2022

  2. [10]

    Multi-modal transformer for video retrieval

    V alentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. InEuropean Conference on Computer V ision (ECCV), 2020

    2020

  3. [11]

    Are we ready for autonomous driving? the KITTI vision benchmark suite

    Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the KITTI vision benchmark suite. In2012 IEEE Conference on Computer V ision and P attern Recognition, pages 3354–3361, 2012. doi: 10.1109/CVPR.2012.6248074

    2012

  4. [12]

    Tenenbaum, Antonio T orralba, Florian Shkurti, and Liam Paull

    Qiao Gu, Alihusein Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, Chuang Gan, Celso Miguel de Melo, Joshua B. Tenenbaum, Antonio T orralba, Florian Shkurti, and Liam Paull. Concep...

    2024 arXiv

  5. [13]

    Open-vocabulary object detection via vision and language knowledge distillation

    Xiuye Gu, Tsung-Yi Lin, W eicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. InInternational Conference on Learning Representations (ICLR), 2022

    2022

  6. [14]

    OneLLM: One framework to align all modalities with language

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi W ang, Kaipeng Zhang, Dahua Lin, Y u Qiao, Peng Gao, and Xiangyu Y ue. OneLLM: One framework to align all modalities with language. In IEEE/CVF Conference on Computer V ision and P attern Recognition (CVPR),

  7. [15]

    Jones, and Vishal M

    Deepti Hegde, Suhas Lohit, Kuan-Chuan Peng, Michael J. Jones, and Vishal M. Patel. Equivariant spatio-temporal self-supervision for LiDAR object detection. In European Conference on Computer V ision (ECCV), 2025

    2025

  8. [16]

    Jones, and Vishal M

    Deepti Hegde, Suhas Lohit, Kuan-Chuan Peng, Michael J. Jones, and Vishal M. Patel. Multimodal 3D object detection on unseen domains. InIEEE/CVF Conference on Computer V ision and P attern Recognition (CVPR) W orkshop, 2025

    2025

  9. [17]

    Long short-term memory.Neural Comput., 9 (8):1735–1780, November 1997

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory.Neural Comput., 9 (8):1735–1780, November 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735. URL https://doi.org/10.1162/neco.1997.9.8.1735

    1997 doi

  10. [18]

    T enenbaum, Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, and Antonio T orralba

    Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, T ao Chen, Alaa Maalouf, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, A yush Tewari, Joshua B. T enenbaum, Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, and Antonio T o...

    2023 arXiv

  11. [19]

    Action genome: Actions as composition of spatio-temporal scene graphs

    Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions as composition of spatio-temporal scene graphs. InIEEE/CVF Conference on Computer V ision and P attern Recognition (CVPR), 2020. URL https://arxiv.org/abs/1912. 06992

    2020

  12. [20]

    PF3Det: A prompted foundation feature assisted visual LiDAR 3D detector

    Kaidong Li, Tianxiao Zhang, Kuan-Chuan Peng, and Guanghui W ang. PF3Det: A prompted foundation feature assisted visual LiDAR 3D detector. InIEEE/CVF Conference on Computer V ision and P attern Recognition (CVPR) W orkshop, 2025

    2025

  13. [21]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Y ang, Chunyuan Li, Yiwu Zhong, Lijuan W ang, Lu Y uan, Lei Zhang, Jenq-Neng Hwang, Kai-W ei Chang, and Jianfeng Gao. Grounded language-image pre-training. In IEEE/CVF Conference on Computer V ision and P attern Recogn...

    2022

  14. [22]

    OpenShape: Scaling up 3D shape representation towards open-world understanding

    Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. OpenShape: Scaling up 3D shape representation towards open-world understanding. InNeural Information Processing Systems (NeurIPS), 2023

    2023

  15. [23]

    Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Y ang, Qing Jiang, Chunyuan Li, Jianwei Y ang, Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. InEuropean Conference on Computer V ision (EC...

    2024

  16. [25]

    Open-vocabulary point-cloud object detection without 3D annotation

    Y uheng Lu, Chenfeng Xu, Xiaobao W ei, Xiaodong Xie, Masayoshi T omizuka, Kurt Keutzer, and Shanghang Zhang. Open-vocabulary point-cloud object detection without 3D annotation. In IEEE/CVF Conference on Computer V ision and P attern Recognition (CVPR), 2023

    2023

  17. [26]

    An end-to-end transformer model for 3D object detection

    Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3D object detection. InIEEE/CVF International Conference on Computer V ision (ICCV), 2021. URL https://arxiv.org/abs/2109.08141

    2021 arXiv

  18. [27]

    Modeling temporal structure of decomposable motion segments for activity classification

    Juan Carlos Niebles, Chih-W ei Chen, and Li Fei-Fei. Modeling temporal structure of decomposable motion segments for activity classification. In European Conference on Computer V ision (ECCV), ECCV’10, page 392–405, 2010. ISBN 3642155510

    2010

  19. [28]

    PyT orch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Y ang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fa...

    2019

  20. [29]

    OpenScene: 3D scene understanding with open vocabularies

    Songyou Peng, Kyle Genova, Chiyu "Max" Jiang, Andrea T agliasacchi, Marc Pollefeys, and Thomas Funkhouser. OpenScene: 3D scene understanding with open vocabularies. In IEEE/CVF Conference on Computer V ision and P attern Recognition (CVPR), 2023. XIANG ET AL.: OPEN-VOCABULARY ...

    2023

  21. [30]

    Qi, Hao Su, Kaichun Mo, and Leonidas J

    Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. InIEEE/CVF Conference on Computer V ision and P attern Recognition (CVPR), 2017. URL https://arxiv.org/abs/1612. 00593

    2017

  22. [31]

    Qi, W ei Liu, Chenxia Wu, Hao Su, and Leonidas J

    Charles R. Qi, W ei Liu, Chenxia Wu, Hao Su, and Leonidas J. Guibas. Frustum PointNets for 3D object detection from RGB-D data. InIEEE/CVF Conference on Computer V ision and P attern Recognition (CVPR), 2018. URL https://arxiv.org/abs/1711.08488

    2018 arXiv

  23. [32]

    Qi, Or Litany, Kaiming He, and Leonidas J

    Charles R. Qi, Or Litany, Kaiming He, and Leonidas J. Guibas. Deep hough voting for 3D object detection in point clouds. InIEEE/CVF International Conference on Computer V ision (ICCV), 2019. URL https://arxiv.org/abs/1904.09664

    2019 arXiv

  24. [33]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong W ook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of M...

    2021 arXiv

  25. [34]

    PointRCNN: 3D object proposal generation and detection from point cloud

    Shaoshuai Shi, Xiaogang W ang, and Hongsheng Li. PointRCNN: 3D object proposal generation and detection from point cloud. InIEEE/CVF Conference on Computer V ision and P attern Recognition (CVPR), 2019. URL https://arxiv.org/abs/1812.04244

    2019 arXiv

  26. [35]

    PV -RCNN: Point-voxel feature set abstraction for 3D object detection

    Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe W ang, Jianping Shi, Xiaogang W ang, and Hongsheng Li. PV -RCNN: Point-voxel feature set abstraction for 3D object detection. In IEEE/CVF Conference on Computer V ision and P attern Recognition (CVPR), 2020

    2020

  27. [36]

    VideoBERT: A joint model for video and language representation learning

    Chen Sun, Austin Myers, Carl V ondrick, Kevin Murphy, and Cordelia Schmid. VideoBERT: A joint model for video and language representation learning. InIEEE/CVF International Conference on Computer V ision (ICCV), 2019. URLhttps://arxiv.org/abs/1904. 01766

    2019

  28. [37]

    Scalability in perception for autonomous driving: W aymo open dataset, 2020

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Y uning Chai, Benjamin Caine, Vijay V asudevan, W ei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Sheng ...

    2020

  29. [38]

    Learning spatiotemporal features with 3D convolutional networks

    Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo T orresani, and Manohar Paluri. Learning spatiotemporal features with 3D convolutional networks. InIEEE/CVF International Con- ference on Computer V ision (ICCV), 2015. URL https://arxiv.org/abs/1412. 0767

    2015

  30. [39]

    DSVT: Dynamic sparse voxel transformer with rotated sets

    Haiyang W ang, Chen Shi, Shaoshuai Shi, Meng Lei, Sen W ang, Di He, Bernt Schiele, and Liwei W ang. DSVT: Dynamic sparse voxel transformer with rotated sets. InIEEE/CVF Conference on Computer V ision and P attern Recognition (CVPR), 2023

    2023

  31. [41]

    Hierarchical open-vocabulary 3D scene graphs for language-grounded robot navigation

    Abdelrhman W erby, Chenguang Huang, Martin Büchner, Abhinav V alada, and W olfram Bur- gard. Hierarchical open-vocabulary 3D scene graphs for language-grounded robot navigation. In Robotics: Science and Systems (RSS), RSS2024, July 2024. doi: 10.15607/rss.2024.xx.077. URL http...

    2024 doi

  32. [42]

    Argoverse 2: Next generation datasets for self-driving perception and forecasting

    Benjamin Wilson, William Qi, T anmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next generation datasets for self-driving perception and f...

    2024

  33. [43]

    Transformation- equivariant 3D object detection for autonomous driving

    Hai Wu, Chenglu W en, W ei Li, Xin Li, Ruigang Y ang, and Cheng W ang. Transformation- equivariant 3D object detection for autonomous driving. InThe Thirty-Seven AAAI Conference on Artificial Intelligence (AAAI), 2023

    2023

  34. [44]

    T owards open vocabulary learning: A survey.IEEE Transactions on P attern Analysis and Machine Intelligence, 2024

    Jianzong Wu, Xiangtai Li, Shilin Xu, Haobo Y uan, Henghui Ding, Yibo Y ang, Xia Li, Jiangning Zhang, Y unhai T ong, Xudong Jiang, Bernard Ghanem, and Dacheng T ao. T owards open vocabulary learning: A survey.IEEE Transactions on P attern Analysis and Machine Intelligence, 2024...

    2024 arXiv

  35. [45]

    FusionViT: Hierarchical 3D object detection via LiDAR- camera vision transformer fusion.arXiv preprint arXiv:2311.03620, 2023

    Xinhao Xiang and Jiawei Zhang. FusionViT: Hierarchical 3D object detection via LiDAR- camera vision transformer fusion.arXiv preprint arXiv:2311.03620, 2023. URL https: //arxiv.org/abs/2311.03620

    2023 arXiv

  36. [46]

    3DifFusionDet: Diffusion model for 3D object detection with robust LiDAR-camera fusion.arXiv preprint arXiv:2311.0374, 2023

    Xinhao Xiang, Simon Dräger, and Jiawei Zhang. 3DifFusionDet: Diffusion model for 3D object detection with robust LiDAR-camera fusion.arXiv preprint arXiv:2311.0374, 2023. URL https://arxiv.org/abs/2311.03742

    2023

  37. [47]

    EffiPerception: An efficient framework for various perception tasks.arXiv preprint arXiv:2403.12317, 2024

    Xinhao Xiang, Simon Dräger, and Jiawei Zhang. EffiPerception: An efficient framework for various perception tasks.arXiv preprint arXiv:2403.12317, 2024. URL https://arxiv. org/abs/2403.12317

    2024 arXiv

  38. [48]

    Graph R-CNN for scene graph generation

    Jianwei Y ang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph R-CNN for scene graph generation. InEuropean Conference on Computer V ision (ECCV), 2018. URL https://arxiv.org/abs/1808.00191

    2018 arXiv

  39. [49]

    Open-vocabulary DETR with conditional matching

    Y uhang Zang, W ei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary DETR with conditional matching. InEuropean Conference on Computer V ision (ECCV), page 106–122, 2022. ISBN 9783031200779. doi: 10.1007/978-3-031-20077-9_7. URL http://dx.doi.org/10.1007/978-3...

    2022 doi

  40. [50]

    Open-vocabulary object detection using captions

    Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. InIEEE/CVF Conference on Computer V ision and P attern Recognition (CVPR), 2021. URL https://arxiv.org/abs/2011.10678

    2021 arXiv

  41. [51]

    FM-OV3D: Foundation model-based cross-modal knowledge blending for open-vocabulary 3D detection

    Dongmei Zhang, Chang Li, Ray Zhang, Shenghao Xie, W ei Xue, Xiaodong Xie, and Shanghang Zhang. FM-OV3D: Foundation model-based cross-modal knowledge blending for open-vocabulary 3D detection. In The Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI), 2024

    2024

  42. [52]

    OpenSight: A simple open-vocabulary framework for LiDAR-based object detection

    Hu Zhang, Jianhua Xu, T ao T ang, Haiyang Sun, Xin Y u, Zi Huang, and Kaicheng Y u. OpenSight: A simple open-vocabulary framework for LiDAR-based object detection. In European Conference on Computer V ision (ECCV), 2024. XIANG ET AL.: OPEN-VOCABULARY OBJECT DETECTION WITH A TT...

    2024

  43. [53]

    G, Anastasis Stathopoulos, Manmohan Chandraker, and Dimitris Metaxas

    Shiyu Zhao, Zhixing Zhang, Samuel Schulter, Long Zhao, Vijay Kumar B. G, Anastasis Stathopoulos, Manmohan Chandraker, and Dimitris Metaxas. Exploiting unlabeled data with vision and language models for object detection. In European Conference on Computer V ision (ECCV), 2022

    2022

  44. [54]

    OcTr: Octree-based transformer for 3D object detection

    Chao Zhou, Y anan Zhang, Jiaxin Chen, and Di Huang. OcTr: Octree-based transformer for 3D object detection. InIEEE/CVF Conference on Computer V ision and P attern Recognition (CVPR), 2023

    2023

  45. [55]

    Object2Scene: Putting objects in context for open-vocabulary 3D detection.arXiv preprint arXiv:2311.03079, 2023

    Chenming Zhu, W enwei Zhang, T ai W ang, Xihui Liu, and Kai Chen. Object2Scene: Putting objects in context for open-vocabulary 3D detection.arXiv preprint arXiv:2311.03079, 2023. URL https://arxiv.org/abs/2309.09456

    2023 arXiv

  46. [56]

    PointCLIP V2: Prompting CLIP and GPT for powerful 3D open-world learning

    Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng, Zipeng Qin, Shanghang Zhang, and Peng Gao. PointCLIP V2: Prompting CLIP and GPT for powerful 3D open-world learning. In IEEE/CVF International Conference on Computer V ision (ICCV), 2023. XIANG ET AL.: OPEN-VOCABULAR...

    2023

  47. [58]

    Then, the model continues to be trained for 20 epochs

    Initially, we train a base 3DETR model for 20 epochs using only class-agnostic distillation. Then, the model continues to be trained for 20 epochs. The hyper-parameters used during training follow the default 3DETR configuration specified in [5, 26]. XIANG ET AL.: OPEN-VOCABUL...

  48. [2024]

    URL https://arxiv.org/abs/2312.03700

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.