Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One point-cloud module unifies 3D dialogue, prompting, and mask output.

desk verdict A clean point-cloud-only 3D LMM with a genuinely unified OST connector; strong results but overclaimed SOTA and two missing analyses (distillation ablation, token-selection recall on referring tasks). read the letter →

arxiv 2501.01163 v2 pith:5H7ZOZK6 submitted 2025-01-02 cs.CV

Jiajun Deng , Tianyu He , Li Jiang , Tianyu Wang , Feras Dayoub , Ian Reid This is my paper · ORCID
classification cs.CV
keywords 3Dlargemultimodalmodelspointcloudunderstandingreferringsegmentationvisualpromptencodingsuperpointtransformerinstructiontuningquestionansweringdensecaptioning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

3D-LLaVA claims that a single module, the Omni Superpoint Transformer (OST), can act as visual feature selector, visual prompt encoder, and referring mask decoder for a 3D large multimodal model, removing the need for offline multi-view feature extraction and task-specific segmentation heads. The paper argues that a point-cloud-only pipeline built around this connector can match or beat substantially more complex systems across question answering, dense captioning, and referring segmentation. If the claim holds, an integrated connector is enough to give a 3D assistant both free-form dialogue and precise spatial grounding, which would make deployment simpler and more accessible.

What carries the argument

The Omni Superpoint Transformer (OST) is a superpoint-based transformer that deliberately omits cross-attention layers, using superpoint features as both queries and source features with distance-adaptive self-attention, and carries a mask head, classification head, and alignment head. It is pre-trained with a hybrid loss combining ScanNet200 instance segmentation and 2D-to-3D knowledge distillation from a 2D LMM's visual encoder, then frozen and reused at inference as the feature selector, visual prompt encoder, and mask decoder. A parameter-free visual sampler turns clicks, boxes, and masks into prompt features that OST projects into the same embedding space as visual tokens.

What would settle it

Compute, on ScanRefer or Multi3DRefer validation, the recall of the ground-truth target instance's superpoints among the top-100 objectness-ranked superpoints; if a large share of failed predictions correspond to targets that fall outside the top-100, the objectness-based selection assumption is the reason for the failure. A direct probe would feed the same scene with and without the target's superpoints forced into the top-100 and measure whether the model then answers or segments correctly.

Watch

Extended reading notes

Core claim

The discovery is that a transformer operating on superpoints, pre-trained with instance segmentation and 2D-to-3D feature distillation, can serve every visual role a 3D LMM needs: it selects which scene regions become LLM tokens, encodes any user-provided click, box, or mask prompt into the same token space, and, when the LLM emits a [SEG] token, decodes the hidden state into a 3D mask. With this one shared connector, 3D-LLaVA reports best results on the evaluated benchmarks among 3D LMMs trained jointly on multiple tasks: 43.3% mIoU on ScanRefer, 42.7% mIoU on Multi3DRefer, 92.6% CiDEr on ScanQA, and best-or-comparable scores on SQA3D and Scan2Cap, all with point clouds as the only input.

Load-bearing premise

The pipeline keeps only the 100 superpoints with the highest objectness score, where objectness is the max foreground-class logit from a classifier trained on ScanNet200 categories; everything rests on those 100 tokens containing the regions needed to answer the user's question or match the referring expression.

Editorial extensions

If this is right

  • Offline multi-view feature extraction and extra task-specific heads can be removed from 3D LMM pipelines without losing performance.
  • The same frozen OST can ground language to 3D masks and answer or describe scenes, so referring segmentation no longer requires a separate aligned segmentation network.
  • On the paper's benchmarks, the model improves referring segmentation over the previous best (1.6 mIoU on ScanRefer, 6.6 on Multi3DRefer) and raises ScanQA CiDEr by 4.9 absolute over the prior best.
  • Visual prompts of different forms (click, box, mask) are handled by one mechanism, including masks from any external proposer such as Mask3D, without retraining.
  • When a referred object is absent, the LLM can answer that it cannot find the object and skip mask decoding, avoiding false-positive masks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A likely pressure point the paper leaves implicit is the objectness-based top-K selection: since only the 100 highest-scoring superpoints reach the LLM, making selection query-aware (for example, conditioned on the text or prompt) could improve recall of low-saliency but referenced objects.
  • The same architecture could be tested on open-vocabulary or zero-shot settings by replacing the fixed category classification head with a text-aligned feature space, which may let the connector ground expressions beyond ScanNet200 categories.
  • Because OST is frozen during instruction tuning, it may be feasible to swap the LLM for a stronger or lighter one later without retraining the visual connector, turning the 3D encoder into a reusable front end.
  • The unified prompting mechanism suggests an interactive point, box, or text control scheme for 3D assistants that could transfer to embodied tasks like robot manipulation, but the paper only benchmarks static ScanNet scenes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces 3D-LLaVA, a 3D large multimodal model built around a single Omni Superpoint Transformer (OST) that serves as a visual feature selector, a visual prompt encoder, and a referring mask decoder. The model takes only point clouds as input, uses superpoint pooling and a top-K objectness-based token selection to obtain visual tokens for the LLM, and decodes a [SEG] token into a 3D mask through the frozen OST. The OST and the Sparse 3D U-Net are pretrained with instance segmentation plus 2D-to-3D feature distillation, and the whole system is instruction-tuned on ScanRefer, Nr3D, Multi3DRefer, ScanQA, SQA3D, and Scan2Cap. Experiments report results on ScanRefer, Multi3DRefer, ScanQA, SQA3D, and Scan2Cap, with the strongest gains on referring segmentation and ScanQA CiDEr.

Significance. If the central claim holds, the paper is a useful step toward unified 3D LMMs: a single point-cloud-based connector can replace offline multi-view feature extraction and task-specific heads for referring segmentation, visual prompting, and mask decoding. The architecture is clean, the code is released, and the evaluation protocol uses standard train/validation splits with held-out test sets, so I found no circularity in the evaluation. The strongest evidence is the referring-segmentation performance (43.3% mIoU on ScanRefer and 42.7% on Multi3DRefer) and the 92.6% CiDEr on ScanQA, obtained without the 2D instance features used by Chat-Scene. However, the query-blind top-100 objectness selection and the use of Mask3D proposals for dense captioning are not adequately validated, and the blanket SOTA claim is not supported by the paper's own table. These issues are fixable but need to be addressed before the central claims can be accepted.

major comments (4)
  1. [Section 3.3 (Visual Feature Selection) and Table 5] The top-K=100 superpoint selection is query-blind: the objectness score is the maximum foreground-class logit from a ScanNet200 classifier computed before the language instruction is seen, and any superpoint outside the top 100 is irrecoverable. This is a load-bearing bottleneck for referring segmentation because the LLM can never ground a [SEG] token to a discarded target, and Section 3.4 states that the Sparse 3D U-Net and OST are kept frozen during instruction tuning, so the selection cannot be adapted by the downstream data. Table 5 sweeps the token count only on ScanQA and Scan2Cap, where global scene statistics and large objects dominate; it does not measure target recall on ScanRefer or Multi3DRefer. I request a recall analysis (fraction of referred objects whose superpoints are among the top K) and a K-ablation on the referring-segmentation mIoU. Without this, the reported 43.3% and 42.7% mIoU are unverified upper bounds set by the selection step.
  2. [Section 4.3 (3D Dense Captioning)] The dense captioning experiment uses Mask3D-predicted mask proposals as visual prompts. This is an external task-specific proposal generator applied at inference, which is in tension with the paper's stated contribution of removing offline preprocessing and task-specific heads. The claim that the pipeline 'only takes point clouds as input' is technically true at the modality level, but it does not address the extra module. Please clarify whether Mask3D is part of the deployed system, report results without Mask3D (e.g., using ground-truth boxes or a simple unsupervised grouping), and discuss how the Scan2Cap comparison is affected for methods that do not use an external proposal generator.
  3. [Abstract, Introduction, and Table 2] The claim of state-of-the-art performance on all evaluated datasets is contradicted by Table 2. On SQA3D, Chat-Scene achieves EM 54.6 versus 54.5 for 3D-LLaVA; on Scan2Cap, Chat-Scene achieves METEOR 28.0 versus 27.1 and Rouge-L 58.1 versus 57.7. The text should report per-metric wins and losses rather than stating blanket SOTA, and the Figure 1 caption ('most of the benchmarks') should be aligned with the main text.
  4. [Section 3.4, Eq. (2)] The hybrid pretraining objective combines classification, mask, and 2D-to-3D distillation losses, and the distillation is motivated as a bridge for language alignment. Yet no experiment isolates L_KD: there is no ablation trained without distillation or without the instance-segmentation supervision. Given that the 2D teacher (CLIP-ViT-L via LLaVA-1.5) is an external component and a stated source of alignment, its contribution should be quantified. I consider this a missing experiment rather than a flaw in the derivation, but it is needed to support the design choice.
minor comments (6)
  1. [Section 3.3] The text says 'the objections score of each superpoint query'; this should be 'objectness score'.
  2. [Section 3.4] The subsection heading uses 'pre-training' but the text contains 'pretaining' (e.g., 'at the pertaining stage'); please correct the typo.
  3. [Table 4] The metric name 'Rough-L' should be 'Rouge-L' for consistency with Table 2 and the main text.
  4. [Section 4.3] The benchmark name 'Scanerfer' in the sentence 'the single-target setting (Scanerfer [7])' should be 'ScanRefer'.
  5. [Section 3.2, Eq. (1)] Please clarify the definition of D in Eq. (1): is it the Euclidean distance between superpoint centroids, and is the softmax computed over all superpoint queries? Also state how the learnable scale sigma is initialized.
  6. [Section 4.2] The implementation details do not report the number of training seeds or whether results are from a single run; adding error bars or at least an explicit statement about single-run results would help assess stability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are supported by held-out benchmark evaluations, not by definitions or fitted predictions.

full rationale

The paper is an empirical systems paper: the claimed contribution is that a single Omni Superpoint Transformer (OST) can serve as visual feature selector, visual prompt encoder, and referring mask decoder, and this is validated on standard external benchmarks (ScanRefer, Multi3DRefer, ScanQA, SQA3D, Scan2Cap) using held-out validation/test splits and standard metrics. No derived quantity is defined in terms of the quantity it is said to predict: the top-K objectness-based token selection is a fixed architectural design choice, and the only related study (Table 5) is an ablation of token count, not a fit of test metrics. The mask decoding path uses the frozen OST with a [SEG] query, and its outputs are scored against external ground-truth masks. There is no fitted parameter that is later renamed as a prediction, no uniqueness theorem imported from the authors' own prior work, and no load-bearing self-citation: the cited prior work overlapping with the authors (e.g., Agent3D-Zero) appears only in related-work context and does not justify the paper's core claims. The known weakness of query-blind top-100 pruning is a potential correctness or generalization risk, not a circularity, because the benchmark numbers would still be independent evidence even if the selection rule is suboptimal. Accordingly, no circular step can be exhibited and the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central results depend on a few hand-chosen hyperparameters and several domain assumptions about the quality of superpoint clustering, 2D-to-3D distillation, and objectness ranking. None of these are validated with ablations beyond the token-number study in Table 5.

free parameters (3)
  • Top-K visual tokens (K) = 100
    Number of superpoint features retained by objectness score, chosen by hand after ablating 50/100/200/400 tokens (Table 5). This directly controls what the LLM can see.
  • Mask loss coefficient = 0.1
    Weight of L_mask in the instruction tuning objective (Eq. 3), chosen by hand for loss balance and not ablated.
  • Pretraining epochs = 512
    Number of epochs for stage-1 pretraining of the 3D encoder and OST; hand-set without reported ablation or sensitivity analysis.
assumptions (3)
  • domain assumption CLIP-ViT-L features lifted from 2D to 3D via geometric correspondence provide reliable dense semantic supervision for superpoint pretraining.
    Section 3.4 Stage 1 uses L_KD with teacher features from LLaVA-1.5/CLIP; no ablation isolates this component, and CLIP patch features are not dense per-pixel semantic labels.
  • domain assumption Superpoint clustering (bottom-up algorithm from [38]) yields segments that align with object instances sufficiently for mask decoding.
    The whole pipeline operates on superpoints; if clustering straddles object boundaries, mask quality degrades. The paper relies on the prior method without validating on their data.
  • domain assumption Objectness scores from a classifier trained on ScanNet200 categories generalize to all objects queried by ScanRefer, Multi3DRefer, ScanQA, SQA3D, and Scan2Cap.
    Top-K selection (Section 3.3) keeps only the 100 highest-objectness superpoints; if a target object is not well-represented in ScanNet200, it may be discarded before the LLM sees it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of 3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer." pith.science (2026). https://pith.science/paper/5H7ZOZK6

@misc{pith2026250101163,
  author       = {Pith},
  title        = {Pith review of: 3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5H7ZOZK6}},
  note         = {Machine review of arXiv:2501.01163}
}
read the original abstract

Current 3D Large Multimodal Models (3D LMMs) have shown tremendous potential in 3D-vision-based dialogue and reasoning. However, how to further enhance 3D LMMs to achieve fine-grained scene understanding and facilitate flexible human-agent interaction remains a challenging problem. In this work, we introduce 3D-LLaVA, a simple yet highly powerful 3D LMM designed to act as an intelligent assistant in comprehending, reasoning, and interacting with the 3D world. Unlike existing top-performing methods that rely on complicated pipelines-such as offline multi-view feature extraction or additional task-specific heads-3D-LLaVA adopts a minimalist design with integrated architecture and only takes point clouds as input. At the core of 3D-LLaVA is a new Omni Superpoint Transformer (OST), which integrates three functionalities: (1) a visual feature selector that converts and selects visual tokens, (2) a visual prompt encoder that embeds interactive visual prompts into the visual token space, and (3) a referring mask decoder that produces 3D masks based on text description. This versatile OST is empowered by the hybrid pretraining to obtain perception priors and leveraged as the visual connector that bridges the 3D data to the LLM. After performing unified instruction tuning, our 3D-LLaVA reports impressive results on various benchmarks.

Figures

Figures reproduced from arXiv: 2501.01163 by the authors.

Figure 1
Figure 1. An intuitive comparison between 3D-LLaVA and other [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of 3D-LLaVA framework. Given input point cloud, language instruction, and optional visual prompt, 3D-LLaVA [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. An illustration of (a) the architecture of Omni Superpoint [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Different paradigms to produce visual prompt embed [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Visualization of 3D-LLaVA’s response on various tasks. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing

    cs.CV 2025-06 conditional novelty 7.0 of 10

    SAVVY-Bench tests audio-visual LLMs on dynamic 3D spatial questions, and the SAVVY pipeline, combining visual tracks with spatial audio and global mapping, lifts Gemini-2.5-pro accuracy from 50.9% to 58.0%.

  2. LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understanding

    cs.CV 2025-05 conditional novelty 7.0 of 10

    A vision-language model that embeds 3D coordinates plus time into visual and linguistic tokens beats 3D-only models on dynamic scene captioning, grounding, and QA.

  3. Multi-View Relational Distillation for Spatial Reasoning with Vision-Language Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Multi-view relational distillation transfers geometric knowledge to vision-language models by matching cross-view patch similarity matrices, improving spatial reasoning with minimal overhead.

Reference graph

Works this paper leans on

68 extracted references · 39 canonical work pages · cited by 3 Pith papers

  1. [1]

    Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In European Conference on Computer Vision , pages 422–

    work page

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  3. [3]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 19129– 19139, 2022. 2, 5, 6, 7, 8

    work page 2022

  4. [4]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural in- formation processing systems, 33:1877–1901, 2020. 1

    1901

  5. [5]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS) , 33:1877–1901,

    1901

  6. [6]

    3djcg: A unified framework for joint dense captioning and visual grounding on 3d point clouds

    Daigang Cai, Lichen Zhao, Jing Zhang, Lu Sheng, and Dong Xu. 3djcg: A unified framework for joint dense captioning and visual grounding on 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16464–16473, 2022. 2, 6

    work page 2022

  7. [7]

    Scanrefer: 3d object localization in rgb-d scans using natu- ral language

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natu- ral language. In European Conference on Computer Vision, pages 202–221. Springer, 2020. 2, 5, 7

    work page 2020

  8. [8]

    D3net: A speaker-listener architecture for semi-supervised dense captioning and visual grounding in rgb-d scans

    Dave Zhenyu Chen, Qirui Wu, Matthias Nießner, and An- gel X Chang. D3net: A speaker-listener architecture for semi-supervised dense captioning and visual grounding in rgb-d scans. arXiv preprint arXiv:2112.01551, 2021. 2, 6

    arXiv 2021

Show all 68 references
  1. [9]

    End-to-end 3d dense captioning with vote2cap-detr

    Sijin Chen, Hongyuan Zhu, Xin Chen, Yinjie Lei, Gang Yu, and Tao Chen. End-to-end 3d dense captioning with vote2cap-detr. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11124– 11133, 2023. 2, 6

    2023

  2. [10]

    V ote2cap-detr++: Decoupling localization and de- scribing for end-to-end 3d dense captioning

    Sijin Chen, Hongyuan Zhu, Mingsheng Li, Xin Chen, Peng Guo, Yinjie Lei, Gang Yu, Taihao Li, and Tao Chen. V ote2cap-detr++: Decoupling localization and de- scribing for end-to-end 3d dense captioning. arXiv preprint arXiv:2309.02999, 2023. 2

    2023 arXiv

  3. [11]

    Ll3da: Visual interactive instruction tuning for omni-3d understand- ing reasoning and planning

    Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. Ll3da: Visual interactive instruction tuning for omni-3d understand- ing reasoning and planning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re...

    2024

  4. [12]

    Grounded 3d-llm with referent tokens

    Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Ruiyuan Lyu, Runsen Xu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-llm with referent tokens. arXiv preprint arXiv:2405.10370, 2024. 3, 6, 7

    2024 arXiv

  5. [13]

    Scan2cap: Context-aware dense captioning in rgb-d scans

    Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. Scan2cap: Context-aware dense captioning in rgb-d scans. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 3193–3203,

  6. [14]

    Unit3d: A unified trans- former for 3d dense captioning and visual grounding

    Zhenyu Chen, Ronghang Hu, Xinlei Chen, Matthias Nießner, and Angel X Chang. Unit3d: A unified trans- former for 3d dense captioning and visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18109–18119, 2023. 6

    2023

  7. [15]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1–113, 2023. 2

    2023

  8. [16]

    4d spatio-temporal convnets: Minkowski convolutional neural networks

    Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3075–3084,

  9. [17]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017. 5

    2017

  10. [18]

    Instructblip: Towards general- purpose vision-language models with instruction tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning,

  11. [19]

    A density-based algorithm for discovering clusters in large spatial databases with noise

    Martin Ester, Hans-Peter Kriegel, J ¨org Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, pages 226–231,

  12. [20]

    Scene-llm: Extending language model for 3d visual understanding and reasoning

    Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wen- han Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11401, 2024. 3, 6

    2024 arXiv

  13. [21]

    Segpoint: Segment any point cloud via large language model

    Shuting He, Henghui Ding, Xudong Jiang, and Bihan Wen. Segpoint: Segment any point cloud via large language model. In European Conference on Computer Vision, pages 349–367, 2024. 1, 2, 3, 6, 7

    2024

  14. [22]

    Training compute-optimal large language mod- els

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language mod- els. arXiv preprint arXiv:2203.15556, 2022. 2

    2022 arXiv

  15. [23]

    3d-llm: In- 9 jecting the 3d world into large language models

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: In- 9 jecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482–20494,

  16. [24]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 5, 6

    2021 arXiv

  17. [25]

    Chat-scene: Bridging 3d scene and large language models with object identifiers

    Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. In The Thirty- eighth Annual Conference on Neural Information P...

    2024

  18. [26]

    An embodied generalist agent in 3d world

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871, 2023. 6

    2023 arXiv

  19. [27]

    Reason3d: Searching and reasoning 3d segmentation via large language model

    Kuan-Chih Huang, Xiangtai Li, Lu Qi, Shuicheng Yan, and Ming-Hsuan Yang. Reason3d: Searching and reasoning 3d segmentation via large language model. arXiv preprint arXiv:2405.17427, 2024. 2

    2024 arXiv

  20. [28]

    Text-guided graph neural networks for re- ferring 3d instance segmentation

    Pin-Hao Huang, Han-Hung Lee, Hwann-Tzong Chen, and Tyng-Luh Liu. Text-guided graph neural networks for re- ferring 3d instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 1610– 1618, 2021. 2, 6

    2021

  21. [29]

    Multi- view transformer for 3d visual grounding

    Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi- view transformer for 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15524–15533, 2022. 2

    2022

  22. [30]

    Oneformer: One transformer to rule universal image segmentation

    Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One transformer to rule universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2989–2998, 2023. 4

    2023

  23. [31]

    More: Multi-order relation min- ing for dense captioning in 3d scenes

    Yang Jiao, Shaoxiang Chen, Zequn Jie, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. More: Multi-order relation min- ing for dense captioning in 3d scenes. arXiv preprint arXiv:2203.05203, 2022. 2, 6

    2022 arXiv

  24. [32]

    Tod3cap: Towards 3d dense captioning in outdoor scenes

    Bu Jin, Yupeng Zheng, Pengfei Li, Weize Li, Yuhang Zheng, Sujie Hu, Xinyu Liu, Jinwei Zhu, Zhijie Yan, Haiyang Sun, et al. Tod3cap: Towards 3d dense captioning in outdoor scenes. In European Conference on Computer Vision, pages 367–384. Springer, 2024. 2

    2024

  25. [33]

    Context-aware alignment and mutual masking for 3d- language pre-training

    Zhao Jin, Munawar Hayat, Yuwei Yang, Yulan Guo, and Yin- jie Lei. Context-aware alignment and mutual masking for 3d- language pre-training. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 10984–10994, 2023. 2, 6

    2023

  26. [34]

    Bi-directional contextual attention for 3d dense captioning

    Minjung Kim, Hyung Suk Lim, Soonyoung Lee, Bumsoo Kim, and Gunhee Kim. Bi-directional contextual attention for 3d dense captioning. In European Conference on Com- puter Vision, pages 385–401. Springer, 2024. 2

    2024

  27. [35]

    Oneformer3d: One transformer for unified point cloud segmentation

    Maxim Kolodiazhnyi, Anna V orontsova, Anton Konushin, and Danila Rukhovich. Oneformer3d: One transformer for unified point cloud segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20943–20953, 2024. 3

    2024

  28. [36]

    Mask-attention-free transformer for 3d in- stance segmentation

    Xin Lai, Yuhui Yuan, Ruihang Chu, Yukang Chen, Han Hu, and Jiaya Jia. Mask-attention-free transformer for 3d in- stance segmentation. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 3693–3703,

  29. [37]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9579–9589, 2024. 5

    2024

  30. [38]

    Large-scale point cloud semantic segmentation with superpoint graphs

    Loic Landrieu and Martin Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. InPro- ceedings of the IEEE conference on computer vision and pat- tern recognition, pages 4558–4567, 2018. 3

    2018

  31. [39]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. 1, 2, 3

    2023 arXiv

  32. [40]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485,

    arXiv

  33. [41]

    Sparsebev: High-performance sparse 3d object de- tection from multi-camera videos

    Haisong Liu, Yao Teng, Tao Lu, Haiguang Wang, and Limin Wang. Sparsebev: High-performance sparse 3d object de- tection from multi-camera videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 18580–18590, 2023. 4

    2023

  34. [42]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 1, 2, 5, 6

    2024

  35. [43]

    Vicuna: An open-source chatbot impressing gpt-4 with 90quality, 2023

    LMSYS.org. Vicuna: An open-source chatbot impressing gpt-4 with 90quality, 2023. https://lmsys.org. 1, 6

    2023

  36. [44]

    Sqa3d: Situated question answering in 3d scenes

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yi- tao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022. 2, 5, 7, 8

    2022 arXiv

  37. [45]

    Clip-guided vision-language pre-training for question answering in 3d scenes

    Maria Parelli, Alexandros Delitzas, Nikolas Hars, Geor- gios Vlassis, Sotirios Anagnostidis, Gregor Bachmann, and Thomas Hofmann. Clip-guided vision-language pre-training for question answering in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2023

  38. [46]

    Openscene: 3d scene understanding with open vocabularies

    Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 815–824, 2023. 2, 5

    2023

  39. [47]

    Pointnet++: Deep hierarchical feature learning on point sets in a metric space

    Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017. 3, 4

    2017

  40. [48]

    X- refseg3d: Enhancing referring 3d instance segmentation via structured cross-modal graph neural networks

    Zhipeng Qian, Yiwei Ma, Jiayi Ji, and Xiaoshuai Sun. X- refseg3d: Enhancing referring 3d instance segmentation via structured cross-modal graph neural networks. In Proceed- 10 ings of the AAAI Conference on Artificial Intelligence, pages 4551–4559, 2024. 2, 6

    2024

  41. [49]

    Language models are unsu- pervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners. OpenAI blog, 1(8):9, 2019. 1

    2019

  42. [50]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  43. [51]

    Language- grounded indoor 3d semantic segmentation in the wild

    David Rozenberszki, Or Litany, and Angela Dai. Language- grounded indoor 3d semantic segmentation in the wild. In European Conference on Computer Vision, pages 125–141,

  44. [52]

    Mask3d: Mask trans- former for 3d semantic instance segmentation

    Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask trans- former for 3d semantic instance segmentation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8216–8223. IEEE, 2023. 4, 7

    2023

  45. [53]

    Superpoint transformer for 3d scene instance segmentation

    Jiahao Sun, Chunmei Qing, Junpeng Tan, and Xiangmin Xu. Superpoint transformer for 3d scene instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 2393–2401, 2023. 3, 4

    2023

  46. [54]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 2

    2023 arXiv

  47. [55]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 1, 2

    2023 arXiv

  48. [56]

    Four ways to improve verbo-visual fusion for dense 3d visual grounding

    Ozan Unal, Christos Sakaridis, Suman Saha, and Luc Van Gool. Four ways to improve verbo-visual fusion for dense 3d visual grounding. InEuropean Conference on Com- puter Vision, pages 196–213. Springer, 2024. 7

    2024

  49. [57]

    Spatiality-guided transformer for 3d dense captioning on point clouds

    Heng Wang, Chaoyi Zhang, Jianhui Yu, and Weidong Cai. Spatiality-guided transformer for 3d dense captioning on point clouds. arXiv preprint arXiv:2204.10688, 2022. 6

    2022 arXiv

  50. [58]

    3d-stmn: Dependency- driven superpoint-text matching network for end-to-end 3d referring expression segmentation

    Changli Wu, Yiwei Ma, Qi Chen, Haowei Wang, Gen Luo, Jiayi Ji, and Xiaoshuai Sun. 3d-stmn: Dependency- driven superpoint-text matching network for end-to-end 3d referring expression segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 5940– 59...

    2024

  51. [59]

    Pointllm: Empowering large language models to understand point clouds

    Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiang- miao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. arXiv preprint arXiv:2308.16911, 2023. 2

    2023 arXiv

  52. [60]

    Sat: 2d semantics assisted training for 3d visual grounding

    Zhengyuan Yang, Songyang Zhang, Liwei Wang, and Jiebo Luo. Sat: 2d semantics assisted training for 3d visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1856–1866, 2021. 2

    2021

  53. [61]

    X-trans2cap: Cross- modal knowledge transfer using transformer for 3d dense captioning

    Zhihao Yuan, Xu Yan, Yinghong Liao, Yao Guo, Guan- bin Li, Shuguang Cui, and Zhen Li. X-trans2cap: Cross- modal knowledge transfer using transformer for 3d dense captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8563– 8573...

    2022

  54. [62]

    Agent3d-zero: An agent for zero-shot 3d understanding

    Sha Zhang, Di Huang, Jiajun Deng, Shixiang Tang, Wanli Ouyang, Tong He, and Yanyong Zhang. Agent3d-zero: An agent for zero-shot 3d understanding. arXiv preprint arXiv:2403.11835, 2024. 3

    2024 arXiv

  55. [63]

    Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding

    Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. arXiv preprint arXiv:2406.19389, 2024. 2

    2024 arXiv

  56. [64]

    Multi3drefer: Grounding text description to multiple 3d ob- jects

    Yiming Zhang, ZeMing Gong, and Angel X Chang. Multi3drefer: Grounding text description to multiple 3d ob- jects. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 15225–15236, 2023. 2, 5, 6, 7

    2023

  57. [65]

    Contextual modeling for 3d dense captioning on point clouds

    Yufeng Zhong, Long Xu, Jiebo Luo, and Lin Ma. Contextual modeling for 3d dense captioning on point clouds. arXiv preprint arXiv:2210.03925, 2022. 8

    2022 arXiv

  58. [66]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1

    2023 arXiv

  59. [67]

    3d-vista: Pre-trained transformer for 3d vision and text alignment

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2911– 2921, 2023. 2, 6, 7 11

    2023

  60. [440]

    Springer, 2020. 2, 5

    2020

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.