Pith

open record

sign in

arxiv: 2307.12981 · v1 · pith:4MPALBNC · submitted 2023-07-24 · cs.CV · cs.AI· cs.CL· cs.LG· cs.RO

3D-LLM: Injecting the 3D World into Large Language Models

Yining Hong , Haoyu Zhen , Peihao Chen , Shuhong Zheng , Yilun Du , Zhenfang Chen , Chuang Gan This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:4MPALBNCrecord.jsonopen to challenge →

classification cs.CV cs.AIcs.CLcs.LGcs.RO
keywords d-llmsmodelslargetasksvlmscaptioninglanguagemodel
0
0 comments X
read the original abstract

Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs. Project Page: : https://vis-www.cs.umass.edu/3dllm/.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Consistent Yet Wrong: Evidence Insensitivity in Spatial Vision-Language Models

    cs.CV 2026-06 unverdicted novelty 7.0

    VLMs produce highly consistent but inaccurate answers to metric spatial queries across views, indicating prior-driven collapse rather than evidence-sensitive reasoning.

  2. Abstract 3D Perception for Spatial Intelligence in Vision-Language Models

    cs.CV 2025-11 unverdicted novelty 7.0

    SandboxVLM enhances VLMs' spatial intelligence by encoding 3D geometry with abstract bounding boxes in a four-stage zero-shot pipeline, yielding an 8.3% improvement on SAT Real benchmark.

  3. 3D-VLA: A 3D Vision-Language-Action Generative World Model

    cs.CV 2024-03 unverdicted novelty 7.0

    3D-VLA is a new embodied foundation model that uses a 3D LLM plus aligned diffusion models to generate future images and point clouds for improved reasoning and action planning in 3D environments.

  4. CAIRN: Cross-Room 3D Scene Understanding with Topology-Aware Large Multimodal Models

    cs.CV 2026-07 conditional novelty 6.0

    A topology-aware 3D-LLM with hierarchical masked attention and geometric bias outperforms prior 3D-LLMs on a new multi-room scene understanding benchmark built from HM3D.

  5. Consistent Yet Wrong: Evidence Insensitivity in Spatial Vision-Language Models

    cs.CV 2026-06 conditional novelty 6.0

    Leading VLMs show high cross-view consistency paired with low metric accuracy on distance queries, indicating evidence-insensitive reasoning rather than geometric grounding.

  6. Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLM

    cs.CV 2026-03 unverdicted novelty 6.0

    Chat-Scene++ improves 3D scene understanding in multimodal LLMs by representing scenes as context-rich object sequences with identifier tokens and grounded chain-of-thought reasoning, reaching state-of-the-art on five...

  7. Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representations

    cs.CV 2026-03 unverdicted novelty 6.0

    GR3D turns 3D scene geometry into ID-indexed text references, enabling zero-shot MLLM spatial reasoning gains of 9% on VSI-Bench and 12% on MindCube.

  8. CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models

    cs.CV 2026-01 unverdicted novelty 5.0

    CG-MLLM is a multimodal LLM using a Mixture-of-Transformer architecture with separate TokenAR and BlockAR components integrated with a pre-trained vision-language backbone and 3D VAE to enable 3D captioning and high-f...

  9. SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models

    cs.CV 2023-11 unverdicted novelty 5.0

    SPHINX improves multi-modal LLMs through joint mixing of weights, tasks, and visual embeddings from varied sources to achieve stronger alignment and multi-purpose capabilities.