Pith. sign in

REVIEW 36 cited by

Grounded 3D-LLM with Referent Tokens

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.10370 v2 pith:JC6ZRCGT submitted 2024-05-16 cs.CV

Yilun Chen , Shuai Yang , Haifeng Huang , Tai Wang , Runsen Xu , Ruiyuan Lyu , Dahua Lin , Jiangmiao Pang This is my paper · ORCID
classification cs.CV
keywords groundedtasksd-llmlanguagereferenttokensdatagithub
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Prior studies on 3D scene understanding have primarily developed specialized models for specific tasks or required task-specific fine-tuning. In this study, we propose Grounded 3D-LLM, which explores the potential of 3D large multi-modal models (3D LMMs) to consolidate various 3D vision tasks within a unified generative framework. The model uses scene referent tokens as special noun phrases to reference 3D scenes, enabling it to handle sequences that interleave 3D and textual data. Per-task instruction-following templates are employed to ensure natural and diversity in translating 3D vision tasks into language formats. To facilitate the use of referent tokens in subsequent language modeling, we provide a large-scale, automatically curated grounded scene-text dataset with over 1 million phrase-to-region correspondences and introduce Contrastive Language-Scene Pre-training (CLASP) to perform phrase-level scene-text alignment using this data. Our comprehensive evaluation covers open-ended tasks like dense captioning and 3D question answering, alongside close-ended tasks such as object detection and language grounding. Experiments across multiple 3D benchmarks reveal the leading performance and the broad applicability of Grounded 3D-LLM. Code and datasets are available at the https://groundedscenellm.github.io/grounded_3d-llm.github.io.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 36 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SocietyBench: Forecasting Counterfactual Social-World Evolution

    cs.CL 2026-08 conditional novelty 8.0 of 10

    SocietyBench anonymizes five real social-event timelines into counterfactual worlds and reports that the best of six frontier LLMs reaches only 75.0 out of 100, with calibration and temporal accuracy dissociating.

  2. LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understanding

    cs.CV 2025-05 conditional novelty 7.0 of 10

    A vision-language model that embeds 3D coordinates plus time into visual and linguistic tokens beats 3D-only models on dynamic scene captioning, grounding, and QA.

  3. Multi-View Relational Distillation for Spatial Reasoning with Vision-Language Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Multi-view relational distillation transfers geometric knowledge to vision-language models by matching cross-view patch similarity matrices, improving spatial reasoning with minimal overhead.

  4. SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding

    cs.CV 2026-08 conditional novelty 6.0 of 10

    SmartMage dynamically selects the most relevant modalities per query and routes tokens to specialized experts, achieving state-of-the-art results on five 3D scene understanding benchmarks.

  5. Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Qwen-3D feeds the full visual-language representation of a 3D scene into a mask decoder, beating prior 3D LMMs on grounding and segmentation benchmarks.

  6. 3D-Aware VLMs with Implicit and Explicit Geometries

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Fusing a VLM's normal 2D tokens with implicit geometry tokens from a video-geometry encoder plus tokens from its own reconstructed depth maps improves 3D detection, grounding, captioning, and spatial reasoning.

  7. Beyond Single Expert: Harmonizing Diverse Visual Priors in MLLMs for Spatial Understanding

    cs.CV 2026-07 conditional novelty 6.0 of 10

    ViPS fuses five complementary visual priors into an MLLM via lightweight distillation proxies and query-conditioned dynamic weighting, reporting state-of-the-art results on VSI-Bench and ScanNet-series benchmarks.

  8. Self in Space: Benchmarking Self-Awareness and Spatial Cognition in UAV Embodied Intelligence

    cs.CV 2026-07 unverdicted novelty 6.0 of 10

    SIS-Bench shows MLLMs lag on UAV self-awareness versus spatial tasks, and motion-aware optical-flow fusion improves perception and memory.

  9. CAIRN: Cross-Room 3D Scene Understanding with Topology-Aware Large Multimodal Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Topology-aware attention over hierarchical scene graphs lets a 3D-LLM ground, caption, and answer questions across multi-room homes, with large gains on a new HM3D benchmark.

  10. Seeing Once is Enough? Online Geometry-Aware Token Pruning for 3D Question Answering

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Online geometry-aware voxel overlap pruning removes up to 50% of visual tokens from multi-view 3D scenes while improving zero-shot 3D QA on Qwen VL models.

  11. EgoMind: Activating Spatial Cognition through Linguistic Reasoning in MLLMs

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    EgoMind uses Role-Play Caption and Progressive Spatial Analysis to give MLLMs competitive multi-frame spatial reasoning without 3D priors, using only 5K SFT and 20K RL samples.

  12. Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding

    cs.CV 2026-03 unverdicted novelty 6.0 of 10

    VEGA-3D extracts intermediate spatiotemporal features from a pretrained video diffusion model and fuses them into MLLMs to improve geometric and embodied reasoning without explicit 3D supervision.

  13. GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Models

    cs.CV 2026-03 conditional novelty 6.0 of 10

    A sparse pointmap-plus-semantics pre-training stage with multi-level gated fusion of geometric and visual tokens improves RGB-only 3D perception in multimodal LLMs.

  14. JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments

    cs.CV 2026-02 conditional novelty 6.0 of 10

    JAEGER extends audio-visual LLMs with RGB-D and first-order ambisonics to achieve accurate 3D sound-source grounding and reasoning in simulated environments.

  15. MeshLLM: Empowering Large Language Models to Progressively Understand and Generate 3D Mesh

    cs.GR 2025-08 unverdicted novelty 6.0 of 10

    MeshLLM improves LLM-based 3D mesh understanding and generation through primitive decomposition, a 1500k+ sample dataset, and topology-focused training strategies.

  16. AutoPartGen: Autogressive 3D Part Generation and Discovery

    cs.CV 2025-07 conditional novelty 6.0 of 10

    AutoPartGen generates 3D objects as a sequence of latent-space parts, conditioning each new part on previously generated parts, and reports state-of-the-art part completion on PartObjaverse-Tiny.

  17. SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A large-scale 3D spatial reasoning segmentation benchmark with human-written queries that avoid object names shows current 3D vision-language models underperform.

  18. GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyond

    cs.CV 2025-07 conditional novelty 6.0 of 10

    GaussianVLM embeds language features per Gaussian splat, sparsifies them by task and location, and reports state-of-the-art results on embodied 3D reasoning benchmarks.

  19. Enhancing Spatial Reasoning in Multimodal Large Language Models through Reasoning-based Segmentation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A two-stage reasoning-segmentation method plus a new LLM-generated 3D dataset improves spatial reasoning in 3D multimodal large language models on several benchmarks.

  20. Pts3D-LLM: Studying the Impact of Token Structure for 3D Scene Understanding With Large Language Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A 3D vision-language model that merges point-cloud features with image tokens and uses a new view-aware point sampling method (FPS6D) achieves state-of-the-art normalized scores on ScanNet benchmarks.

  21. SORT3D: Spatial Object-centric Reasoning Toolbox for Zero-Shot 3D Grounding Using Large Language Models

    cs.CV 2025-04 conditional novelty 6.0 of 10

    SORT3D is a zero-shot 3D grounding system where an LLM calls hand-built spatial functions and uses 2D captions, matching or beating prior zero-shot methods on several view-dependent subsets while running on real robots.

  22. 3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer

    cs.CV 2025-01 conditional novelty 6.0 of 10

    A multi-purpose Omni Superpoint Transformer lets a single 3D large multimodal model achieve state-of-the-art results on 3D question answering, dense captioning, and referring segmentation using point clouds only.

  23. 3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A learnable 3D scene graph representation that injects semantic relation embeddings between objects improves LLM performance on 3D grounding, captioning, and question answering.

  24. Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding

    cs.CV 2024-11 conditional novelty 6.0 of 10

    Adding 3D coordinate encodings to sampled RGB-D video frames lets a video LLM beat prior 3D scene understanding models on five benchmarks.

  25. g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Tasks

    cs.CV 2024-11 conditional novelty 6.0 of 10

    A 3D-language feature field pretrained on about 5,000 indoor scenes with nearly a million text descriptions improves vision-language navigation, zero-shot object navigation, and situated question answering.

  26. Multimodal 3D Reasoning Segmentation with Complex Scenes

    cs.CV 2024-11 conditional novelty 6.0 of 10

    A new benchmark and model enable multi-object 3D reasoning segmentation, where a point-cloud scene and a question produce both explanations and masks for several objects at once.

  27. ViewMind3D: Modular View-Aware Inference for Training-Free 3D-QA

    cs.CV 2026-07 conditional novelty 5.0 of 10

    A four-stage training-free pipeline of view selection, object grounding, BEV pose cues, and role-based LLM reasoning reaches competitive ScanQA/SQA3D scores using general VLMs.

  28. Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Models

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Spatial 3D-LLM adds a progressive spatial awareness scheme to a 3D vision-language model, improving several 3D understanding and grounding metrics and introducing new distance and layout-editing tasks.

  29. Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Fast3D prunes up to 90% of object-centric visual tokens in 3D MLLMs while preserving about 96.8% of original benchmark performance, using a trained attention predictor and adaptive layer-wise pruning.

  30. Does Your 3D Encoder Really Work? When Pretrain-SFT from 2D VLMs Meets 3D VLMs

    cs.CV 2025-06 conditional novelty 5.0 of 10

    3D scene-centric VLMs underutilize the 3D encoder, overfit to linguistic and answer-frequency shortcuts, and the proposed 3D-RDQA dataset helps expose and mitigate this problem.

  31. Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A sparse mixture-of-experts 3D multimodal LLM adaptively fuses RGB, RGBD, BEV, point cloud, and voxel tokens, achieving SOTA on several ScanNet-based 3D scene understanding benchmarks.

  32. RAZER: Robust Accelerated Zero-Shot 3D Open-Vocabulary Panoptic Reconstruction with Spatio-Temporal Aggregation

    cs.CV 2025-05 conditional novelty 5.0 of 10

    RAZER fuses online TSDF reconstruction with open-vocabulary instance embeddings and tracking to produce real-time, queryable 3D semantic maps without any 3D training.

  33. Hyperbolic Contrastive Learning for Hierarchical 3D Point Cloud Embedding

    cs.CV 2025-01 conditional novelty 5.0 of 10

    Adding hierarchy-encouraging losses to hyperbolic text-image-point cloud contrastive training yields hierarchical 3D point cloud embeddings with small downstream gains.

  34. PerLA: Perceptive 3D Language Assistant

    cs.CV 2024-11 conditional novelty 5.0 of 10

    PerLA improves 3D question answering and dense captioning by combining Hilbert-curve partitioned local point-cloud details with global context through cross-attention and graph convolution.

  35. RelMap: Reliable Spatiotemporal Sensor Data Visualization via Imputative Spatial Interpolation

    cs.LG 2025-08 unverdicted novelty 4.0 of 10

    RelMap combines GNN-based imputation with spatial interpolation and uncertainty-aware heatmaps for spatiotemporal sensor data.

  36. From 2D to 3D Cognition: A Brief Survey of General World Models

    cs.CV 2025-06 conditional novelty 3.0 of 10

    A survey proposing a two-pillar, three-capability framework that organizes recent AI world models by their transition from 2D visual prediction to 3D cognition.

Pith tools