SAVVY-Bench tests audio-visual LLMs on dynamic 3D spatial questions, and the SAVVY pipeline, combining visual tracks with spatial audio and global mapping, lifts Gemini-2.5-pro accuracy from 50.9% to 58.0%.
3d-llava: Towards generalist 3d lmms with omni superpoint transformer
6 Pith papers cite this work. Polarity classification is still indexing.
6
Pith papers citing it
abstract
Current 3D Large Multimodal Models (3D LMMs) have shown tremendous potential in 3D-vision-based dialogue and reasoning. However, how to further enhance 3D LMMs to achieve fine-grained scene understanding and facilitate flexible human-agent interaction remains a challenging problem. In this work, we introduce 3D-LLaVA, a simple yet highly powerful 3D LMM designed to act as an intelligent assistant in comprehending, reasoning, and interacting with the 3D world. Unlike existing top-performing methods that rely on complicated pipelines-such as offline multi-view feature extraction or additional task-specific heads-3D-LLaVA adopts a minimalist design with integrated architecture and only takes point clouds as input. At the core of 3D-LLaVA is a new Omni Superpoint Transformer (OST), which integrates three functionalities: (1) a visual feature selector that converts and selects visual tokens, (2) a visual prompt encoder that embeds interactive visual prompts into the visual token space, and (3) a referring mask decoder that produces 3D masks based on text description. This versatile OST is empowered by the hybrid pretraining to obtain perception priors and leveraged as the visual connector that bridges the 3D data to the LLM. After performing unified instruction tuning, our 3D-LLaVA reports impressive results on various benchmarks.
citation-role summary
background 2
citation-polarity summary
fields
cs.CV 6roles
background 2polarities
background 2representative citing papers
A vision-language model that embeds 3D coordinates plus time into visual and linguistic tokens beats 3D-only models on dynamic scene captioning, grounding, and QA.
Spatial-MLLM adds a 3D spatial encoder initialized from a visual geometry model and space-aware frame sampling to MLLMs to improve spatial understanding and reasoning from purely 2D visual inputs.
A navigation agent trained to also predict how its top-down map changes after each move reaches higher success on R2R-CE, RxR-CE, and real-robot trials.
MAG-3D is a training-free multi-agent framework that coordinates planning, grounding, and coding agents with off-the-shelf VLMs to achieve grounded 3D reasoning and state-of-the-art benchmark results.
citing papers explorer
-
SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing
SAVVY-Bench tests audio-visual LLMs on dynamic 3D spatial questions, and the SAVVY pipeline, combining visual tracks with spatial audio and global mapping, lifts Gemini-2.5-pro accuracy from 50.9% to 58.0%.
-
LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understanding
A vision-language model that embeds 3D coordinates plus time into visual and linguistic tokens beats 3D-only models on dynamic scene captioning, grounding, and QA.
-
Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence
Spatial-MLLM adds a 3D spatial encoder initialized from a visual geometry model and space-aware frame sampling to MLLMs to improve spatial understanding and reasoning from purely 2D visual inputs.
-
InsightMap: Structured Spatial Modeling for Embodied Multimodal Reasoning
A navigation agent trained to also predict how its top-down map changes after each move reaches higher success on R2R-CE, RxR-CE, and real-robot trials.
-
MAG-3D: Multi-Agent Grounded Reasoning for 3D Understanding
MAG-3D is a training-free multi-agent framework that coordinates planning, grounding, and coding agents with off-the-shelf VLMs to achieve grounded 3D reasoning and state-of-the-art benchmark results.
- Multi-View Relational Distillation for Spatial Reasoning with Vision-Language Models