pith. sign in

arxiv: 2503.13111 · v2 · pith:HI7OBLFCnew · submitted 2025-03-17 · 💻 cs.CV · cs.CL· cs.LG

MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs

Erik Daxberger , Nina Wenzel , David Griffiths , Haiming Gang , Justin Lazarow , Gefen Kohavi , Kai Kang , Marcin Eichner
show 3 more authors
Yinfei Yang Afshin Dehghan Peter Grasch
This is my paper
classification 💻 cs.CV cs.CLcs.LG
keywords spatialunderstandingca-vqadatadepthestimationincludingmetric
0
0 comments X
read the original abstract

Multimodal large language models (MLLMs) excel at 2D visual understanding but remain limited in their ability to reason about 3D space. In this work, we leverage large-scale high-quality 3D scene data with open-set annotations to introduce 1) a novel supervised fine-tuning dataset and 2) a new evaluation benchmark, focused on indoor scenes. Our Cubify Anything VQA (CA-VQA) data covers diverse spatial tasks including spatial relationship prediction, metric size and distance estimation, and 3D grounding. We show that CA-VQA enables us to train MM-Spatial, a strong generalist MLLM that also achieves state-of-the-art performance on 3D spatial understanding benchmarks, including our own. We show how incorporating metric depth and multi-view inputs (provided in CA-VQA) can further improve 3D understanding, and demonstrate that data alone allows our model to achieve depth perception capabilities comparable to dedicated monocular depth estimation models.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoning

    cs.CV 2026-06 unverdicted novelty 8.0

    A blank-image ablation test reveals that high probe accuracy on VLM spatial reasoning frequently reflects priors or inverted signs rather than image grounding, with horizontal grounded, vertical prior, and depth inverted.

  2. World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoning

    cs.CV 2026-04 unverdicted novelty 6.0

    Distilling view-consistent future views and action-outcome supervision from a generative world model into a VLM via two-stage post-training improves dynamic spatial reasoning on SAT-Real, VSI-Bench and similar benchma...

  3. Multimodal Language Models Cannot Spot Spatial Inconsistencies

    cs.CV 2026-04 unverdicted novelty 6.0

    Multimodal LLMs significantly underperform humans at spotting objects that break 3D consistency in multi-view image pairs.

  4. Lost in Space? Vision-Language Models Struggle with Relative Camera Pose Estimation

    cs.CV 2026-01 unverdicted novelty 6.0

    VLMs reach only 0.66 accuracy on relative camera pose estimation while humans achieve 0.91 and specialized pipelines reach 0.99, exposing weaknesses in multi-view spatial reasoning.

  5. SpaCE: Rethinking Spatial Capacity and Generalization in Multi-Frame Multimodal Large Language Models

    eess.IV 2026-06 unverdicted novelty 5.0

    SpaCE derives four theoretical results on spatial capacity, sample complexity, generalization, and bias-variance trade-offs for multi-frame MLLM reasoning, validated on MultiSPA, CA-VQA, and SpatialRGPT.