Pith. sign in

REVIEW 4 major objections 5 minor 62 references

A new dataset with over one million monocular spatial-reasoning question-answer pairs shows that current vision-language models struggle most with 3D geometry, not object recognition.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 20:33 UTC pith:MK5GEPM5

load-bearing objection A genuinely useful monocular spatial reasoning benchmark built on Omni3D, but the missing dataset release and unquantified label/human validation make the headline claims provisional. the 4 major comments →

arxiv 2511.19119 v2 pith:MK5GEPM5 submitted 2025-11-24 cs.CV

MonoSR: Open-Vocabulary Spatial Reasoning from Monocular Images

Qirui Wang , Jingyi He , Yining Pan , Si Yong Yeo , Xulei Yang , Shijie Li This is my paper
classification cs.CV
keywords monocular spatial reasoningvision-language models3D bounding boxesvisual question answeringopen-world datasetscene graphauxiliary informationbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that existing spatial-reasoning benchmarks are too narrow—indoor-only, multi-view, or video-based—and that the most common real-world setting, a single monocular image, has been left without a large-scale training and evaluation resource. MonoSR fills that gap: over 1.02 million QA pairs built from 230,000+ images spanning indoor, outdoor, and object-centric scenes, with all answers computed deterministically from ground-truth 3D bounding boxes. The paper further claims that current vision-language models are far from adequate on this task, and, through a controlled auxiliary-information study, that the binding constraint is recovery of explicit 3D geometry rather than object recognition or scene semantics. If correct, MonoSR provides a foundation for training and benchmarking open-world monocular spatial reasoning, a capability needed for embodied AI and autonomous driving.

Core claim

MonoSR is a large-scale monocular spatial reasoning dataset: more than 1.02 million QA pairs built from over 230,000 images, spanning indoor, outdoor, and object-centric scenes across 98 object categories. Questions are organized into three hierarchical levels—foundational 3D perception (spatial relationship, distance, size, dimension comparison), perspective-aware imagination (occlusion, viewpoint-relative relationship, object grounding), and situational reasoning (questions embedded in professional scenarios). All answers are derived deterministically from ground-truth 3D bounding boxes sourced from Omni3D, through a pipeline that filters scenes, builds a scene graph of captioned objects,

What carries the argument

The load-bearing mechanism is MonoSR's QA-generation pipeline, which converts Omni3D ground-truth 3D bounding boxes into questions whose answers are deterministic functions of box center, dimensions, orientation, and camera parameters. A scene graph of fine-grained captioned objects feeds handcrafted templates; an LLM then paraphrases the question text under strict constraints that the computed answer remains unchanged. The second mechanism is the auxiliary-information experiment: the same Qwen2.5-VL-3B backbone is fine-tuned with scene information, 2D visual prompts, 3D bounding boxes, and their combinations as input, isolating which information source drives spatial accuracy.

Load-bearing premise

The dataset's answers inherit Omni3D's 3D bounding-box annotations without independent re-verification; if those boxes are noisy, misaligned, or inconsistently defined across the many source datasets, a large fraction of the million answers are wrong, which would undercut both the benchmark numbers and the auxiliary-information conclusions.

What would settle it

Take a random sample of MonoSR questions that require numeric values (distance, size, footprint, volume), re-project the corresponding Omni3D 3D boxes into the images using the published camera intrinsics, and compare the computed answers with measurements a human would make from the image (or from the original sensor data such as LiDAR or RGB-D where available). If a substantial fraction of answers differ by more than the paper's own 10% tolerance, the ground truth is contaminated and the reported model rankings and auxiliary-information effects are not trustworthy.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Existing VLMs are far from reliable monocular spatial reasoners: the best models score below 0.44 overall on MonoSR, with the weakest results on object-centric and quantitative questions.
  • Explicit 3D geometry, not image semantics, is the dominant driver of accuracy: a 3B model given 3D bounding boxes plus 2D prompts reaches near-perfect scores on many tasks.
  • MonoSR's training split provides a supervised resource for learning open-world monocular spatial reasoning across indoor, outdoor, and object-centric scenes.
  • Adding irrelevant or excessive auxiliary cues degrades performance, so simply supplying more context to a VLM is not a reliable route to spatial competence.
  • The hierarchical task levels (perception, imagination, situational reasoning) offer a diagnostic ladder for separating low-level geometric failures from reasoning failures in future models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: feeding the model noisy 3D estimates from a monocular detector at inference time, rather than oracle Omni3D boxes, would test whether the near-perfect auxiliary-information results survive real-world label noise.
  • Beyond the paper: MonoSR inherits Omni3D's box approximations of real object extents; recomputing answers from finer point clouds or meshes on a subset of scenes would test how sensitive model rankings are to box-level ground truth.
  • Beyond the paper: a stress test for the open-world claim is to fine-tune on MonoSR and evaluate on images from unseen sources (user photos, aerial views, synthetic scenes); a large drop would indicate dataset-specific heuristics rather than general spatial understanding.
  • Beyond the paper: numeric answers are judged with a fixed 10% relative-error tolerance; recomputing rankings at stricter tolerances (e.g., 1% or 5%) would show whether conclusions about VLM limits are threshold-dependent.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces MonoSR, a large-scale monocular spatial reasoning dataset constructed from Omni3D: over 1.02M QA pairs from 230K+ images spanning indoor, outdoor, and object-centric domains, with 8 task types organized into three cognitive levels. Answers are derived deterministically from Omni3D 3D ground-truth bounding boxes via scene graphs and handcrafted templates, with LLM-based paraphrase for linguistic diversity. The paper benchmarks six open- and closed-source VLMs on MonoSR, reports generally low accuracies, and fine-tunes Qwen2.5-VL-3B under different auxiliary information conditions (scene type, 2D visual prompts, 3D bounding boxes), finding large gains from explicit 3D geometry. The central claim is that MonoSR provides a foundation for open-world monocular spatial reasoning and that current VLMs primarily lack fine-grained 3D geometric information rather than object recognition.

Significance. If the dataset is released and its labels independently validated, MonoSR would fill a real gap: most prior 3D spatial reasoning benchmarks are indoor-only, multi-view, or reconstruction-based. The deterministic answer generation from 3D boxes is a reproducibility strength, and the controlled auxiliary-information comparison is a useful protocol for guiding monocular 3D reasoning research. The VLM evaluation is broad across model families. However, the contribution is currently conditional: the dataset artifact is not shipped, inherited Omni3D label noise is unquantified, and the headline 3D-bbox conclusion is partly circular because the model is given the same ground-truth geometry used to generate the answers. These issues are load-bearing for the paper's central claims.

major comments (4)
  1. [Sec. 3.3 / Availability] MonoSR's central artifact is not linked or downloadable. The paper references a project page and code links but provides no dataset URL, release license, split files, or download instructions. Without the dataset, the 1M+ QA pairs, the VLM benchmark, and the fine-tuning experiments cannot be reproduced or used by the community. A dataset paper must ship the dataset (or a hosted subset) and provide a datasheet with splits, category frequencies, and annotation protocols.
  2. [Sec. 3.2 / Sec. 3.3] All QA answers are derived from Omni3D 3D bounding boxes, but the paper provides no independent verification of those boxes. The reported 'geometric consistency checks against 3D annotations' only verify pipeline self-consistency, not external correctness. The human validation is described only as a 'large, randomly sampled subset' with no N, sampling method, per-task error rates, or inter-annotator agreement. Because Omni3D aggregates heterogeneous datasets (KITTI, nuScenes, SUN RGB-D, ARKitScenes, Hypersim) with different sensors and label conventions, label noise propagates directly into numeric answers, relative labels, and occlusion judgments. Quantify this error and its effect on benchmark numbers.
  3. [Sec. 5.3 / Tab. 2] The 3D-bbox auxiliary condition is circular: the model receives the exact center, size, and orientation from the same ground-truth boxes used to generate the answers. Near-ceiling accuracy (e.g., 0.986 object-centric overall) is therefore expected by construction. The conclusion that 'the primary bottleneck is not object recognition, but the recovery of fine-grained spatial attributes' is too strong; the condition is an upper bound, not a realistic input signal. Add a condition using predicted 3D boxes from an off-the-shelf monocular detector, or rephrase the conclusion accordingly.
  4. [Sec. 5.1 / Eq. (1) / Sec. 11.1] Benchmark numbers are single-run with no error bars or significance tests, and the 10% numeric tolerance is arbitrary without sensitivity analysis. Some reported numbers are extremely low (e.g., High-Dist indoor 0.002 for Gemini-2.5-Pro; object-centric overall 0.069 for ChatGPT-4), which is consistent with evaluation instability. Additionally, 'deterministic decoding with temperature 1.0' is internally contradictory: temperature 1.0 is stochastic. Report the exact decoding settings and the variance across multiple runs or seeds.
minor comments (5)
  1. [Sec. 3.1 / Fig. 1 / Fig. 3 / Sec. 8.1] Task count is inconsistent: Sec. 3.1 says 'nine tasks', while Fig. 1, Fig. 3, and Sec. 8.1 list eight. Align the text and figures.
  2. [Throughout] Typos: 'Futhuremore' (Sec. 1), 'Distnce' (Sec. 8.1), 'examplre' (Sec. 3.2), '3b Bbox' (Sec. 4), 'F oundational' (Sec. 3.2).
  3. [Title / Sec. 3.3] The term 'open-vocabulary' is not defined; the dataset is built from 98 fixed Omni3D categories. Clarify whether 'open-vocabulary' refers to free-form captions or to an open-set evaluation protocol.
  4. [Sec. 5.1] Eq. (1) is not numbered and the threshold rule is stated as a strict inequality; add a number and clarify handling of exact ties (|d-dhat|/d exactly 0.1).
  5. [Fig. 12] The situational template examples show answers labeled 'A: yes' for what appear to be numeric/volume questions; ensure the template and example answers are consistent in type and format.

Circularity Check

2 steps flagged

Auxiliary 3D-bbox condition is a definitional oracle: the model is handed the exact Omni3D coordinates from which answers were computed, so its near-ceiling results do not support the paper's 'primary bottleneck' conclusion.

specific steps
  1. self definitional [Sec. 3.2 (QA Data Generation), Sec. 4 (3D Bounding Box), Sec. 5.3 / Tab. 2]
    ""all answers are deterministically derived from the 3D ground-truth annotations. For instance, answers for Foundational 3D Perception tasks are computed directly from object coordinates" (Sec. 3.2); "3D Bounding Box (3D Bbox): Represents each object using explicit 3D center coordinates, spatial dimensions, and orientation, and is provided on the textual side" (Sec. 4); Tab. 2 Object-Centric shows 0.986 overall, 1.000 Dist, 0.994 Size under "3D bbox"."

    The QA answers (distance, size, spatial relation) are defined as deterministic functions of the Omni3D 3D boxes. The 3D-bbox auxiliary condition feeds those same boxes into the prompt and asks for the answer, so the model can read off the exact center/size/rotation values. The near-ceiling numbers are therefore forced by construction; the condition bypasses monocular 3D recovery entirely. The paper's conclusion that the "primary bottleneck is not object recognition, but the recovery of fine-grained spatial attributes from monocular inputs" is an interpretive leap from an oracle condition that supplies those attributes rather than testing their recovery.

  2. other [Sec. 3.3 (Statistics & Analysis)]
    ""To ensure reliability, we conducted both automated and human validations. Geometric consistency checks against 3D annotations achieve exceptionally high correctness for perception-level tasks.""

    Because the answers were "deterministically derived from the 3D ground-truth annotations" (Sec. 3.2), a geometric consistency check against those same 3D annotations only verifies that the deterministic generator performed the intended arithmetic on the input boxes. It cannot detect errors in the Omni3D annotations themselves, so the reported "exceptionally high correctness" is partly by construction. The human inspection is an independent check, but its protocol and error rate are not specified.

full rationale

The central artifact, the MonoSR QA dataset, is not circular: it is externally grounded in Omni3D's 3D annotations, and the answers are deterministic text renderings of those annotations into spatial questions. There is no load-bearing self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via citation. The circularity is confined to two places. First, the auxiliary-information study's '3D bbox' condition feeds the model the exact Omni3D center/size/rotation coordinates from which the ground-truth answers are computed, so its near-perfect accuracy is a definitional oracle, not evidence that a monocular model's bottleneck is 'recovery of fine-grained spatial attributes'; the paper even acknowledges in Limitations that the setup 'relies on ground-truth spatial information.' Second, the geometric consistency validation checks answers against the same annotations used to generate them, so it is a self-consistency check rather than independent label validation. These issues are partial and non-central: the benchmark itself remains externally grounded, and the paper is transparent about its reliance on ground-truth boxes. Score 4 reflects one clearly construction-forced analysis and one self-referential validation, without the central dataset claim reducing to its inputs.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central claims rest on the fidelity of Omni3D ground-truth boxes and on template/LLM generation preserving answers; both are asserted but not independently verified. The auxiliary-information analysis includes a leakage-style condition (3D bbox input) that explains near-perfect scores, which is why it is scored under circularity burden rather than treated as a fully independent prediction.

free parameters (2)
  • image annotation-count filtering threshold = per-dataset; e.g. >2000 annotations discarded in Hypersim
    Sec. 9.1: hand-chosen cutoff to remove noisy images; changes dataset composition and difficulty.
  • numeric answer tolerance = 10% relative error
    Sec. 5.1: predictions for numeric questions are counted correct only if relative error < 0.1; this hand-set threshold directly affects reported accuracies.
axioms (4)
  • domain assumption Omni3D ground-truth 3D bounding boxes are accurate and consistently defined across all constituent datasets.
    Sec. 3.2 says all answers are derived from those boxes; no independent verification of Omni3D annotation quality is provided.
  • domain assumption LLM-based question refinement preserves the original ground-truth answer semantics.
    Sec. 9.4/Fig. 6 instruct the LLM not to change answers, but no automated consistency check on the refined corpus is reported.
  • domain assumption Perspective-aware tasks can be answered from 3D bounding-box geometry alone, with objects approximated as boxes.
    Sec. 8.1 defines occlusion and viewpoint tasks via reference-object-to-anchor-object directions; real object shape, articulation, and truncation are ignored, so some ground-truth visibility labels may be wrong.
  • ad hoc to paper Manual inspection of a 'large, randomly sampled subset' is representative and sufficient to establish high semantic accuracy.
    Sec. 3.3 asserts 'very high rate of semantic accuracy' without reporting sample size, error counts, or inter-annotator agreement.

pith-pipeline@v1.3.0-alltime-deepseek · 21393 in / 12791 out tokens · 131547 ms · 2026-08-03T20:33:39.023931+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of MonoSR: Open-Vocabulary Spatial Reasoning from Monocular Images." pith.science (2026). https://pith.science/paper/MK5GEPM5

@misc{pith2026251119119,
  author       = {Pith},
  title        = {Pith review of: MonoSR: Open-Vocabulary Spatial Reasoning from Monocular Images},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MK5GEPM5}},
  note         = {Machine review of arXiv:2511.19119}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Spatial reasoning (SR), the ability to infer 3D spatial information from 2D inputs, is essential for real-world applications such as embodied AI and autonomous driving. However, existing research primarily focuses on indoor environments and typically relies on multi-view observations, which limits their generalizability to outdoor scenarios and constrains their applicability to monocular images, the most common real-world setting. In this work, we propose MonoSR, a large-scale monocular spatial reasoning dataset that spans diverse scenarios including indoor, outdoor, and object-centric settings, and supports multiple question types. MonoSR provides a path toward open-world monocular spatial reasoning. Beyond introducing the dataset, we evaluate advanced vision-language models to reveal their limitations on this challenging task. We further analyze whether auxiliary information is crucial for monocular spatial reasoning and offer practical guidance for designing future models. These contributions collectively establish a foundation for advancing monocular spatial reasoning in real-world, open-world environments.

Figures

Figures reproduced from arXiv: 2511.19119 by Jingyi He, Qirui Wang, Shijie Li, Si Yong Yeo, Xulei Yang, Yining Pan.

Figure 1
Figure 1. Figure 1: Overview of the proposed MonoSR dataset, which spans three levels of spatial reasoning—Foundational Perception, Perspective [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Demonstration of MonoSR curation pipeline [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Dataset composition of MonoSR across hierarchical rea [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Auxiliary Information, which can inject more context [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Dataset visualization. Examples from indoor, outdoor, and object-centric scenarios covering multiple spatial reasoning tasks with paired questions and ground-truth answers. information is available. 5.3. Impact of Auxiliary Information Tab. 2 presents the performance improvements attained by incorporating different types of auxiliary information across the Indoor, Outdoor, and Object-Centric settings. Deta… view at source ↗
Figure 6
Figure 6. Figure 6: A structured instruction prompt for transforming raw 3D spatial-reasoning QA pairs into professionally rewritten scenario [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Structure of the scene graph used in our dataset. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Example postfix containing the object list and instruc [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Postfix containing the detected objects and their 3D bounding box information used for spatial reasoning. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Template examples for foundational 3D perception, including basic spatial relations and metric queries over 3D bounding boxes. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Template examples for perspective-aware imagination, where questions are conditioned on a reference viewpoint and require [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Template examples for situational reasoning, where spatial queries are explicitly grounded in safety, accessibility, or layout [PITH_FULL_IMAGE:figures/full_fig_p019_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: A subset of the dataset, showing from left to right: the original image, the image with visual prompts, and the image with 3D [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: A subset of the dataset, showing from left to right: the original image, the image with visual prompts, and the image with 3D [PITH_FULL_IMAGE:figures/full_fig_p021_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: A subset of the dataset, showing from left to right: the original image, the image with visual prompts, and the image with 3D [PITH_FULL_IMAGE:figures/full_fig_p022_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 15 linked inside Pith

  1. [2]

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas J. Guibas. ReferIt3D: Neural lis- teners for fine-grained 3d object identification in real-world scenes. InECCV, 2020. 1, 3

    2020

  2. [3]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. InCVPR. 3

  3. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 2, 6

    Pith/arXiv arXiv 2025

  4. [5]

    Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023

    Yuxiao Bai, Lijie Dong, Zihan Wang, Qingyang Zhang, Zhihong Yan, Bin Xie, Xianpei Ma, Haowei Zhang, Shi- long Chen, Ming Bai, et al. Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023. 1, 3

    Pith/arXiv arXiv 2023

  5. [6]

    Omni3d: A large benchmark and model for 3d object detection in the wild

    Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3d: A large benchmark and model for 3d object detection in the wild. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 3

    2023

  6. [7]

    Spatialbot: Precise spatial understanding with vision language models

    Wenxiao Cai, Yaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models. arXiv preprint arXiv:2406.13642, 2024. 3

    Pith/arXiv arXiv 2024

  7. [8]

    Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties. InCVPR, pages 14455–14465, 2024. 3

    2024

  8. [9]

    Perception before reasoning: Two-stage reinforce- ment learning for visual reasoning in vision-language mod- els.arXiv preprint arXiv:2509.13031, 2025

    Yan Chen, Long Li, Teng Xi, Long Zeng, and Jingdong Wang. Perception before reasoning: Two-stage reinforce- ment learning for visual reasoning in vision-language mod- els.arXiv preprint arXiv:2509.13031, 2025. 4

    arXiv 2025

  9. [10]

    Spatial- rgpt: Grounded spatial reasoning in vision-language models

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Rui- han Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatial- rgpt: Grounded spatial reasoning in vision-language models. InNeurIPS, 2024. 3, 4

    2024

  10. [11]

    Collins, Ilia Sucholutsky, Umang Bhatt, Kartik Chandra, Lionel Wong, Mina Lee, Cedegao E

    Katherine M. Collins, Ilia Sucholutsky, Umang Bhatt, Kartik Chandra, Lionel Wong, Mina Lee, Cedegao E. Zhang, Tan Zhi-Xuan, Mark Ho, Vikash Mansinghka, Adrian Weller, Joshua B. Tenenbaum, and Thomas L. Griffiths. Building machines that learn and think with people.Nature Human Behaviour, 8(10):1851–1863, 2024. 2

    2024

  11. [12]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blis- tein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025. 6

    Pith/arXiv arXiv 2025

  12. [13]

    Vlm-3r: Vision-language models aug- mented with instruction-aligned 3d reconstruction, 2025

    Zhiwen Fan, Jian Zhang, Renjie Li, Junge Zhang, Runjin Chen, Hezhen Hu, Kevin Wang, Huaizhi Qu, Dilin Wang, Zhicheng Yan, et al. Vlm-3r: Vision-language models aug- mented with instruction-aligned 3d reconstruction, 2025. URL https://arxiv. org/abs/2505.20279, 3(5):6. 3, 5

    Pith/arXiv arXiv 2025

  13. [14]

    Surds: Benchmarking spatial understand- ing and reasoning in driving scenarios with vision language models

    Xianda Guo, Ruijun Zhang, Yiqun Duan, Yuhang He, Dujun Nie, Wenke Huang, Chenming Zhang, Shuai Liu, Hao Zhao, and Long Chen. Surds: Benchmarking spatial understand- ing and reasoning in driving scenarios with vision language models. InNeurIPS, 2025. 1

    2025

  14. [15]

    3d-llm: Inject- ing the 3d world into large language models.NeurIPS, 2023

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Inject- ing the 3d world into large language models.NeurIPS, 2023. 3

    2023

  15. [16]

    What’s ”up” with vision-language models? investigating their strug- gle with spatial reasoning

    Amita Kamath, Jack Hessel, and Kai-Wei Chang. What’s ”up” with vision-language models? investigating their strug- gle with spatial reasoning. InThe 2023 Conference on Em- pirical Methods in Natural Language Processing, 2023. 1

    2023

  16. [17]

    Lee, Jihyeon Je, Chanho Park, Mikaela Angelina Uy, Leonidas Guibas, and Minhyuk Sung

    Phillip Y . Lee, Jihyeon Je, Chanho Park, Mikaela Angelina Uy, Leonidas Guibas, and Minhyuk Sung. Perspective- aware reasoning in vision-language models via mental im- agery simulation. InICCV, 2025. 2

    2025

  17. [18]

    Seed-bench: Bench- marking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Bench- marking multimodal large language models. InCVPR, pages 13299–13308, 2024. 3

    2024

  18. [19]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 6

    Pith/arXiv arXiv 2024

  19. [20]

    Viewspatial-bench: Evaluating multi-perspective spatial localization in vision-language models, 2025

    Dingming Li, Hongxing Li, Zixuan Wang, Yuchen Yan, Hang Zhang, Siqi Chen, Guiyang Hou, Shengpei Jiang, Wenqi Zhang, Yongliang Shen, Weiming Lu, and Yueting Zhuang. Viewspatial-bench: Evaluating multi-perspective spatial localization in vision-language models, 2025. 1

    2025

  20. [21]

    Hoi, and Li Fei-Fei

    Haotian Li, Dongxu Li, Quan Zhang, Steven C.H. Hoi, and Li Fei-Fei. Llava: Large language and vision assistant.arXiv preprint arXiv:2304.08485, 2023. 1, 3

    Pith/arXiv arXiv 2023

  21. [22]

    Spatialladder: Progressive train- ing for spatial reasoning in vision-language models, 2025

    Hongxing Li, Dingming Li, Zixuan Wang, Yuchen Yan, Hang Wu, Wenqi Zhang, Yongliang Shen, Weiming Lu, Jun Xiao, and Yueting Zhuang. Spatialladder: Progressive train- ing for spatial reasoning in vision-language models, 2025. 4

    2025

  22. [23]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven CH Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. 2022. 1

    2022

  23. [24]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C.H. Hoi. Blip-2: Bootstrapping language–image pre-training with frozen image encoders and large language models. In CVPR, 2023

    2023

  24. [25]

    Xueyan Li, Fan Yang, Dongxu Li, Junnan Li, Yu Qiao, Peng Gao, and Steven C.H. Hoi. Grounded segment anything. arXiv preprint arXiv:2306.02684, 2023. 1

    Pith/arXiv arXiv 2023

  25. [26]

    Visual spa- tial reasoning.Transactions of the Association for Computa- tional Linguistics, 11:635–651, 2023

    Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spa- tial reasoning.Transactions of the Association for Computa- tional Linguistics, 11:635–651, 2023. 3

    2023

  26. [27]

    Grounding dino: Mar- rying dino with grounded pre-training for open-set object 10 detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Mar- rying dino with grounded pre-training for open-set object 10 detection. InProceedings of the European Conference on Computer Vision (ECCV), pages 38–55, 2024. 1

    2024

  27. [28]

    3dsrbench: A compre- hensive 3d spatial reasoning benchmark.arXiv preprint arXiv:2412.07825, 2024

    Wufei Ma, Haoyu Chen, Guofeng Zhang, Celso M de Melo, Alan Yuille, and Jieneng Chen. 3dsrbench: A compre- hensive 3d spatial reasoning benchmark.arXiv preprint arXiv:2412.07825, 2024. 3

    arXiv 2024

  28. [29]

    Sqa3d: Situated question answering in 3d scenes

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. InICLR, 2023. 1, 3

    2023

  29. [30]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. InCVPR, 2019. 1

    2019

  30. [31]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    OpenAI. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. 1, 3

    Pith/arXiv arXiv 2023

  31. [32]

    Shapellm: Universal 3d object understanding for embodied interaction

    Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma. Shapellm: Universal 3d object understanding for embodied interaction. InProceedings of the European Conference on Computer Vi- sion (ECCV), pages 214–238, 2024. 3

    2024

  32. [33]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. 2021. 1

    2021

  33. [34]

    Deepseekmath: Pushing the limits of math- ematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of math- ematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. 8

    Pith/arXiv arXiv 2024

  34. [35]

    Space3D-Bench: Spatial 3D Question Answering Benchmark

    Emilia Szymanska, Mihai Dusmanu, Jan-Willem Buurlage, Mahdi Rad, and Marc Pollefeys. Space3D-Bench: Spatial 3D Question Answering Benchmark. InEuropean Confer- ence on Computer Vision (ECCV) Workshops, 2024. 3

    2024

  35. [36]

    Gemini: A family of highly capable multimodal models

    Gemini Team, Mark Reid, Maarten Bosma, Nan Chen, et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 1

    Pith/arXiv arXiv 2023

  36. [37]

    Nuscenes-spatialqa: A spatial understanding and reasoning benchmark for vision- language models in autonomous driving.arXiv preprint arXiv:2504.03164, 2025

    Kexin Tian, Jingrui Mao, Yunlong Zhang, Jiwan Jiang, Yang Zhou, and Zhengzhong Tu. Nuscenes-spatialqa: A spatial understanding and reasoning benchmark for vision- language models in autonomous driving.arXiv preprint arXiv:2504.03164, 2025. 1

    Pith/arXiv arXiv 2025

  37. [38]

    Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aure- lien Rodriguez, Armand Joulin, Edouard Grave, and Guil- laume Lample. Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971, 2023. 1, 3

    Pith/arXiv arXiv 2023

  38. [39]

    Cross-modal pro- jection in multimodal llms doesn’t really project visual at- tributes to textual space

    Gaurav Verma, Minje Choi, Kartik Sharma, Jamelle Watson- Daniels, Sejoon Oh, and Srijan Kumar. Cross-modal pro- jection in multimodal llms doesn’t really project visual at- tributes to textual space. 1

  39. [40]

    Learning 3d semantic scene graphs from 3d indoor reconstructions

    Johanna Wald, Helisa Dhamo, Nassir Navab, and Federico Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions. InCVPR, 2020. 1

    2020

  40. [41]

    Is a picture worth a thou- sand words? delving into spatial reasoning for vision lan- guage models

    Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Yixuan Li, and Neel Joshi. Is a picture worth a thou- sand words? delving into spatial reasoning for vision lan- guage models. InNeurIPS, 2024. 3

    2024

  41. [42]

    Spatial 3d-llm: Exploring spatial awareness in 3d vision-language models,

    Xiaoyan Wang, Zeju Li, Yifan Xu, Jiaxing Qi, Zhifei Yang, Ruifei Ma, Xiangde Liu, and Chao Zhang. Spatial 3d-llm: Exploring spatial awareness in 3d vision-language models,

  42. [43]

    Spatial-mllm: Boosting mllm capabilities in visual-based spatial intelligence.arXiv preprint arXiv:2505.23747, 2025

    Diankun Wu, Fangfu Liu, Yi-Hsin Hung, and Yueqi Duan. Spatial-mllm: Boosting mllm capabilities in visual-based spatial intelligence.arXiv preprint arXiv:2505.23747, 2025. 3, 5

    Pith/arXiv arXiv 2025

  43. [44]

    Pointllm: Empowering large lan- guage models to understand point clouds

    Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiang- miao Pang, and Dahua Lin. Pointllm: Empowering large lan- guage models to understand point clouds. InECCV, 2024. 3

    2024

  44. [45]

    Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie

    Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How mul- timodal large language models see, remember, and recall spaces. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10632–10643, 2025. 3

    2025

  45. [46]

    Open-vocabulary object detection using cap- tions

    Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih- Fu Chang. Open-vocabulary object detection using cap- tions. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 14388– 14397, 2021. 3

    2021

  46. [47]

    How to enable llm with 3d capacity? a survey of spatial reasoning in llm, 2025

    Jirong Zha, Yuxuan Fan, Xiao Yang, Chen Gao, and Xinlei Chen. How to enable llm with 3d capacity? a survey of spatial reasoning in llm, 2025. 1

    2025

  47. [48]

    From flatland to space: Teaching vision-language models to perceive and reason in 3d.arXiv preprint arXiv:2503.22976, 2025

    Jiahui Zhang, Yurui Chen, Yanpeng Zhou, Yueming Xu, Ze Huang, Jilin Mei, Junhui Chen, Yu-Jie Yuan, Xinyue Cai, Guowei Huang, et al. From flatland to space: Teaching vision-language models to perceive and reason in 3d.arXiv preprint arXiv:2503.22976, 2025. 2

    arXiv 2025

  48. [49]

    Spinbench: Perspective and rotation as a lens on spatial reasoning in vlms, 2025

    Yuyou Zhang, Radu Corcodel, Chiori Hori, Anoop Cherian, and Ding Zhao. Spinbench: Perspective and rotation as a lens on spatial reasoning in vlms, 2025. 4

    2025

  49. [50]

    Video-3d llm: Learning position-aware video representation for 3d scene understanding

    Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d llm: Learning position-aware video representation for 3d scene understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8995–9006, 2024. 1

    2024

  50. [51]

    Scanreason: Empowering 3d visual grounding with reasoning capabilities

    Chenming Zhu, Tai Wang, Wenwei Zhang, Kai Chen, and Xihui Liu. Scanreason: Empowering 3d visual grounding with reasoning capabilities. InEuropean Conference on Computer Vision (ECCV), 2024. 3

    2024

  51. [52]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shen- glong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025. 6

    Pith/arXiv arXiv 2025

  52. [53]

    A detailed description of each level and its cor- responding tasks is provided below

    Task Descriptions Our proposed tasks are organized into three difficulty levels, progressing from low to high complexity: foundational 3D perception, perspective-aware imagination, and situational reasoning. A detailed description of each level and its cor- responding tasks is provided below. 11 8.1. Task Definition Foundational 3D Perception.This level a...

  53. [54]

    Specifically, after obtaining high-quality raw data through filtering, we generate image captions and construct scene graphs to serve as our underlying database

    Detailed Data Annotation Our pipeline leverages handcrafted templates for automated annotation, complemented by a human-in-the-loop verifica- tion stage to eliminate implausible or low-quality QA pairs. Specifically, after obtaining high-quality raw data through filtering, we generate image captions and construct scene graphs to serve as our underlying da...

  54. [55]

    - {variation_instruction} - The rewritten question MUST include: (a) A brief motivation clause describing WHY we need this information, consistent with the motivation hint

    Rewrite the original question into a professional real-world scenario. - {variation_instruction} - The rewritten question MUST include: (a) A brief motivation clause describing WHY we need this information, consistent with the motivation hint. (b) A concrete technical request asking for {metric_phrase} of {target_obj} (include units if relevant). - {objec...

  55. [56]

    - {answer_constraint} - {task_extra} - You MUST NOT flip yesno

    Provide a concise professional answer that is semantically consistent with the original technical answer. - {answer_constraint} - {task_extra} - You MUST NOT flip yesno. - You MUST NOT change numeric truth (e.g. 2.35m stays 2.35m). - Keep this answer short and factual

  56. [57]

    thinking

    Provide a step-by-step reasoning trace in a field called "thinking". - {thinking_instruction} - The reasoning MUST mention that we already have ground-truth 3D bounding boxes. - The reasoning MUST mention {metric_phrase} and {target_obj}. - The reasoning must NOT include the final short answer sentence. - The reasoning MUST have exactly three steps: Step ...

  57. [58]

    Scene In- formation adds global context, 2D Visual Prompts improve local grounding, and 3D Bounding Boxes deliver the largest gains through explicit geometric structure

    Auxiliary Information Prompt As discussed in main paper, auxiliary information consis- tently strengthens monocular spatial reasoning. Scene In- formation adds global context, 2D Visual Prompts improve local grounding, and 3D Bounding Boxes deliver the largest gains through explicit geometric structure. Combining 2D and 3D cues yields the best overall per...

  58. [59]

    For the 3D bounding box information, each object is rep- resented by its center coordinates, spatial dimensions (size), and orientation expressed as a rotation quaternion

    Together, the annotated image and the structured Object List offer a compact yet informative representation of the 2D scene context used by the model. For the 3D bounding box information, each object is rep- resented by its center coordinates, spatial dimensions (size), and orientation expressed as a rotation quaternion. All nu- merical fields are reporte...

  59. [60]

    All inputs are processed with the officialQwen2.5-VLprocessor, which supports dy- namic image resolutions up to 262,144 pixels (512×512)

    Implementation Details Implementation Details We fine-tuneQwen2.5-VL-3B-Instructusing super- vised next-token prediction. All inputs are processed with the officialQwen2.5-VLprocessor, which supports dy- namic image resolutions up to 262,144 pixels (512×512). The native Qwen2.5-VL vision encoder is kept frozen dur- ing training, while the multimodal proje...

  60. [61]

    - Use the format <obj>...</obj> to describe your mapping between textual entities and object IDs

    Identify which object(s) correspond to the entities mentioned in the question, in the same order as they appear. - Use the format <obj>...</obj> to describe your mapping between textual entities and object IDs

  61. [62]

    Reason about their spatial relationship based on their 3D bounding boxes (e.g., relative position, distance, orientation)

  62. [63]

    Your output format should strictly follow: <obj> Object mapping: - entity_1 object {id_a} ({caption_a}) - entity_2 object {id_b} </obj> <reasoning>

    Provide the final answer clearly inside <answer>...</answer> tags. Your output format should strictly follow: <obj> Object mapping: - entity_1 object {id_a} ({caption_a}) - entity_2 object {id_b} </obj> <reasoning> ... </reasoning> <answer> ... </answer> Figure 9. Postfix containing the detected objects and their 3D bounding box information used for spati...