Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SpatialLLM claims a staged 3D-data recipe lifts image-only spatial reasoning to 62.7%, beating GPT-4o by 8.7 points.

desk verdict A useful training recipe and a new orientation-heavy benchmark, but the benchmark's construct validity is not established, so the headline SOTA claim overreaches. read the letter →

arxiv 2505.00788 v3 pith:N3OZ5MRL submitted 2025-05-01 cs.CV

Wufei Ma , Luoxin Ye , Celso M de Melo , Jieneng Chen , Alan Yuille This is my paper · ORCID
classification cs.CV
keywords largemultimodalmodels3Dspatialreasoningvisualquestionansweringobjectorientation3D-informeddatainstructiontuningalignmentVQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the missing ingredient in large multimodal models' 3D spatial reasoning is not larger models or specialized 3D input modules, but a deliberate staging of 3D-informed text data across ordinary training stages. To show this it introduces SpatialLLM, trained from images alone with two new data types: object-level 3D probing questions (depth, distance, azimuth/elevation) and one million multi-turn conversations about 3D spatial relationships. On the paper's new SpatialVQA benchmark, the recipe scores 62.7% average accuracy, 8.7 points above GPT-4o and 10.5 points above the spatial-centric SpatialVLM. Controlled ablations attribute 13.7 points of the gain to data placement and only 1.3 points to architecture, with human-annotated orientation data at the alignment stage working better than pseudo-labels. If correct, this gives the community a reproducible roadmap for adding 3D awareness to LMMs without extra depth maps, masks, or boxes at inference.

What carries the argument

The load-bearing mechanism is the compound training recipe, not any single network module. Concretely: (1) 3DI-Pb-IN166K, a set of object-level 3D probing questions converted from human-annotated ImageNet3D poses, is inserted at the multimodal alignment stage; (2) 3DI-Ft1M, one million instruction-tuning conversations about distances, facing directions, and relative 3D positions, is inserted at instruction tuning; (3) the architecture is a LLaVA-style model with a CLIP+DINOv2 mixed vision encoder and Llama3-8B, with the encoder kept frozen; and (4) SpatialVQA, 1,323 rule-generated questions built from Omni3D 3D bounding boxes, supplies the yardstick for distance, orientation, and combined spatial reasoning. The design-space search shows that moving either dataset to the wrong stage, or replacing human pose annotations with pseudo-labels, lowers accuracy.

What would settle it

A pure 2D heuristic that answers SpatialVQA's distance and side questions from bounding-box area, pixel position, and occlusion would settle the claim: if it reaches or exceeds 62.7%, the benchmark does not isolate 3D reasoning.

Watch

Extended reading notes

Core claim

The paper's central claim is that 3D spatial reasoning in an LMM is primarily a data-placement problem. Starting from LLaVA-v1.5 at 47.7% on SpatialVQA, the paper moves through a roadmap: mixing a self-supervised DINOv2 encoder's features with CLIP and upgrading the LLM to Llama3-8B adds about 1.3 points; adding 3D-informed instruction conversations (3DI-Ft1M) at the tuning stage adds 10.7 points; and adding human-annotated 3D orientation probing data (3DI-Pb-IN166K) at multimodal alignment adds another 3 points, reaching 62.7%. The authors report that this is the first VQA data to include 3D orientation relationships on real images, and they show that pre-training the vision encoder on 3D probing data (stage 0) does not help, so the final design keeps visual features frozen.

Load-bearing premise

The headline numbers rest on SpatialVQA's rule-generated answers being correct and genuinely 3D; if those answers are noisy or reachable from 2D cues, the gap to GPT-4o measures something other than 3D reasoning.

Editorial extensions

If this is right

  • Adding 3D-informed instruction tuning data alone yields a 10.7-point jump on SpatialVQA, making it the single strongest design decision.
  • Adding human-annotated 3D pose probing data during multimodal alignment adds 3 more points and outperforms semi-automated pseudo-labels.
  • Architecture upgrades contribute about 1.3 points total, so future gains are more likely to come from data composition than from new encoders or LLMs.
  • Fine-tuning the vision encoder on 3D data before alignment is counterproductive; frozen pretrained encoders remain the better design.
  • The final image-only model reaches 62.7% average accuracy and 86.3% on 3D distance questions, while orientation reasoning lags at 52.9%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the 86.3% distance accuracy versus 52.9% orientation accuracy suggests the field's next bottleneck is orientation perception; datasets with higher-quality 6D pose labels, not more conversations, are likely to drive the next jump.
  • Editorial inference: since stage-0 vision encoder tuning hurt performance, injecting 3D awareness through adapters or late fusion may be safer than fine-tuning the encoder, and the paper's negative result points there.
  • Editorial inference: if the 13.7-point dominance of data over architecture reproduces on larger backbones, training-stage data placement should become a standard reporting axis for multimodal model papers, not just spatial ones.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SpatialLLM, a large multimodal model designed for 3D spatial reasoning, and introduces SpatialVQA, a 1,323-question benchmark generated from Omni3D 3D bounding boxes. The method combines a compound design: 3D-informed probing data for alignment, 3D-informed instruction-tuning data, a mixed vision encoder (CLIP+DINOv2), and an upgraded LLM (Llama3-8B). The paper reports 62.7% accuracy on SpatialVQA, surpassing GPT-4o by 8.7% and SpatialVLM by 10.5%, and attributes the gains to architecture improvements (1.3%) and 3D-informed data (13.7%). The systematic ablation over training stages and data types is the paper's main methodological contribution.

Significance. The paper's strengths are its systematic coverage of the design space (data type, training stage, vision encoder, LLM) and the construction of large 3D-informed training datasets (3DI-Pb-OI1M, 3DI-Pb-IN166K, 3DI-Ft1M). The idea of injecting 3D orientation probing data at the alignment stage and the 13.7% gain from data-side changes are, if confirmed, nontrivial and useful for practitioners. The new SpatialVQA benchmark, however, is the sole basis for all headline comparisons; because its construct validity is not demonstrated and the evaluation is on the same family of rule-generated data used for training, the central claim of state-of-the-art 3D spatial reasoning is not yet established. With external validation and better statistical controls, the contribution would be a valuable reference for the community.

major comments (4)
  1. [Sec. 3.2.2 / Appendix B] The central assertion that all SpatialVQA questions 'cannot be answered from 2D spatial reasoning only' is not supported by the question inventory. 'Closer to camera' is largely determined by apparent size and occlusion; 'Facing camera' reduces to which side is visible in the 2D image; 'Higher' and 'On which side' are mostly readable from image-plane coordinates; 'Same direction' can be approximated from projected 2D orientation. Only 'Closer to object' and possibly 'Facing object' seem to require true 3D relationships. The paper provides no 2D-cue oracle baseline, so the construct validity of the benchmark is unestablished. I request a control evaluation: implement a rule-based oracle using only 2D signals (2D box sizes, image-plane centers, occlusion/ordering, projected orientation) and report its accuracy per question type, along with the fraction of questions that are ambiguous or degenerate under 2D cues. This is necessary to interpret the headline 62.7% and the 8.7% margin over GPT-4o in Table 1.
  2. [Sec. 3.3.2 / Table 2 / Appendix A] The training data (3DI-Pb, 3DI-Ft1M) and the evaluation benchmark (SpatialVQA) are generated by the same group using the same rule-based conversion of 3D bounding box annotations (Omni3D for the benchmark; OpenImages/ImageNet3D for training) and the same question templates. The +10.7% from 3DI-Ft1M instruction tuning (Fig. 6) may therefore reflect learning benchmark-specific templates and label distributions rather than general 3D spatial reasoning. The manuscript does not report whether SpatialVQA images overlap with training images, nor does it evaluate on any independent benchmark. Please add (i) a check for image/annotation overlap between SpatialVQA and all training sets; (ii) evaluation on at least one external spatial reasoning benchmark or a human-annotated subset; and (iii) a discussion of how much of the gain persists when the evaluation template is varied.
  3. [Table 2 / Fig. 6] All conclusions are based on single-run accuracies on a 1,323-question set. For the overall accuracy, the approximate 95% binomial confidence interval is ±2.7 percentage points; for the per-type subsets (roughly 240 questions each) it is about ±6.3 points. Thus the reported differences of 0.3% (CLIP+DINOv2 vs CLIP+MAE) and 1.0% (LLaVA-v1.5 to Llama3) are within sampling noise, yet the roadmap in Fig. 6 treats them as design recommendations. Report multiple seeds or bootstrap confidence intervals, and correct for the multiple comparisons inherent in selecting a model on the same benchmark that produced the headline number.
  4. [Fig. 7 / Table 1] The comparison with GPT-4o is not quantified transparently. In Fig. 7, GPT-4o's response is a refusal ('without the ability to visually assess the image'), which is a limitation of the interaction protocol, not necessarily a spatial reasoning error. If refusals are scored as incorrect, the 8.7% margin over GPT-4o is inflated. The paper must report the exact prompt, the number of calls, retries, refusal detection, and the accuracy after excluding refusals or providing the image again. The same protocol should be documented for Claude 3.5 Sonnet.
minor comments (6)
  1. [Sec. 1] Typo: 'for for complex spatial relationships' should read 'for complex spatial relationships'.
  2. [Fig. 6] '3DI-Pb-1N166K' appears to be a typo for '3DI-Pb-IN166K'.
  3. [Fig. 5(d)] 'azimuth elevator' should be 'azimuth elevation'.
  4. [Related Work] SpatialRGPT is cited as [14,16]; reference [14] is SpatialVLM, which is a different method. Please correct the citation.
  5. [Appendix B] The angle threshold used to define 'same direction' and 'on which side' is not specified; please state the exact rule for completeness and reproducibility.
  6. [Supplementary Sec. A] The statement that 'We pretrain the visual connector only while freezing other modules' is inconsistent with the Stage 0 rows of Table 2, which update LoRA layers of the CLIP encoder; please clarify which modules are updated at each stage.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains are empirical ablations evaluated on a benchmark built from independent Omni3D annotations, not quantities forced by construction.

full rationale

The paper's central claim is an empirical result on SpatialVQA, a benchmark constructed from Omni3D's object-level 3D bounding boxes with rule-based question generation (Sec. 3.2.2, Appendix B). The model is trained on separate data, 3DI-Pb-OI1M, 3DI-Pb-IN166K, and 3DI-Ft1M, drawn from OpenImages and ImageNet3D; SpatialVQA images are not used in training, and the benchmark labels come from Omni3D annotations rather than from the authors' ImageNet3D pose estimator. The reported +10.7% and +3% gains are measured ablations, not fitted parameters renamed as predictions, and the final 62.7% is an evaluation result, not a quantity derived from the training labels by construction. The use of the authors' prior ImageNet3D work as a data and pose-estimation source is a normal tool citation; the paper does not invoke it as a uniqueness theorem, nor does it smuggle in an ansatz through that citation. The concern that some SpatialVQA question types may be answerable from 2D cues is a benchmark-validity and correctness issue, not a circular derivation: even if true, it would not make the reported accuracy equal to the training input by construction. No self-definitional, fitted-input, self-citation-chain, uniqueness-import, ansatz-smuggling, or renaming step can be exhibited from the paper's equations or data-generation rules.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central comparison rests on four load-bearing assumptions about the pseudo-annotation pipeline, the validity of the self-created benchmark, the absence of train/test overlap, and the transferability of the LLaVA recipe. The one explicit hand-chosen parameter is the angle threshold used to discretize orientations, which is not reported.

free parameters (1)
  • Angle threshold for 'same direction' and 'on which side' classification
    The rule-based QA generation must convert continuous relative orientations into discrete answers (same vs different direction, front/back/left/right). The threshold angle is not reported in the paper, yet it determines both the 3DI-Ft1M training labels and the SpatialVQA ground-truth answers, so it directly controls the central accuracy numbers.
assumptions (5)
  • domain assumption Metric depth and 3D pose estimates produced by foundation models (DepthAnything, the ImageNet3D pose estimator) are accurate enough to serve as training supervision.
    The 3D-informed probing and conversation data (3DI-Pb-OI1M, 3DI-Ft1M) are generated semi-automatically with these tools. If depth scale or pose estimates are systematically biased, the model learns incorrect 3D relationships. Invoked in Sec. 3.3.1 and Appendix A.
  • domain assumption Omni3D's 3D bounding box annotations are correct and can be converted into reliable orientation and spatial relationship labels for SpatialVQA.
    The benchmark answers are derived from these boxes by rule-based code described in Sec. 3.2.2 and Appendix B. No manual verification or noise analysis of the converted labels is provided.
  • domain assumption SpatialVQA questions cannot be answered from 2D image cues alone.
    The paper asserts this in Sec. 3.2.2, but provides no 2D-only baseline or control. Questions such as 'closer to camera' can be answered from apparent size and occlusion, so this assumption is load-bearing for the claim that the benchmark measures 3D reasoning.
  • domain assumption Training images (OpenImages, ImageNet3D) are disjoint from evaluation images (Omni3D).
    No deduplication or overlap check is reported. If Omni3D images or near-duplicates appear in the large OpenImages or ImageNet3D pools, the reported results would be inflated. This is relevant to Sec. 3.3 and Appendix A.
  • domain assumption LLaVA-v1.5 hyperparameters and training recipe transfer to the new 3D-informed data without re-tuning.
    The paper states 'all hyperparameters remain unchanged unless explicitly stated otherwise' in Sec. 4.1, which assumes the standard recipe applies to the mixed data and the Llama3 backbone.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models." pith.science (2026). https://pith.science/paper/N3OZ5MRL

@misc{pith2026250500788,
  author       = {Pith},
  title        = {Pith review of: SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N3OZ5MRL}},
  note         = {Machine review of arXiv:2505.00788}
}
read the original abstract

Humans naturally understand 3D spatial relationships, enabling complex reasoning like predicting collisions of vehicles from different directions. Current large multimodal models (LMMs), however, lack of this capability of 3D spatial reasoning. This limitation stems from the scarcity of 3D training data and the bias in current model designs toward 2D data. In this paper, we systematically study the impact of 3D-informed data, architecture, and training setups, introducing SpatialLLM, a large multi-modal model with advanced 3D spatial reasoning abilities. To address data limitations, we develop two types of 3D-informed training datasets: (1) 3D-informed probing data focused on object's 3D location and orientation, and (2) 3D-informed conversation data for complex spatial relationships. Notably, we are the first to curate VQA data that incorporate 3D orientation relationships on real images. Furthermore, we systematically integrate these two types of training data with the architectural and training designs of LMMs, providing a roadmap for optimal design aimed at achieving superior 3D reasoning capabilities. Our SpatialLLM advances machines toward highly capable 3D-informed reasoning, surpassing GPT-4o performance by 8.7%. Our systematic empirical design and the resulting findings offer valuable insights for future research in this direction. Our project page is available at: https://3d-spatial-reasoning.github.io/spatial-llm/

Figures

Figures reproduced from arXiv: 2505.00788 by the authors.

Figure 1
Figure 1. 3D spatial reasoning is crucial for LMMs to ground ob [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Examples from our SpatialVQA benchmark featuring a broad range of questions that require 3D spatial reasoning. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Design space for LMMs capable of spatial reason￾ing. The dashed boxes and lines highlight our new design space compared to LLaVA-v1.5. This compound design simultaneously considers 3D-informed data, architecture, and training methods to search for the best-performing models for spatial reasoning. image 2D bbox image caption CC558K detailed image descriptions image Metric Depth Est (e.g., DepthAnything) Camera Calibr… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Comparison of data cards showcasing the curation pro [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Design instantiation and comparison. (a) Architecture and Training of our proposed design. We investigate the 3D-awareness of mixed visual encoders, and incorporate 3D-informed data at each training stage across all architecture components. (b) LLaVA comparison, where …
Figure 6
Figure 6. Figure 6: We modernize a standard LLaVA-v1.5 towards the de [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Our model answers question correctly that needs ac [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Qualitative examples of our 3DI-Ft1M data. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RL

    cs.CV 2025-12 conditional novelty 6.0 of 10

    A two-phase interactive RL framework (DIRL) lets a 3B VLM learn to coordinate multiple vision and robot tools, reaching top benchmark scores and 86% real-robot pick-and-place success.

Reference graph

Works this paper leans on

70 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  2. [2]

    SpaceLLaV A.https://huggingface

    Remyx AI. SpaceLLaV A.https://huggingface. co/remyxai/SpaceLLaVA, 2024. 1, 7

    work page 2024

  3. [3]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736,

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736,

  4. [4]

    Claude 3.5 Sonnet.https : / / www

    Anthropic. Claude 3.5 Sonnet.https : / / www . anthropic . com / news / claude - 3 - 5 - sonnet,

    work page

  5. [5]

    Apollo syntheic dataset, 2019

    Apollo Team. Apollo syntheic dataset, 2019. 1

    work page 2019

  6. [6]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. Inproceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19129– 19139, 2022. 4, 1

    work page 2022

  7. [7]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023. 2, 3

    arXiv 2023

  8. [8]

    Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023. 2

    arXiv 2023

Show all 70 references
  1. [9]

    Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data.arXiv preprint arXiv:2111.08897, 2021

    Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, et al. Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data.arXiv preprint arXiv:2111.08897,...

    2021 arXiv

  2. [10]

    Zoedepth: Zero-shot trans- fer by combining relative and metric depth.arXiv preprint arXiv:2302.12288, 2023

    Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias M ¨uller. Zoedepth: Zero-shot trans- fer by combining relative and metric depth.arXiv preprint arXiv:2302.12288, 2023. 1, 2

    2023 arXiv

  3. [11]

    Omni3D: A large benchmark and model for 3D object detection in the wild

    Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3D: A large benchmark and model for 3D object detection in the wild. InCVPR, Vancouver, Canada, 2023. IEEE. 2, 4, 1

    2023

  4. [12]

    nuscenes: A multi- modal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom. nuscenes: A multi- modal dataset for autonomous driving. InProceedings of the IEEE/CVF conference on computer vision and patter...

    2020

  5. [13]

    Honeybee: Locality-enhanced projector for multimodal llm

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal llm. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 13817–13827, 2024. 2, 3

    2024

  6. [14]

    Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 14455–14465,

  7. [15]

    Vitamin: Designing scalable vision models in the vision-language era

    Jieneng Chen, Qihang Yu, Xiaohui Shen, Alan Yuille, and Liang-Chieh Chen. Vitamin: Designing scalable vision models in the vision-language era. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 2

    2024

  8. [16]

    Spatial- rgpt: Grounded spatial reasoning in vision language model

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Rui- han Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatial- rgpt: Grounded spatial reasoning in vision language model. arXiv preprint arXiv:2406.01584, 2024. 1, 2, 3, 4, 5

    2024 arXiv

  9. [17]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 2, 3

    2009

  10. [18]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  11. [19]

    Prob- ing the 3d awareness of visual foundation models

    Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Ab- hishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Prob- ing the 3d awareness of visual foundation models. InPro- ceedings of the IEEE/CVF Conference on Computer V...

    2024

  12. [20]

    Dat- acomp: In search of the next generation of multimodal datasets.arXiv preprint arXiv:2304.14108, 2023

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Dat- acomp: In search of the next generation of multimodal datasets.arXiv preprint arXiv:2304.14108, 2023. 3

    2023 arXiv

  13. [21]

    Are we ready for autonomous driving? the kitti vision benchmark suite

    Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In2012 IEEE conference on computer vision and pat- tern recognition, pages 3354–3361. IEEE, 2012. 4, 1

    2012

  14. [22]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017. 3

    2017

  15. [23]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000– 16009, 2022. 3, 5, 8

    2022

  16. [24]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021. 8

    2021 arXiv

  17. [25]

    Open- clip, 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Han- naneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Open- clip, 2021. If you use this software, please cite it as be...

    2021

  18. [26]

    Novum: Neural object volumes for robust object classification

    Artur Jesslen, Guofeng Zhang, Angtian Wang, Wufei Ma, Alan Yuille, and Adam Kortylewski. Novum: Neural object volumes for robust object classification. InEuropean Con- ference on Computer Vision, pages 264–281. Springer, 2024. 2

    2024

  19. [27]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. InICML, 2021. 2

    2021

  20. [28]

    Mistral 7b.arXiv preprint arXiv:2310.06825, 2023

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lam- ple, Lucile Saulnier, et al. Mistral 7b.arXiv preprint arXiv:2310.06825, 2023. 2

    arXiv

  21. [29]

    Perspective fields for single image cam- era calibration

    Linyi Jin, Jianming Zhang, Yannick Hold-Geoffroy, Oliver Wang, Kevin Blackburn-Matzen, Matthew Sticha, and David F Fouhey. Perspective fields for single image cam- era calibration. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17307...

    2023

  22. [30]

    Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick. Segment anything.arXiv:2304.02643, 2023. 1

    2023 arXiv

  23. [31]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. InProceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 2023. 1, 2

    2023

  24. [32]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer vision, 123:32–73, 2017

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer...

    2017

  25. [33]

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection...

    1956

  26. [34]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 2, 8

    2024 arXiv

  27. [35]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InICML,

  28. [36]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InIn- ternational conference on machine learning, pages 19730– 19742. PMLR, 2023. 8

    2023

  29. [37]

    What if we recaption billions of web images with llama-3?arXiv preprint arXiv:2406.08478,

    Xianhang Li, Haoqin Tu, Mude Hui, Zeyu Wang, Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjie Zheng, et al. What if we recaption billions of web images with llama-3?arXiv preprint arXiv:2406.08478,

    arXiv

  30. [38]

    Learning customized visual models with retrieval-augmented knowledge

    Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jae Lee, and Chunyuan Li. Learning customized visual models with retrieval-augmented knowledge. InProceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 15148–15158, 2023. 8

    2023

  31. [39]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 1, 2, 3, 7, 8

    2024

  32. [40]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 7

    2024

  33. [41]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024. 1, 2, 3, 4, 5, 8

    2024

  34. [42]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499, 2023

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499, 2023. 1

    2023 arXiv

  35. [43]

    Deepseek-vl: towards real-world vision- language understanding.arXiv preprint arXiv:2403.05525,

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision- language understanding.arXiv preprint arXiv:2403.05525,

    arXiv

  36. [44]

    Robust category-level 6d pose estimation with coarse-to-fine rendering of neural features

    Wufei Ma, Angtian Wang, Alan Yuille, and Adam Ko- rtylewski. Robust category-level 6d pose estimation with coarse-to-fine rendering of neural features. InEuropean Conference on Computer Vision, pages 492–508. Springer,

  37. [45]

    Imagenet3d: Towards general-purpose object-level 3d understanding

    Wufei Ma, Guanning Zeng, Guofeng Zhang, Qihao Liu, Letian Zhang, Adam Kortylewski, Yaoyao Liu, and Alan Yuille. Imagenet3d: Towards general-purpose object-level 3d understanding. InNeurIPS, 2024. 2, 4, 5, 7, 1

    2024

  38. [46]

    Mm1: Methods, analysis & insights from multimodal llm pre-training.arXiv preprint arXiv:2403.09611, 2024

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xi- anzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis & insights from multimodal llm pre-training.arXiv preprint arXiv:2403.09611, 2024. 3

    2024 arXiv

  39. [47]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. TMLR, 2024. 2, 3, 5, 6, 8

    2024

  40. [48]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. InICML, 2021. 2, 3, 5, 6, 8

    2021

  41. [49]

    Gsr-bench: A benchmark for grounded spatial reasoning evaluation via multimodal llms.arXiv preprint arXiv:2406.13246, 2024

    Navid Rajabi and Jana Kosecka. Gsr-bench: A benchmark for grounded spatial reasoning evaluation via multimodal llms.arXiv preprint arXiv:2406.13246, 2024. 2

    2024 arXiv

  42. [50]

    Susskind

    Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. Hypersim: A photorealistic syn- thetic dataset for holistic indoor scene understanding. In International Conference on Computer Vision (ICCV) 2021,

    2021

  43. [51]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 3

    2022

  44. [52]

    Laion-5b: An open large-scale dataset for training next generation image-text models.Advances in Neural In- formation Processing Systems, 35:25278–25294, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models.Advances in Neural In- fo...

    2022

  45. [53]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. InACL,

  46. [54]

    Sun rgb-d: A rgb-d scene understanding benchmark suite

    Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 567–576, 2015. 4, 1

    2015

  47. [55]

    Core knowl- edge.Developmental science, 10(1):89–96, 2007

    Elizabeth S Spelke and Katherine D Kinzler. Core knowl- edge.Developmental science, 10(1):89–96, 2007. 1

    2007

  48. [56]

    Revisiting unreasonable effectiveness of data in deep learning era

    Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhi- nav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. InProceedings of the IEEE international conference on computer vision, pages 843–852, 2017. 3

    2017

  49. [57]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023. 1, 2

    2023 arXiv

  50. [58]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal llms.arXiv preprint arXiv:2406.16860, 2024

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms.arXiv preprint arXiv:2406.16860, 2024. 1, 3, 4, 7

    2024 arXiv

  51. [59]

    Llama 2: Open foundation and fine- tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fer- nandes, Jere...

    2023 arXiv

  52. [60]

    3d-aware visual question answering about parts, poses and occlusions.Advances in Neural Information Processing Systems, 36, 2024

    Xingrui Wang, Wufei Ma, Zhuowan Li, Adam Kortylewski, and Alan L Yuille. 3d-aware visual question answering about parts, poses and occlusions.Advances in Neural Information Processing Systems, 36, 2024. 2, 3

    2024

  53. [61]

    Compositional 4d dynamic scenes understanding with physics priors for video question answering.arXiv preprint arXiv:2406.00622, 2024

    Xingrui Wang, Wufei Ma, Angtian Wang, Shuo Chen, Adam Kortylewski, and Alan Yuille. Compositional 4d dynamic scenes understanding with physics priors for video question answering.arXiv preprint arXiv:2406.00622, 2024. 2

    2024 arXiv

  54. [62]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. InCVPR, 2024. 1

    2024

  55. [63]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10371–10381, 2024. 1, 2

    2024

  56. [64]

    3d question answering.arXiv preprint arXiv:2112.08359,

    Shuquan Ye, Dongdong Chen, Songfang Han, and Jing Liao. 3d question answering.arXiv preprint arXiv:2112.08359,

    arXiv

  57. [65]

    Coca: Contrastive captioners are image-text foundation models.arXiv preprint arXiv:2205.01917, 2022

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models.arXiv preprint arXiv:2205.01917, 2022. 3

    2022 arXiv

  58. [66]

    Recognize anything: A strong image tagging model.arXiv preprint arXiv:2306.03514,

    Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging model.arXiv preprint arXiv:2306.03514,

    arXiv

  59. [67]

    Recognize anything: A strong image tagging model

    Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging model. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 172...

    2024

  60. [68]

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023. 6, 8

    2023

  61. [69]

    ibot: Image bert pre-training with online tokenizer.arXiv preprint arXiv:2111.07832,

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer.arXiv preprint arXiv:2111.07832,

    arXiv

  62. [70]

    yes” as the answer and 120 questions have “no

    Shengjie Zhu, Abhinav Kumar, Masa Hu, and Xiaoming Liu. Tame a wild camera: In-the-wild monocular camera calibra- tion. InNeurIPS, 2023. 1 SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models Supplementary Material A. 3D-Informed Data...

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.