Pith. sign in

REVIEW 3 major objections 4 minor 52 references

VLM-IE3D shows that a vision-language model can gain 3D spatial awareness from RGB video alone by fusing implicit scene-level tokens with explicit tokens of reconstructed depth, raising 3D video detection F1 from 30.9 to 42.8.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:59 UTC pith:JSFQXEAX

load-bearing objection A mostly solid RGB-only 3D-VLM recipe whose headline complementarity claim outruns the evidence: the EGT gain is real but the mechanism is underdetermined. the 3 major comments →

arxiv 2607.21595 v1 pith:JSFQXEAX submitted 2026-07-23 cs.CV cs.AIcs.LG

3D-Aware VLMs with Implicit and Explicit Geometries

Wenhao Li , Xueying Jiang , Quanhao Qian , Deli Zhao , Ran Xu , Shijian Lu , Gongjie Zhang This is my paper
classification cs.CV cs.AIcs.LG
keywords vision-language modelsimplicit geometry tokensexplicit geometry tokens3D scene understanding3D video detection3D visual groundingspatial reasoningRGB-only 3D perception
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that a vision-language model's 3D understanding is limited not by lack of geometric information in RGB video but by how that information is represented. It proposes adding two complementary token streams to a standard VLM: implicit geometry tokens that capture coarse global scene layout, and explicit geometry tokens that encode reconstructed depth, point maps, or Gaussian splats as quantitative spatial measurements. A lightweight adapter fuses both with 2D visual tokens. On 3D video detection the combined model reaches an F1 of 42.8, up from 30.9 for the RGB-only baseline, 34.7 with explicit tokens alone, and 40.5 with implicit tokens alone; similar gains appear on 3D grounding, dense captioning, and spatial reasoning. The claim is that implicit and explicit geometries are complementary, and that explicit reconstructed attributes — not ground-truth 3D input — are enough to supply the fine-grained structure that VLMs lack.

Core claim

The paper's central discovery is that the two kinds of geometric representation play distinct, complementary roles in a vision-language model. Implicit Geometry Tokens, taken from the internal features of a 3D reconstruction encoder, give the model a coarse 'cognitive map' of the scene — global layout and object relations — but their compressed form hides quantitative measurements. Explicit Geometry Tokens, obtained by tokenizing the encoder's reconstructed depth maps, point maps, or 3D Gaussians, supply a 'reconstruction map' with fine-grained, numerically interpretable spatial detail. Fused with 2D visual tokens through a 3D-aware adapter whose core is an implicit–explicit cross-attention

What carries the argument

The central mechanism is the pair of token streams and the 3D-aware adapter. IGTs are the latent features of a frozen 3D reconstruction encoder — high-level scene priors describing layout and object relations. EGTs are patch-embedded tokens of the encoder's reconstructed depth, point maps, or 3D Gaussians — fine-grained quantitative geometry. The adapter runs multi-head cross-attention with IGTs as queries and EGTs as keys and values, adds the result to compressed 2D visual tokens via element-wise addition, and feeds the fused tokens into the VLM. The key operation is the complementarity: the adapter lets the VLM consult fine-grained measurements while retaining global context, and the paper

Load-bearing premise

The method assumes the frozen encoder's reconstructed depth, point maps, and Gaussians are geometrically faithful enough to serve as 'explicit' quantitative truth; if those reconstructions carry scale ambiguity or systematic bias, the VLM is learning to trust a moving target and the measured gains may not transfer to new scenes or sensors.

What would settle it

Run VLM-IE3D on scenes with ground-truth depth (e.g., LiDAR scans or synthetic rooms), split results by reconstruction error of the frozen encoder, and check whether the EGT contribution (+2.3 F1 in detection) concentrates in low-error scenes. If high-error scenes show no EGT benefit or a negative effect, the claim that EGTs provide faithful explicit geometry is refuted; if the benefit persists regardless of depth quality, the tokens are behaving more like generic learned features than like explicit measurements.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A VLM trained only on RGB video can reach performance on 3D dense captioning, grounding, and detection that is competitive with models that consume point clouds or depth.
  • Implicit and explicit geometry streams are complementary: jointly adding them yields more than either alone (42.8 vs 40.5 vs 34.7 F1).
  • The explicit stream delivers its benefit across multiple attribute types — depth, point maps, and 3D Gaussians — so the framework does not depend on one particular reconstruction format.
  • The approach transfers across different 3D geometry encoders, and even across separate encoders for the implicit and explicit streams, indicating it is robust to the choice of reconstruction backbone.
  • The added cost is small — roughly 0.1B parameters and 1 FPS — so the gains are available without a major compute penalty.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the gains are driven by giving the VLM literal numeric coordinates, then improving the encoder's depth accuracy — or substituting real sensor depth when available — should improve downstream tasks without retraining; this is a direct testable prediction.
  • One implication left implicit is that the same method could adapt to a new sensor by swapping the explicit attribute source while keeping the VLM and adapter fixed.
  • The fusion is simple addition after cross-attention, which suggests the VLM does most spatial reasoning once geometry is in token form; a future version might compress or distill the geometry streams into the 2D encoder, or learn to suppress unreliable reconstructed regions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes VLM-IE3D, a VLM enhancement that injects 3D geometry from RGB video by adding two token streams to a frozen 2D VLM: Implicit Geometry Tokens (IGTs) from the fusion-decoder features of a frozen AnySplat encoder, and Explicit Geometry Tokens (EGTs) obtained by embedding reconstructed 3D attributes (depth, point maps, Gaussians) from the same encoder. A 3D-aware adapter with an implicit-explicit attention module fuses these with 2D visual tokens before the LLM. Experiments are reported on 3D video detection (EmbodiedScan), 3D visual grounding (ScanRefer), 3D dense captioning (Scan2Cap), and spatial reasoning (VSI-Bench). The central empirical claim is that IGTs and EGTs are complementary: on 3D video detection, adding both improves F125 from 30.9 (RGB baseline) to 42.8, which is +8.1 over EGTs alone and +2.3 over IGTs alone (Sec. 4.3, Table 5). The paper also reports consistent gains over the implicit-only baseline VG LLM on the other tasks.

Significance. If the complementary effect is attributable to explicit geometric content, the contribution is significant and practical: it shows a lightweight way to improve VLM 3D spatial understanding without 3D sensors, and the code/model release supports reproducibility. The ablation matrix is a strength. However, the main mechanism claim is not yet isolated from confounds (extra trainable stream, statistical noise), and the VSI-Bench gains are mixed.

major comments (3)
  1. [Sec. 4.3, Table 5] The +2.3 F125 gain of IGTs+EGTs over IGTs alone is the pivotal evidence for complementarity, but the design does not control for the possibility that any additional trainable token stream of the same shape, or just extra adapter capacity, produces the gain. Because EGTs are derived from the same frozen AnySplat encoder that generates IGTs (Sec. 3.1), and because Table 7 shows nearly identical results for depth, point maps, and Gaussians (F125 42.5–42.8) while Table 9 shows replacing AnySplat depth with DepthAnything V2 changes F125 by only 0.3, the results are consistent with the EGT branch acting as a generic second stream rather than as faithful explicit geometry. A control with non-geometric input (e.g., shuffled or corrupted depth, or a random tensor) and, if possible, a ground-truth-depth control is needed to demonstrate that geometric fidelity drives the improvement.
  2. [Sec. 4.2, Sec. 4.3] No error bars, multiple seeds, or evaluation sampling seed are reported. The 3D video detection evaluation randomly selects 10 sequences per scene without a stated seed (Sec. 4.2), which introduces nontrivial sampling noise into the central +2.3 F125 gap. The authors should report the evaluation seed or, preferably, mean±std over at least 3 training seeds and multiple evaluation samples. This is essential to assess whether the complementarity gain is robust.
  3. [Sec. 4.2, Table 4] The spatial reasoning results are presented as consistent gains, but the breakdown in Table 4 shows a mixed pattern: VLM-IE3D improves on Obj. Count (+1.5), Abs. Dist. (+0.7), and Rel. Dist. (+3.1) over VG LLM, yet loses on Room Size (−0.2), Rel. Dir. (−0.5), and Appr. Order (−4.5), with an average gain of only +0.3. Given the lack of confidence intervals, the claim that explicit representations effectively capture fine-grained information across spatial reasoning tasks is not fully supported. Please discuss these per-task trade-offs and, if possible, analyze which sub-skills benefit from EGTs.
minor comments (4)
  1. [Sec. 4.1] The statement that depth maps 'can be more readily obtained from RGB-D sensors' is at odds with the RGB-only motivation; clarify that this refers to the default attribute choice for the embedding, not to input requirements.
  2. [Table 4] The column headers are misaligned in the rendered table, making it hard to map sub-task names to scores; reformat.
  3. [Sec. 3.1] The text says 'The point maps are derived by back-projecting reconstructed depth maps' but does not specify the intrinsic matrix or whether it is from the encoder's estimated pose; add a sentence.
  4. [Sec. 5] The conclusion says 'superior performance' but Table 4 shows some VSI-Bench sub-tasks below VG LLM; qualify the claim.

Circularity Check

0 steps flagged

No significant circularity: the central claim is empirically grounded in external benchmarks and ablations; only minor non-load-bearing self-citations appear.

full rationale

VLM-IE3D's central claim—that adding EGTs to IGTs improves 3D VLM understanding—is not circular. EGTs are tokenized outputs of the frozen AnySplat reconstruction heads (depth, point maps, Gaussians), and IGTs are fusion-decoder features of the same encoder; both are inputs to a trainable adapter/VLM, not fitted constants relabeled as predictions. The VLM is supervised on ground-truth 3D boxes/captions from external benchmarks (EmbodiedScan, ScanRefer, Scan2Cap, VSI-Bench), and the ablations in Tables 5-9 toggle components and sources against those benchmarks, so the reported +2.3 F1 over IGT-only is an empirical result, not a construction. The concern that EGTs inherit AnySplat's reconstruction error, and that Tables 7/9 show little sensitivity to attribute type or depth source, is a correctness/robustness issue, not circularity: the paper never claims the reconstructed attributes are ground truth, and the gain could in principle come from an extra feature stream. The only self-citation issue is related-work support: refs [20] and [43] share authors and are cited as examples of RGB-only 3D VLMs, but the same claim is independently supported by [11,40,50], so the citations are not load-bearing. No self-definitional reduction, fitted-input-as-prediction, or imported uniqueness theorem appears.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

All of the following are assumed and unverified in-paper: the frozen AnySplat encoder's outputs are geometrically faithful; VLM visual-token space is additive enough for Eq. 3 to inject 3D structure without destroying 2D semantics; the multi-task data mixture and the 150/10 random splits are representative; and a single training epoch is sufficient. None of these are tested beyond the reported task numbers, and all feed directly into the headline margins.

free parameters (4)
  • Training epochs = 1
    One-epoch training (Sec. 4.1) with no ablation; all headline results are produced under this schedule, and under/overfitting sensitivity is unexamined.
  • Default explicit 3D attribute = depth (c'=1)
    Depth selected as the default explicit attribute after Table 7 showed depth (42.8), point (42.6), and Gaussian (42.5) F125 are statistically indistinguishable; a hand-made, post-hoc default, though with negligible effect on the results.
  • Multi-task data mixture = unspecified ratio
    The three scene-understanding datasets are described as 'combined' (Sec. 4.2) with no mixing ratios or per-task sampling weights; Tables 1-3 numbers can shift with this hand-made mixture, which is never reported.
  • Evaluation sampling for 3D video detection = 150 train / 10 eval samples per scene, random
    Random selection of 150 training / 10 evaluation samples per scene without a stated seed (Sec. 4.2); Table 3 and Table 5 margins are sensitive to which samples are drawn.
axioms (5)
  • domain assumption AnySplat's fusion-decoder outputs and prediction heads provide useful implicit 3D priors and geometrically faithful depth/point/Gaussian attributes from RGB video
    Section 3.1 builds both IGTs and EGTs on this single frozen encoder; no validation of reconstruction fidelity on the target benchmarks is reported, and depth error propagates directly into EGTs.
  • domain assumption Element-wise addition in VLM token space preserves 2D semantics while injecting 3D structure
    Section 3.2 / Eq. 3 fuses T3D = T2D + T3D, justified by one ablation ('direct and non-parametric addition is sufficient'); the linearity of VLM visual-token space is assumed, not demonstrated.
  • standard math Standard ML background: pretrained VLM and geometry-encoder features transfer, Adam optimization works, and dot-product attention (Eq. 1) behaves as specified
    Standard machinery invoked throughout Sections 3-4.
  • domain assumption The curated EmbodiedScan split and the 20-category protocol are representative of 3D video detection
    Section 4.2 follows the split of [50]; the small per-scene evaluation sample (10) and random selection make the headline F1 fragile to sampling.
  • domain assumption Qwen2.5-VL's visual-token stream tolerates adjoined, cross-attended fusion tokens without distributional collapse
    The method injects new tokens into T3D before the LLM; the paper assumes the pretrained VLM accommodates this without catastrophic shift, with only a single epoch of training to limit drift.
invented entities (3)
  • Implicit Geometry Tokens (IGTs) no independent evidence
    purpose: Latent 3D spatial priors drawn from AnySplat's fusion-decoder output, providing coarse global scene layout to the VLM.
    Learned internal representations with no falsifiable handle outside the paper's own benchmark tables; validation is internal only.
  • Explicit Geometry Tokens (EGTs) no independent evidence
    purpose: Tokens encoding AnySplat's reconstructed depth/point/Gaussian attributes, intended to give the VLM readable quantitative geometry.
    The paper's headline contribution; its value is supported only by the internal ablation delta (+2.3 F1 on top of IGTs), and the underlying depth is the same model's estimate rather than measured geometry.
  • Implicit-Explicit Attention (IEA) module no independent evidence
    purpose: Cross-attention adapter that aligns IGTs with EGTs before element-wise addition into the 2D visual token stream.
    An architectural construct preferred over Concat/Addition/Weighted in one ablation (Table 6); no external evidence isolates its effect.

pith-pipeline@v1.3.0-alltime-deepseek · 16710 in / 28491 out tokens · 249923 ms · 2026-08-01T06:59:05.724751+00:00 · methodology

0 comments
read the original abstract

Despite rapid progress, most existing vision-language models (VLMs) built from 2D visual inputs often struggle when handling various 3D tasks that require fine-grained spatial understanding and reasoning. To bridge this gap, we present VLM-IE3D, a unified framework that enhances the 3D spatial awareness of VLMs by equipping them with both implicit and explicit 3D geometries learned from RGB videos. Our VLM-IE3D introduces Implicit Geometry Tokens (IGTs) that capture high-level geometric priors from input videos, as well as complementary Explicit Geometry Tokens (EGTs) that encode detailed geometric structures from reconstructed 3D attributes. On top of that, VLM-IE3D comes with a 3D-aware adapter that effectively fuses the two types of geometric representations with 2D visual cues. This RGB-only design injects strong 3D inductive biases for fine-grained spatial understanding and reasoning without requiring any additional 3D inputs. Extensive experiments show that VLM-IE3D achieves superior performance consistently across various 3D tasks including 3D video detection, 3D visual grounding, 3D dense captioning, and spatial reasoning. Code and models are available at https://github.com/Vegetebird/VLM-IE3D.

Figures

Figures reproduced from arXiv: 2607.21595 by Deli Zhao, Gongjie Zhang, Quanhao Qian, Ran Xu, Shijian Lu, Wenhao Li, Xueying Jiang.

Figure 1
Figure 1. Figure 1: (a) Existing VLMs acquire 3D awareness by learning solely implicit geometric representations that provide a coarse “3D cognitive map”. (b) The proposed VLM-IE3D introduces explicit geometric representations from reconstructed 3D attributes, offering a “3D reconstruction map” that is equipped with fine-grained local geometries that are supplementary to the implicit representations. These two complementary r… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the VLM-IE3D framework. VLM-IE3D directly processes RGB frames without explicit 3D inputs, using three streams: (1) a 2D visual encoder that extracts 2D visual tokens; (2) a 3D geometry encoder that generates Implicit Geometry Tokens (IGTs) for high-level 3D information; (3) a 3D explicit embedding that converts the reconstructed 3D explicit representations (e.g., depth maps and point clouds) i… view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of our 3D-aware adapter architecture. It takes the implicit and explicit 3D geometry tokens along with 2D visual tokens as input to generate the final 3D-aware visual tokens. The token compression operation is omitted for clarity. XE ∈ R f×h×w×c ′ are then embedded into EGTs TE ∈ R f×n×c through a lightweight 3D explicit embedding module, where c ′ represents the channel dimension, which varie… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison of 3D visual grounding among Qwen2.5-VL, VG LLM, and our VLM-IE3D. This task aims to localize the first frame in the video that contains the object described in the text. Our VLM-IE3D achieves superior localization accuracy and produces more precise object bounding boxes. Qwen2.5-VL VG LLM VLM-IE3D Qwen2.5-VL VG LLM VLM-IE3D v [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative comparison of 3D video detection among Qwen2.5-VL, VG LLM, and our VLM-IE3D. Our method produces more accurate 3D bounding box predictions with superior shape alignment and spatial localization, precisely capturing object geometry. including proprietary models like Gemini-1.5-Pro (45.4%), and open-source mod￾els with significantly larger parameter counts such as LLaVA-NeXT-Video-72B (40.9%). On… view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative comparison of 3D video detection among Qwen2.5-VL, VG LLM, and our VLM-IE3D [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative comparison of 3D visual grounding among Qwen2.5-VL, VG LLM, and our VLM-IE3D. This task aims to localize the first frame in the video that contains the object described in the text [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 7 linked inside Pith

  1. [1]

    In: NeurIPS

    Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. In: NeurIPS. vol. 35, pp. 23716–23736 (2022)

    2022

  2. [2]

    arXiv preprint arXiv:2502.13923 (2025)

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923 (2025)

    Pith/arXiv arXiv 2025

  3. [3]

    In: CVPR

    Cai, D., Zhao, L., Zhang, J., Sheng, L., Xu, D.: 3DJCG: A unified framework for joint dense captioning and visual grounding on 3D point clouds. In: CVPR. pp. 16464–16473 (2022)

    2022

  4. [4]

    In: ECCV

    Chen, D.Z., Chang, A.X., Nießner, M.: ScanRefer: 3D object localization in RGB-D scans using natural language. In: ECCV. pp. 202–221 (2020)

    2020

  5. [5]

    In: ECCV

    Chen, D.Z., Wu, Q., Nießner, M., Chang, A.X.: D3net: A unified speaker-listener architecture for 3D dense captioning and visual grounding. In: ECCV. pp. 487–505 (2022)

    2022

  6. [6]

    In: NeurIPS

    Chen, S., Guhur, P.L., Tapaswi, M., Schmid, C., Laptev, I.: Language conditioned spatial relation reasoning for 3D object grounding. In: NeurIPS. vol. 35, pp. 20522– 20535 (2022)

    2022

  7. [7]

    In: CVPR

    Chen, S., Chen, X., Zhang, C., Li, M., Yu, G., Fei, H., Zhu, H., Fan, J., Chen, T.: LL3DA: Visual interactive instruction tuning for omni-3D understanding, reasoning, and planning. In: CVPR. pp. 26428–26438 (2024)

    2024

  8. [8]

    In: CVPR

    Chen, S., Zhu, H., Chen, X., Lei, Y., Yu, G., Chen, T.: End-to-end 3D dense captioning with Vote2Cap-DETR. In: CVPR. pp. 11124–11133 (2023)

    2023

  9. [9]

    arXiv preprint arXiv:2405.10370 (2024)

    Chen, Y., Yang, S., Huang, H., Wang, T., Xu, R., Lyu, R., Lin, D., Pang, J.: Grounded 3D-LLM with referent tokens. arXiv preprint arXiv:2405.10370 (2024)

    Pith/arXiv arXiv 2024

  10. [10]

    In: CVPR

    Chen, Z., Gholami, A., Nießner, M., Chang, A.X.: Scan2Cap: Context-aware dense captioning in RGB-D scans. In: CVPR. pp. 3193–3203 (2021)

    2021

  11. [11]

    In: CVPR

    Fan, Z., Zhang, J., Li, R., Zhang, J., Chen, R., Hu, H., Wang, K., Wang, P., Qu, H., Zhou, S., et al.: VLM-3R: Vision-language models augmented with instruction- aligned 3D reconstruction. In: CVPR. pp. 31054–31065 (2026)

    2026

  12. [12]

    In: WACV

    Fu, R., Liu, J., Chen, X., Nie, Y., Xiong, W.: Scene-LLM: Extending language model for 3D visual reasoning. In: WACV. pp. 2195–2206 (2025)

    2025

  13. [13]

    In: NeurIPS

    Hong, Y., Zhen, H., Chen, P., Zheng, S., Du, Y., Chen, Z., Gan, C.: 3D-LLM: Injecting the 3D world into large language models. In: NeurIPS. vol. 36, pp. 20482– 20494 (2023)

    2023

  14. [14]

    In: NeurIPS

    Huang, H., Chen, Y., Wang, Z., Huang, R., Xu, R., Wang, T., Liu, L., Cheng, X., Zhao, Y., Pang, J., et al.: Chat-Scene: Bridging 3D scene and large language models with object identifiers. In: NeurIPS. vol. 37, pp. 113991–114017 (2024)

    2024

  15. [15]

    In: ICML (2023)

    Huang, J., Yong, S., Ma, X., Linghu, X., Li, P., Wang, Y., Li, Q., Zhu, S.C., Jia, B., Huang, S.: An embodied generalist agent in 3D world. In: ICML (2023)

    2023

  16. [16]

    In: CVPR

    Huang, S., Chen, Y., Jia, J., Wang, L.: Multi-view transformer for 3D visual grounding. In: CVPR. pp. 15524–15533 (2022)

    2022

  17. [17]

    arXiv preprint arXiv:2506.01946 (2025)

    Huang, X., Wu, J., Xie, Q., Han, K.: MLLMs need 3D-aware representation super- vision for scene understanding. arXiv preprint arXiv:2506.01946 (2025)

    arXiv 2025

  18. [18]

    In: ECCV

    Jia, B., Chen, Y., Yu, H., Wang, Y., Niu, X., Liu, T., Li, Q., Huang, S.: Sceneverse: Scaling 3D vision-language learning for grounded scene understanding. In: ECCV. pp. 289–310 (2024) 16 Li et al

    2024

  19. [19]

    ACM Transactions on Graphics44(6), 1–16 (2025)

    Jiang, L., Mao, Y., Xu, L., Lu, T., Ren, K., Jin, Y., Xu, X., Yu, M., Pang, J., Zhao, F., et al.: AnySplat: Feed-forward 3D gaussian splatting from unconstrained views. ACM Transactions on Graphics44(6), 1–16 (2025)

    2025

  20. [20]

    arXiv preprint arXiv:2605.19528 (2026)

    Jiang, X., Li, W., Qian, Q., Zhao, D., Lu, S., Zhang, G., Xu, R.: Towards camera- robust 3D localization: Equation-anchored tool-use for MLLMs. arXiv preprint arXiv:2605.19528 (2026)

    Pith/arXiv arXiv 2026

  21. [21]

    Cognitive science4(1), 71–115 (1980)

    Johnson-Laird, P.N.: Mental models in cognitive science. Cognitive science4(1), 71–115 (1980)

    1980

  22. [22]

    Johnson-Laird, P.N.: Mental models: Towards a cognitive science of language, inference, and consciousness. No. 6, Harvard University Press (1983)

    1983

  23. [23]

    In: CoRL (2024)

    Kim, M.J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al.: OpenVLA: An open-source vision- language-action model. In: CoRL (2024)

    2024

  24. [24]

    In: ICML

    Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: ICML. pp. 12888–12900 (2022)

    2022

  25. [25]

    arXiv preprint arXiv:2305.06355 (2023)

    Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., Qiao, Y.: Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355 (2023)

    Pith/arXiv arXiv 2023

  26. [26]

    In: EMNLP

    Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., Yuan, L.: Video-LLaVA: Learning united visual representation by alignment before projection. In: EMNLP. pp. 5971–5984 (2024)

    2024

  27. [27]

    In: CVPR

    Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. In: CVPR. pp. 26296–26306 (2024)

    2024

  28. [28]

    In: NeurIPS

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: NeurIPS. vol. 36, pp. 34892–34916 (2023)

    2023

  29. [29]

    IEEE Transactions on Neural Networks and Learning Systems (2026)

    Ma, Y., Song, Z., Zhuang, Y., Hao, J., King, I.: A survey on vision–language–action models for embodied AI. IEEE Transactions on Neural Networks and Learning Systems (2026)

    2026

  30. [30]

    In: ICLR

    Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., Ye, Q., Wei, F.: Grounding multimodal large language models to the world. In: ICLR. vol. 2024, pp. 51575–51598 (2024)

    2024

  31. [31]

    In: ICLR (2026)

    Qi, Z., Zhang, Z., Fang, Y., Wang, J., Zhao, H.: GPT4Scene: Understand 3D scenes from videos with vision-language models. In: ICLR (2026)

    2026

  32. [32]

    In: ICML

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748–8763 (2021)

    2021

  33. [33]

    In: CVPR

    Tang, Z., Fan, Y., Wang, D., Xu, H., Ranjan, R., Schwing, A., Yan, Z.: MV- DUSt3R+: Single-stage scene reconstruction from sparse views in 2 seconds. In: CVPR. pp. 5283–5293 (2025)

    2025

  34. [34]

    In: NeurIPS

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: NeurIPS. pp. 5998–6008 (2017)

    2017

  35. [35]

    In: ICCV

    Wang, H., Zhao, Y., Wang, T., Fan, H., Zhang, X., Zhang, Z.: Ross3D: Recon- structive visual instruction tuning with 3D-awareness. In: ICCV. pp. 9275–9286 (2025)

    2025

  36. [36]

    In: CVPR

    Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., Novotny, D.: VGGT: Visual geometry grounded transformer. In: CVPR. pp. 5294–5306 (2025)

    2025

  37. [37]

    In: CVPR

    Wang, S., Leroy, V., Cabon, Y., Chidlovskii, B., Revaud, J.: Dust3R: Geometric 3D vision made easy. In: CVPR. pp. 20697–20709 (2024) 3D-Aware VLMs with Implicit and Explicit Geometries 17

    2024

  38. [38]

    In: CVPR

    Wang, T., Mao, X., Zhu, C., Xu, R., Lyu, R., Li, P., Chen, X., Zhang, W., Chen, K., Xue, T., et al.: EmbodiedScan: A holistic multi-modal 3D perception suite towards embodied AI. In: CVPR. pp. 19757–19767 (2024)

    2024

  39. [39]

    arXiv preprint arXiv:2507.13347 (2025)

    Wang, Y., Zhou, J., Zhu, H., Chang, W., Zhou, Y., Li, Z., Chen, J., Pang, J., Shen, C., He, T.:π3: Scalable permutation-equivariant visual geometry learning. arXiv preprint arXiv:2507.13347 (2025)

    Pith/arXiv arXiv 2025

  40. [40]

    In: NeurIPS

    Wu, D., Liu, F., Hung, Y.H., Duan, Y.: Spatial-MLLM: Boosting MLLM capabilities in visual-based spatial intelligence. In: NeurIPS. vol. 38, pp. 13569–13597 (2026)

    2026

  41. [41]

    In: ECCV

    Xu, R., Wang, X., Wang, T., Chen, Y., Pang, J., Lin, D.: PointLLM: Empowering large language models to understand point clouds. In: ECCV. pp. 131–147 (2024)

    2024

  42. [42]

    In: NeurIPS

    Yang, L., Kang, B., Huang, Z., Zhao, Z., Xu, X., Feng, J., Zhao, H.: Depth anything v2. In: NeurIPS. vol. 37, pp. 21875–21911 (2024)

    2024

  43. [43]

    In: ICLR (2026)

    Zhang,G.,Li, W.,Qian,Q.,Wang,J.,Zhao,D., Lu,S.,Xu,R.:On thegeneralization capacities of MLLMs for spatial intelligence. In: ICLR (2026)

    2026

  44. [44]

    In: EMNLP

    Zhang, H., Li, X., Bing, L.: Video-LLaMA: An instruction-tuned audio-visual language model for video understanding. In: EMNLP. pp. 543–553 (2023)

    2023

  45. [45]

    In: NeurIPS

    Zhang, J., Chen, Y., Xu, Y., Huang, Z., Mei, J., Chen, C., Zhou, Y., Yuan, Y.J., Cai, X., Huang, G., et al.: From flatland to space: Teaching vision-language models to perceive and reason in 3D. In: NeurIPS. vol. 38 (2026)

    2026

  46. [46]

    In: AAAI

    Zhang, T., He, S., Dai, T., Wang, Z., Chen, B., Xia, S.T.: Vision-language pre- training with object contrastive learning for 3D scene understanding. In: AAAI. vol. 38, pp. 7296–7304 (2024)

    2024

  47. [47]

    arXiv preprint arXiv:2410.02713 (2024)

    Zhang, Y., Wu, J., Li, W., Li, B., Ma, Z., Liu, Z., Li, C.: LLaVA-Video: Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713 (2024)

    Pith/arXiv arXiv 2024

  48. [48]

    arXiv preprint arXiv:2407.07035 (2024)

    Zhang, Y., Ma, Z., Li, J., Qiao, Y., Wang, Z., Chai, J., Wu, Q., Bansal, M., Kordjamshidi, P.: Vision-and-language navigation today and tomorrow: A survey in the era of foundation models. arXiv preprint arXiv:2407.07035 (2024)

    Pith/arXiv arXiv 2024

  49. [49]

    In: CVPR

    Zheng, D., Huang, S., Wang, L.: Video-3D LLM: Learning position-aware video representation for 3D scene understanding. In: CVPR. pp. 8995–9006 (2025)

    2025

  50. [50]

    In: NeurIPS

    Zheng, D., Li, Y., Wang, L., et al.: Learning from videos for 3D world: Enhancing mllms with 3D vision geometry priors. In: NeurIPS. vol. 38, pp. 20560–20586 (2026)

    2026

  51. [51]

    In: AAAI

    Zhou, G., Hong, Y., Wu, Q.: NavGPT: Explicit reasoning in vision-and-language navigation with large language models. In: AAAI. vol. 38, pp. 7641–7649 (2024)

    2024

  52. [52]

    frame": 7,

    Zhu, C., Wang, T., Zhang, W., Pang, J., Liu, X.: LLaVA-3D: A simple yet effective pathway to empowering LMMs with 3D capabilities. In: ICCV. pp. 4295–4305 (2025) 18 Li et al. Supplementary Material This supplementary material covers the following details: –The prompts for 3D scene understanding tasks (Sec. A). –Additional visualization results (Sec. B). A...

    2025