Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Multimodal LLMs miss 3D structure not for lack of geometry in RGB inputs but because standard fine-tuning never switches that geometry on; this paper shows a sparse joint pre-training stage that fixes it, lifting grounding, captioning, and

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 18:00 UTC pith:PEMU67Y4

load-bearing objection Solid training recipe with consistent gains, but the headline causal claim that the RGB-only 3D gap is purely a training-paradigm problem outruns the evidence: pre-training data/compute is uncontrolled and scene overlap is never ruled out. the 5 major comments →

arxiv 2603.16461 v2 pith:PEMU67Y4 submitted 2026-03-17 cs.CV

GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Models

Jiaxin Zhang , Junjun Jiang , Haijie Li , Youyu Chen , Kui Jiang , Dave Zhenyu Chen This is my paper
classification cs.CV MSC 68T45
keywords multimodal large language models3D spatial perceptiongeometry-aligned pre-trainingimplicit geometric priorspointmap prediction3D visual groundinggated feature fusionmetric 3D reconstruction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that the gap between RGB-only multimodal language models and explicit-3D models in spatial perception is not a shortage of geometric information, but a failure of training: standard text-dominated fine-tuning never switches on the geometry already contained in the image features. To fix this, the authors introduce GAP-MLLM, a geometry-aligned pre-training stage in which the model must output, for a single prompted pixel, both its metric 3D coordinate (a pointmap value) and its semantic label. They pair this with a multi-level gated fusion module that blends geometric and visual tokens layer by layer with learned token-level weights. Across three tasks—3D visual grounding, dense captioning, and 3D video object detection—the method improves over prior RGB-only baselines and approaches the accuracy of systems that consume explicit point clouds, using only about two images' worth of sparse labels for pre-training. The authors also show the recipe transfers to an existing model (VG-LLM) and improves it, suggesting the finding is about the training paradigm rather than a particular architecture.

Core claim

The paper's central claim is that the performance gap in 3D perception between RGB-only MLLMs with implicit geometric priors and methods with explicit 3D input is caused by a geometry-semantics imbalance in the training paradigm, not by insufficient geometric priors. Prior work naively concatenates geometric features and fine-tunes on text-heavy downstream objectives, so geometry stays weakly activated. GAP-MLLM instead runs a sparse joint pre-training objective—predict the metric 3D pointmap coordinate and the semantic label of a red-cross pixel under a unified first-frame metric coordinate system—before task fine-tuning. Results show this activation step consistently raises performance on

What carries the argument

Two devices carry the argument. First, the sparse geometry-semantics joint pre-training task: given a multi-frame image sequence and a red-cross visual prompt, the model must jointly output the semantic label and the metric (x,y,z) pointmap coordinate of that pixel in the first-frame coordinate system, forcing the LLM to hold geometry and semantics in one representation. Second, the multi-level progressive fusion module: at each of 24 layers, a sigmoid-gated MLP computes per-token weights g that blend visual tokens T_V and geometric tokens T_G via T_S = g ⊙ T_V + (1−g) ⊙ T_G; final-layer tokens enter the LLM decoder while selected intermediate layers (5, 11, 17) are injected into early decod

Load-bearing premise

The gated fusion in Eq. (3) blends geometric and visual tokens element-wise, so it only works if the two branches' token grids genuinely correspond pixel-to-pixel after token merging; the paper asserts this via identical architectural design but never verifies that geometric-branch and visual-branch tokens at the same grid position actually describe the same scene point.

What would settle it

Take a set of test frames, extract the geometric token at grid position (i,j) from VGGT and the visual token at (i,j) from Qwen3-VL; reproject the VGGT pointmap coordinates for that token into the image and compute the semantic/visual similarity between the visual token at (i,j) and the visual token at the reprojected location. If similarity is no higher than random pairs, the tokens are not aligned and the fusion mechanism is not what the paper claims. Alternatively, randomly permute the geometric tokens' spatial positions before fusion; if detection/grounding performance barely drops, spatia

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A 3B RGB-only model reaches Acc@0.25 of 53.1 on the ScanRefer grounding benchmark, within about 5 points of the best explicit-3D methods and far above prior RGB-only baselines (VG-LLM-4B: 36.4).
  • The GAP recipe transfers: applying the same pre-training to VG-LLM improves it on all three tasks, so geometry-aligned pre-training, not a specific fusion architecture, is the driver.
  • Sparse supervision suffices—roughly 500K prompted pixels, equivalent to about two 680×480 images—meaning the activation stage is cheap to run.
  • Joint semantic-plus-geometry supervision outperforms pointmap-only training, in both downstream detection F1 and metric reconstruction accuracy, indicating semantic labels help stabilize metric structure.
  • The model produces metric-consistent pointmaps under the unified first-frame coordinate system, which is what makes the same representation transferable across grounding, captioning, and detection.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Testable extension the paper does not run: shuffle the geometric tokens spatially before gated fusion. If accuracy barely drops, the fusion's element-wise alignment is not the real mechanism; if it collapses, alignment is confirmed as load-bearing.
  • The paradigm implies a general recipe for any implicit prior—depth, optical flow, surface normals—that a cheap 'predict prior + semantic label' stage should beat naive feature concatenation; worth probing on priors beyond VGGT.
  • The authors' stated limitation that sparse supervision yields coarse boundary maps suggests a density-ablation path: scaling pre-training labels from one prompted pixel to a low-resolution full pointmap could reveal whether fine-grained geometry follows, or whether task transfer degrades.
  • If the central claim is right, future RGB-only 3D benchmark comparisons should control for the pre-training stage; otherwise architecture wins may be confounded by whether geometry was activated in training.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper argues that the gap between RGB-only MLLM-based 3D perception and explicit-3D methods is not due to insufficient geometric priors but to a training-paradigm misalignment: text-dominated fine-tuning fails to activate geometric representations. To address this, the authors propose GAP-MLLM, which adds (1) a sparse geometry–semantics joint pre-training stage that asks the MLLM to predict pointmap coordinates and semantic labels for prompted pixels under a unified first-frame metric coordinate system, and (2) a multi-level gated fusion module that adaptively combines visual and geometric tokens from multiple layers. Experiments on ScanRefer 3D visual grounding, Scan2Cap dense captioning, EmbodiedScan video object detection, and ScanNet metric reconstruction report consistent improvements over RGB-only baselines, with ablations showing contributions from both main components.

Significance. If the central causal claim is correct, the paper offers a practical recipe for activating implicit geometric priors in RGB-only MLLMs with sparse supervision, potentially reducing the reliance on explicit 3D data. The reported gains are consistent and nontrivial across three downstream tasks and two backbone configurations, and the paper ships detailed prompt templates and ablations that support reproducibility of the method itself. However, the strength of the causal claim depends on controls that are not currently provided: the pre-training stage adds data and compute without a non-geometric control, and the pre-training/evaluation scene overlap is not addressed. The metric-reconstruction probe in Table 4 is essentially the pre-training task and therefore does not independently validate downstream transfer. The paper is a solid empirical contribution, but the attribution of the gains to 'geometry alignment' rather than to additional training data or architectural components needs further evidence.

major comments (5)
  1. [§4.1 and Table 13] The central claim—that gains stem from geometry-aligned pre-training rather than from extra data/compute—is not isolated. The sparse joint pre-training stage uses approximately 500K additional samples, but no control run with the same data budget and a non-geometric objective (e.g., semantic-only pre-training, generic captioning, or depth-prediction without coordinate semantics) is reported. Table 13 isolates the presence/absence of components A and B, but A always includes the geometric objective. Please provide a control with the same extra data and a non-geometric pre-training task; otherwise the gains in Tables 1–3 and 5 could be attributed to additional training data/compute rather than to activation of geometric priors.
  2. [§4.1 (Pre-training dataset) and Tables 1–3] The paper never states whether the scenes used for sparse joint pre-training (ScanNet and EmbodiedScan) are disjoint from the scenes used for downstream evaluation (ScanRefer and Scan2Cap on ScanNet; EmbodiedScan detection). If pre-training includes evaluation scenes, the headline numbers partially reflect test-set leakage rather than transfer of activated geometric priors. Please include an explicit statement of scene-level disjointness and, ideally, report results trained only on the training-split scenes. This is load-bearing for the transfer claim.
  3. [Table 1 vs. Supplementary Table 14] The main-paper comparison 'VG-LLM-4B' (36.4/11.8) vs. 'VG-LLM-4B (w/ GAP)' (49.7/23.2) conflates the GAP components with the two-stage grounding change. Supplementary Table 14 shows that two-stage grounding alone improves Acc@0.25 from 37.5 to 45.4 on the VG-LLM baseline. Since all GAP-MLLM numbers in Table 13 use two-stage grounding, the direct comparison in Table 1 attributes the full +13.3 points to GAP. The main paper should either report the two-stage-grounding baseline in Table 1 or clearly add a note pointing to Table 14, so that the reader can see the true contribution of the geometry-aligned pre-training and fusion (about +4–7 points after removing the grounding reformulation).
  4. [Table 4 and §4.2 (Pointmaps and Semantic labels)] The metric reconstruction probe in Table 4 evaluates essentially the same task used in pre-training (predicting pointmap coordinates from sparse prompts). While this serves as a sanity check that the model learned the pre-training objective, it does not provide independent evidence for the claim that geometry-aligned pre-training improves downstream 3D perception. The downstream tasks (Tables 1–3) are the appropriate tests. The presentation should be adjusted to avoid implying that Table 4 is a transfer result; at most it demonstrates that the pre-training objective was learned.
  5. [§3.1, Eq. (3)] The gated fusion in Eq. (3) performs element-wise blending of visual and geometric tokens, which requires the two token grids to be spatially aligned after token merging. The paper asserts this alignment by giving the geometric branch an 'identical architectural design' to the visual branch, but provides no direct verification that VGGT features and Qwen3-VL features at corresponding spatial positions correspond to the same image locations. If the two branches use different input resolutions, patch embedding strides, or feature normalization, the alignment may be only approximate. Please provide a quantitative check of spatial correspondence (e.g., mutual-nearest-neighbor agreement on a small set of frames) or an ablation that uses a learned cross-attention instead of element-wise fusion to assess sensitivity to misalignment.
minor comments (6)
  1. [Abstract / §1] The phrase 'activating structural perception' is used many times without a formal definition. Consider stating what activation means quantitatively (e.g., higher feature norm, lower entropy, or improved pointmap prediction) to make the claim falsifiable.
  2. [Table 13 and §4.3] All results are reported as single runs without variance or significance tests. Given the moderate differences in some ablations (e.g., Table 8: Gated 50.6 vs. Cross-Attention 49.1), please report at least two runs or a standard-error estimate to support the claim of consistent improvement.
  3. [Fig. 9] The labels 'Layer ??', 'Layer ??', 'Layer ??' are placeholder-like. Please replace with the actual layer numbers (L1, L2, L3) or describe them in the caption.
  4. [§4.1 (Implementation Details)] The authors mention that the visual encoder of the MLLM and the geometric encoder are frozen. However, the fusion module includes MLPs that are trained from scratch. It would be useful to state the number of trainable parameters and the computational cost of the pre-training stage to contextualize the comparisons.
  5. [References] The related-work discussion would benefit from a citation to recent works on cross-modal feature alignment in MLLMs, which the current §2 does not cover in detail. Also, reference [7] appears to be a duplicate of [8] with slightly different titles; please verify.
  6. [Supplementary Table 14] The caption says 'Introducing Two-Stage Grounding consistently improves performance across multiple 3D perception tasks', but the Scan2Cap gains are small (76.9→77.0 CIDEr). This is not a problem, but the phrasing 'consistently' should be softened to avoid overclaiming.

Circularity Check

1 steps flagged

Partial circularity in the pointmap 'probe' (Tab. 4); downstream transfer claims are independent.

specific steps
  1. fitted input called prediction [Sec. 3.2 (Sparse Geometry-Semantics Joint Pre-training) and Sec. 4.2 / Tab. 4 (Results on Pointmaps and Semantic labels)]
    "we introduce a sparse geometry–semantics joint pre-training objective ... the model predicts both the 3D coordinate (x, y, z) of the prompted pixel and its semantic label c ... Tab. 4 reports aligned and metric evaluations on 10 ScanNet scenes. ... GAP-MLLM performs best under metric evaluation, indicating stronger metric-aware reconstruction."

    The metric reconstruction probe outputs pointmaps, which are exactly the supervised output of the pre-training objective. Demonstrating pointmap accuracy after pointmap-supervised pre-training is a measure of the training objective, not an independent prediction of activated geometric priors. The non-circular evidence for the central claim is the downstream transfer to grounding/captioning/detection (Tabs 1-3), which are not the pre-training task. Thus this is a partial, non-central circularity rather than a collapse of the main derivation.

full rationale

The paper's central claim — that geometry-aligned pre-training activates structural perception and transfers to downstream 3D perception — is evaluated on ScanRefer, Scan2Cap, and EmbodiedScan detection, which are not the sparse pointmap+label pre-training objective, so those headline results are not forced by construction. The Tab. 4 pointmap probe does re-test essentially the training objective, so it should be read as a sanity check rather than as an independent prediction; this is the only circular component. The causal attribution is also underdetermined by confounds (extra 500K pre-training samples/compute and unstated train/eval scene overlap), but those are experimental validity risks, not definitional circularity. Self-citations such as [56] for the two-stage grounding formulation are backed by the Tab. 14 ablation and are not load-bearing. Overall, the derivation is substantially independent, with one partial same-task probe slightly inflating the 'activation' evidence.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

No new physical entities or forces are introduced; the fusion module and pre-training objective are architectural/training mechanisms. The only hand-chosen design parameter with the potential to affect the central claim is the selection of intermediate fusion layers.

free parameters (1)
  • intermediate fusion layers L1, L2, L3 = 5, 11, 17
    Hand-chosen indices for injecting intermediate fused tokens into early decoder layers; no sensitivity analysis or ablation over layer choices is provided.
axioms (3)
  • domain assumption VGGT provides sufficiently accurate, pixel-aligned geometric priors from RGB frames
    The pipeline freezes VGGT and uses its features as the only geometric signal (Sec. 3.1). If VGGT's geometry is poor on the target scenes, pre-training cannot activate reliable structure.
  • domain assumption Visual and geometric token grids are spatially aligned after token merging
    Eq. (2)-(3) blend tokens element-wise, requiring corresponding spatial positions. The paper asserts alignment from identical merging architecture but shows no direct verification.
  • ad hoc to paper First-frame metric coordinate system is a consistent supervision space across all tasks
    Introduced to unify outputs; it assumes that a single coordinate frame can express grounding, captioning, and detection well (Sec. 3.2).

pith-pipeline@v1.3.0-alltime-deepseek · 18022 in / 10685 out tokens · 105294 ms · 2026-08-02T18:00:19.230931+00:00 · methodology

0 comments
read the original abstract

Multimodal Large Language Models (MLLMs) demonstrate exceptional semantic reasoning but struggle with 3D spatial perception when restricted to pure RGB inputs. Despite leveraging implicit geometric priors from 3D reconstruction models, image-based methods still exhibit a notable performance gap compared to methods using explicit 3D data. We argue that this gap does not arise from insufficient geometric priors, but from a misalignment in the training paradigm: text-dominated fine-tuning fails to activate geometric representations within MLLMs. Existing approaches typically resort to naive feature concatenation and optimize directly for downstream tasks without geometry-specific supervision, leading to suboptimal structural utilization. To address this limitation, we propose GAP-MLLM, a Geometry-Aligned Pre-training paradigm that explicitly activates structural perception before downstream adaptation. Specifically, we introduce a visual-prompted joint task that compels the MLLMs to predict sparse pointmaps alongside semantic labels, thereby enforcing geometric awareness. Furthermore, we design a multi-level progressive fusion module with a token-level gating mechanism, enabling adaptive integration of geometric priors without suppressing semantic reasoning. Extensive experiments demonstrate that GAP-MLLM significantly enhances geometric feature fusion and consistently enhances performance across 3D visual grounding, 3D dense captioning, and 3D video object detection tasks.

Figures

Figures reproduced from arXiv: 2603.16461 by Dave Zhenyu Chen, Haijie Li, Jiaxin Zhang, Junjun Jiang, Kui Jiang, Youyu Chen.

Figure 1
Figure 1. Figure 1: Geometry-aligned pre-training significantly improves 3D perception in MLLMs. (A) Naive fusion without geometry-aware pre-training leads to limited geometric uti￾lization and inaccurate 3D detection. (B) Our sparse geometry–semantics joint pre￾training and multi-level fusion module progressively integrate geometric priors, acti￾vating structural perception and yielding substantial gains in 3D detection mAP.… view at source ↗
Figure 2
Figure 2. Figure 2: Geometry–semantics imbalance in existing 3D perception paradigms. Point cloud-based methods ensure geometric accuracy but lack semantics, whereas image￾based geometric encoders (GE) with VLMs retain semantics yet under-utilize geometry. As a result, both paradigms exhibit suboptimal performance in 3D perception. tasks, we introduce a joint reconstruction-perception objective that aligns ge￾ometric priors w… view at source ↗
Figure 3
Figure 3. Figure 3: Failure Analysis. Global attention maps of the geometric encoder are visual￾ized using three representative tokens across layers. The same token exhibits distinct attention patterns at different layers. Prior method that relies on last-layer tokens as implicit geometric priors [57] leads to inaccurate 3D bounding box prediction. 3 Method We present GAP-MLLM, a geometry-aligned pre-training framework design… view at source ↗
Figure 4
Figure 4. Figure 4: Network Architecture. An image sequence is processed by parallel geometric and visual branches to extract multi-level structural and semantic tokens. After gated multi-level fusion, the final-layer fused tokens are aligned with task-related textual representations in the video LLM decoder, while selected intermediate-layer tokens are injected into early decoder blocks to preserve hierarchical geometric inf… view at source ↗
Figure 5
Figure 5. Figure 5: Training Strategy. Sparse joint pre-training (left) activates 3D representa￾tions under a unified first-frame metric coordinate system. The learned representations are then transferred to object-level fine-tuning (right) for downstream 3D perception. object-level supervision remains insufficient to activate implicit geometric pri￾ors, as optimization is dominated by language-driven objectives. We argue tha… view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative comparison on 3D visual grounding. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative comparison on 3D video object detection. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative results on pointmap and semantic prediction. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Visualization of Layer-wise gating weights. Intermediate layers emphasize geome￾try, while later layers focus on semantics [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Qualitative comparison on metric reconstruction. [PITH_FULL_IMAGE:figures/full_fig_p025_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Additional qualitative results on 3D visual grounding. [PITH_FULL_IMAGE:figures/full_fig_p026_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Additional qualitative results on 3D video object detection. [PITH_FULL_IMAGE:figures/full_fig_p027_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SpatialFusion: Endowing Unified Image Generation with Intrinsic 3D Geometric Awareness

    cs.CV 2026-04 unverdicted novelty 6.0

    SpatialFusion internalizes 3D geometric awareness into unified image generation models by pairing an MLLM with a spatial transformer that produces depth maps to constrain diffusion generation.

Reference graph

Works this paper leans on

61 extracted references · 20 linked inside Pith · cited by 1 Pith paper

  1. [1]

    In: European conference on computer vision

    Achlioptas, P., Abdelreheem, A., Xia, F., Elhoseiny, M., Guibas, L.: Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In: European conference on computer vision. pp. 422–440. Springer (2020)

    2020

  2. [2]

    arXiv preprint arXiv:2511.21631 (2025)

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hui, B., Jiang, S., Li, Z., Li, M., Li, M., Li, K., Lin, Z., Lin, J., Liu, X., Liu, J., Liu, C., Liu, Y., Liu, D., Liu, S., Lu, D., Luo, R., Lv, C., Men, R., Meng, L., Ren, X., Ren, X., Song, S., Sun, Y., Tang, ...

    Pith/arXiv arXiv 2025

  3. [3]

    5-vl technical report

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    Pith/arXiv arXiv 2025

  4. [4]

    arXiv preprint arXiv:2509.25413 (2025)

    Cai, Z., Yeh, C.F., Xu, H., Liu, Z., Meyer, G., Lei, X., Zhao, C., Li, S.W., Chandra, V., Shi, Y.: Depthlm: Metric depth from vision language models. arXiv preprint arXiv:2509.25413 (2025)

    arXiv 2025

  5. [5]

    arXiv preprint arXiv:2410.01647 (2024)

    Cao, Y., Jv, Y., Xu, D.: 3dgs-det: Empower 3d gaussian splatting with bound- ary guidance and box-focused sampling for 3d object detection. arXiv preprint arXiv:2410.01647 (2024)

    arXiv 2024

  6. [6]

    arXiv preprint arXiv:2603.00912 (2026)

    Cao,Y.,Wu,F.,Chen,D.Z.,Zhong,Y.,Hong,L.,Xu,D.:Vggt-det:Miningvggtin- ternal priors for sensor-geometry-free multi-view indoor 3d object detection. arXiv preprint arXiv:2603.00912 (2026)

    arXiv 2026

  7. [7]

    Advances in Neu- ral Information Processing Systems36, 71862–71873 (2023)

    Cao, Y., Yihan, Z., Xu, H., Xu, D.: Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection. Advances in Neu- ral Information Processing Systems36, 71862–71873 (2023)

    2023

  8. [8]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2025) 16 J

    Cao, Y., Zeng, Y., Xu, H., Xu, D.: Collaborative novel object discovery and box-guided cross-modal alignment for open-vocabulary 3d object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence (2025) 16 J. Zhang et al

    2025

  9. [9]

    In: ECCV (2020)

    Chen, D.Z., Chang, A.X., Nießner, M.: Scanrefer: 3d object localization in RGB-D scans using natural language. In: ECCV (2020)

    2020

  10. [10]

    Chen, D.Z., Wu, Q., Nießner, M., Chang, A.X.: D 3 net: A unified speaker-listener architecturefor3ddensecaptioningandvisualgrounding.In:EuropeanConference on Computer Vision. pp. 487–505. Springer (2022)

    2022

  11. [11]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Chen, S., Chen, X., Zhang, C., Li, M., Yu, G., Fei, H., Zhu, H., Fan, J., Chen, T.: Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 26428–26438 (2024)

    2024

  12. [12]

    arXiv preprint arXiv:2405.10370 (2024)

    Chen, Y., Yang, S., Huang, H., Wang, T., Xu, R., Lyu, R., Lin, D., Pang, J.: Grounded 3d-llm with referent tokens. arXiv preprint arXiv:2405.10370 (2024)

    Pith/arXiv arXiv 2024

  13. [13]

    arXiv preprint arXiv:2510.13800 (2025)

    Chen, Y., Qi, Z., Zhang, W., Jin, X., Zhang, L., Liu, P.: Reasoning in space via grounding in the world. arXiv preprint arXiv:2510.13800 (2025)

    arXiv 2025

  14. [14]

    In: Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition

    Chen, Z., Gholami, A., Nießner, M., Chang, A.X.: Scan2cap: Context-aware dense captioning in rgb-d scans. In: Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition. pp. 3193–3203 (2021)

    2021

  15. [15]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Chen, Z., Hu, R., Chen, X., Nießner, M., Chang, A.X.: Unit3d: A unified trans- former for 3d dense captioning and visual grounding. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 18109–18119 (2023)

    2023

  16. [16]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Dai,A.,Chang,A.X.,Savva,M.,Halber,M.,Funkhouser,T.,Nießner,M.:Scannet: Richly-annotated 3d reconstructions of indoor scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5828–5839 (2017)

    2017

  17. [17]

    arXiv preprint arXiv:2310.19516 (2023)

    Dwedari, M.M., Niessner, M., Chen, D.Z.: Generating context-aware natural an- swers for questions in 3d scenes. arXiv preprint arXiv:2310.19516 (2023)

    Pith/arXiv arXiv 2023

  18. [18]

    arXiv preprint arXiv:2505.20279 (2025)

    Fan, Z., Zhang, J., Li, R., Zhang, J., Chen, R., Hu, H., Wang, K., Qu, H., Wang, D., Yan, Z., et al.: Vlm-3r: Vision-language models augmented with instruction-aligned 3d reconstruction. arXiv preprint arXiv:2505.20279 (2025)

    Pith/arXiv arXiv 2025

  19. [19]

    arXiv preprint arXiv:2503.21776 (2025)

    Feng, K., Gong, K., Li, B., Guo, Z., Wang, Y., Peng, T., Wu, J., Zhang, X., Wang, B., Yue, X.: Video-r1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776 (2025)

    Pith/arXiv arXiv 2025

  20. [20]

    arXiv preprint arXiv:2601.11442 (2026)

    Gao, X., Zhang, Z., Chen, D.Z., Xu, S., Quan, L., Pérez-Pellitero, E., Jang, Y.: Map2thought: Explicit 3d spatial reasoning via metric cognitive maps. arXiv preprint arXiv:2601.11442 (2026)

    arXiv 2026

  21. [21]

    Cambridge university press (2003)

    Hartley, R., Zisserman, A.: Multiple view geometry in computer vision. Cambridge university press (2003)

    2003

  22. [22]

    Advances in Neural Information Processing Systems36, 20482–20494 (2023)

    Hong, Y., Zhen, H., Chen, P., Zheng, S., Du, Y., Chen, Z., Gan, C.: 3d-llm: In- jecting the 3d world into large language models. Advances in Neural Information Processing Systems36, 20482–20494 (2023)

    2023

  23. [23]

    Advances in Neural Information Processing Systems 37, 113991–114017 (2024)

    Huang, H., Chen, Y., Wang, Z., Huang, R., Xu, R., Wang, T., Liu, L., Cheng, X., Zhao, Y., Pang, J., et al.: Chat-scene: Bridging 3d scene and large language models with object identifiers. Advances in Neural Information Processing Systems 37, 113991–114017 (2024)

    2024

  24. [24]

    arXiv preprint arXiv:2311.12871 (2023)

    Huang, J., Yong, S., Ma, X., Linghu, X., Li, P., Wang, Y., Li, Q., Zhu, S.C., Jia, B., Huang, S.: An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871 (2023)

    Pith/arXiv arXiv 2023

  25. [25]

    arXiv preprint arXiv:2410.21276 (2024)

    Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Os- trow, A., Welihinda, A., Hayes, A., Radford, A., et al.: Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024)

    Pith/arXiv arXiv 2024

  26. [26]

    arXiv preprint arXiv:2509.13414 (2025) GAP-MLLM 17

    Keetha, N., Müller, N., Schönberger, J., Porzi, L., Zhang, Y., Fischer, T., Knapitsch, A., Zauss, D., Weber, E., Antunes, N., et al.: Mapanything: Univer- sal feed-forward metric 3d reconstruction. arXiv preprint arXiv:2509.13414 (2025) GAP-MLLM 17

    Pith/arXiv arXiv 2025

  27. [27]

    arXiv preprint arXiv:2406.09246 (2024)

    Kim, M.J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al.: Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246 (2024)

    Pith/arXiv arXiv 2024

  28. [28]

    arXiv preprint arXiv:2512.14364 (2025)

    Koch, S., Wald, J., Matsuki, H., Hermosilla, P., Ropinski, T., Tombari, F.: Unified semantic transformer for 3d scene understanding. arXiv preprint arXiv:2512.14364 (2025)

    Pith/arXiv arXiv 2025

  29. [29]

    arXiv preprint arXiv:2408.03326 (2024)

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al.: Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024)

    Pith/arXiv arXiv 2024

  30. [30]

    arXiv preprint arXiv:2510.22706 (2025)

    Li, H., Zou, Z., Liu, F., Zhang, X., Hong, F., Cao, Y., Lan, Y., Zhang, M., Yu, G., Zhang, D., et al.: Iggt: Instance-grounded geometry transformer for semantic 3d reconstruction. arXiv preprint arXiv:2510.22706 (2025)

    arXiv 2025

  31. [31]

    In: International conference on machine learning

    Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: International conference on machine learning. pp. 19730–19742. PMLR (2023)

    2023

  32. [32]

    arXiv preprint arXiv:2511.10647 (2025)

    Lin, H., Chen, S., Liew, J., Chen, D.Y., Li, Z., Shi, G., Feng, J., Kang, B.: Depth anything 3: Recovering the visual space from any views. arXiv preprint arXiv:2511.10647 (2025)

    Pith/arXiv arXiv 2025

  33. [33]

    arXiv preprint arXiv:2405.10255 (2024)

    Ma, X., Smart, B., Bhalgat, Y., Chen, S., Li, X., Ding, J., Gu, J., Chen, D.Z., Peng, S., Bian, J.W., et al.: When llms step into the 3d world: A survey and meta-analysis of 3d tasks via multi-modal large language models. arXiv preprint arXiv:2405.10255 (2024)

    arXiv 2024

  34. [34]

    In: Ad- vances in Neural Information Processing Systems (2025)

    Mao, Y., Zhong, J., Fang, C., Zheng, J., Tang, R., Zhu, H., Tan, P., Zhou, Z.: Spatiallm: Training large language models for structured indoor modeling. In: Ad- vances in Neural Information Processing Systems (2025)

    2025

  35. [35]

    Ad- vances in Neural Information Processing Systems37, 23464–23487 (2024)

    Meng, L., Yang, J., Tian, R., Dai, X., Wu, Z., Gao, J., Jiang, Y.G.: Deepstack: Deeply stacking visual tokens is surprisingly simple and effective for lmms. Ad- vances in Neural Information Processing Systems37, 23464–23487 (2024)

    2024

  36. [36]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Qi, C.R., Chen, X., Litany, O., Guibas, L.J.: Imvotenet: Boosting 3d object detec- tion in point clouds with image votes. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4404–4413 (2020)

    2020

  37. [37]

    In: proceedings of the IEEE/CVF International Conference on Computer Vision

    Qi, C.R., Litany, O., He, K., Guibas, L.J.: Deep hough voting for 3d object detec- tion in point clouds. In: proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9277–9286 (2019)

    2019

  38. [38]

    arXiv preprint arXiv:2501.01428 (2025)

    Qi, Z., Zhang, Z., Fang, Y., Wang, J., Zhao, H.: Gpt4scene: Understand 3d scenes from videos with vision-language models. arXiv preprint arXiv:2501.01428 (2025)

    Pith/arXiv arXiv 2025

  39. [39]

    arXiv preprint arXiv:2501.15830 (2025)

    Qu, D., Song, H., Chen, Q., Yao, Y., Ye, X., Ding, Y., Wang, Z., Gu, J., Zhao, B., Wang, D., et al.: Spatialvla: Exploring spatial representations for visual-language- action model. arXiv preprint arXiv:2501.15830 (2025)

    Pith/arXiv arXiv 2025

  40. [40]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

    2021

  41. [41]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4104–4113 (2016)

    2016

  42. [42]

    arXiv preprint arXiv:2509.02560 (2025) 18 J

    Shen, Y., Zhang, Z., Qu, Y., Zheng, X., Ji, J., Zhang, S., Cao, L.: Fastvggt: Training-free acceleration of visual geometry transformer. arXiv preprint arXiv:2509.02560 (2025) 18 J. Zhang et al

    Pith/arXiv arXiv 2025

  43. [43]

    arXiv preprint arXiv:2505.23044 (2025)

    Sheng, Y., Deng, J., Zhang, X., Zhang, Y., Hua, B., Zhang, Y., Ji, J.: Spatialsplat: Efficient semantic 3d from sparse unposed images. arXiv preprint arXiv:2505.23044 (2025)

    arXiv 2025

  44. [44]

    arXiv preprint arXiv:2403.05530 (2024)

    Team, G., Georgiev, P., Lei, V.I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vin- cent,D.,Pan,Z.,Wang,S.,etal.:Gemini1.5:Unlockingmultimodalunderstanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 (2024)

    Pith/arXiv arXiv 2024

  45. [45]

    arXiv preprint arXiv:2511.18416 (2025)

    Wang, H., Zhou, H., Liu, H., Yan, L.: 4d-vggt: A general foundation model with spatiotemporal awareness for dynamic scene geometry estimation. arXiv preprint arXiv:2511.18416 (2025)

    arXiv 2025

  46. [46]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., Novotny, D.: Vggt: Visual geometry grounded transformer. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 5294–5306 (2025)

    2025

  47. [47]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Wang, Q., Zhang, Y., Holynski, A., Efros, A.A., Kanazawa, A.: Continuous 3d perception model with persistent state. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 10510–10522 (2025)

    2025

  48. [48]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wang, S., Leroy, V., Cabon, Y., Chidlovskii, B., Revaud, J.: Dust3r: Geometric 3d vision made easy. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20697–20709 (2024)

    2024

  49. [49]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wang, T., Mao, X., Zhu, C., Xu, R., Lyu, R., Li, P., Chen, X., Zhang, W., Chen, K., Xue, T., et al.: Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19757–19767 (2024)

    2024

  50. [50]

    arXiv preprint arXiv:2507.13347 (2025)

    Wang, Y., Zhou, J., Zhu, H., Chang, W., Zhou, Y., Li, Z., Chen, J., Pang, J., Shen, C., He, T.:π 3: Permutation-equivariant visual geometry learning. arXiv preprint arXiv:2507.13347 (2025)

    Pith/arXiv arXiv 2025

  51. [51]

    arXiv preprint arXiv:2505.23747 (2025)

    Wu, D., Liu, F., Hung, Y.H., Duan, Y.: Spatial-mllm: Boosting mllm capabilities in visual-based spatial intelligence. arXiv preprint arXiv:2505.23747 (2025)

    Pith/arXiv arXiv 2025

  52. [52]

    arXiv preprint arXiv:2508.11952 (2025)

    Xu, Y., Zhang, J., Huang, Z., Chen, Y., Zhou, Y., Chen, Z., Yuan, Y.J., Xia, P., Huang, G., Cai, X., et al.: Uniugg: Unified 3d understanding and generation via geometric-semantic encoding. arXiv preprint arXiv:2508.11952 (2025)

    arXiv 2025

  53. [53]

    In: Pro- ceedings of the Computer Vision and Pattern Recognition Conference

    Yang, J., Yang, S., Gupta, A.W., Han, R., Fei-Fei, L., Xie, S.: Thinking in space: How multimodal large language models see, remember, and recall spaces. In: Pro- ceedings of the Computer Vision and Pattern Recognition Conference. pp. 10632– 10643 (2025)

    2025

  54. [54]

    arXiv preprint arXiv:2511.05491 (2025)

    Yang, R., Zhu, Z., Li, Y., Huang, J., Yan, S., Zhou, S., Liu, Z., Li, X., Li, S., Wang, W., et al.: Visual spatial tuning. arXiv preprint arXiv:2511.05491 (2025)

    arXiv 2025

  55. [55]

    arXiv preprint arXiv:2601.02281 (2026)

    Yuan, S., Yang, Y., Yang, X., Zhang, X., Zhao, Z., Zhang, L., Zhang, Z.: In- finitevggt: Visual geometry grounded transformer for endless streams. arXiv preprint arXiv:2601.02281 (2026)

    arXiv 2026

  56. [56]

    arXiv preprint arXiv:2503.22976 (2025)

    Zhang, J., Chen, Y., Zhou, Y., Xu, Y., Huang, Z., Mei, J., Chen, J., Yuan, Y.J., Cai, X., Huang, G., et al.: From flatland to space: Teaching vision-language models to perceive and reason in 3d. arXiv preprint arXiv:2503.22976 (2025)

    arXiv 2025

  57. [57]

    arXiv preprint arXiv:2505.24625 (2025)

    Zheng, D., Huang, S., Li, Y., Wang, L.: Learning from videos for 3d world: En- hancing mllms with 3d vision geometry priors. arXiv preprint arXiv:2505.24625 (2025)

    arXiv 2025

  58. [58]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Zheng, D., Huang, S., Wang, L.: Video-3d llm: Learning position-aware video rep- resentation for 3d scene understanding. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 8995–9006 (2025)

    2025

  59. [59]

    arXiv preprint arXiv:2510.25760 (2025) GAP-MLLM 19

    Zheng, X., Dongfang, Z., Jiang, L., Zheng, B., Guo, Y., Zhang, Z., Albanese, G., Yang, R., Ma, M., Zhang, Z., et al.: Multimodal spatial reasoning in the large model era: A survey and benchmarks. arXiv preprint arXiv:2510.25760 (2025) GAP-MLLM 19

    arXiv 2025

  60. [60]

    arXiv preprint arXiv:2409.18125 (2024)

    Zhu, C., Wang, T., Zhang, W., Pang, J., Liu, X.: Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness. arXiv preprint arXiv:2409.18125 (2024)

    Pith/arXiv arXiv 2024

  61. [61]

    label"and the point’s 3D coordinate in

    Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., et al.: Rt-2: Vision-language-action models transfer web knowledge to robotic control. In: Conference on Robot Learning. pp. 2165–2183. PMLR (2023) 20 J. Zhang et al. GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimo...

    2023