Pith. sign in

REVIEW 2 major objections 5 minor 29 references

Fusing several foundation-model visual priors at single-model cost beats any one prior for MLLM spatial understanding.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 00:16 UTC pith:EDITCILO

load-bearing objection Solid systems paper with honest ablations; the VSI-Bench SOTA needs a scene-disjointness statement before you trust the 6.6-point win. the 2 major comments →

arxiv 2607.15054 v1 pith:EDITCILO submitted 2026-07-16 cs.CV

Beyond Single Expert: Harmonizing Diverse Visual Priors in MLLMs for Spatial Understanding

Xiao Lin , Xiaohu Huang , Kai Han This is my paper
classification cs.CV
keywords multimodal large language modelsspatial understandingvisual priorsfoundation modelsdynamic prior fusionefficient prior proxy3D visual groundingVSI-Bench
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ViPS is built on a measured observation: no individual visual foundation model is best at every spatial task—geometry, depth, motion, and generic visual experts carry different strengths. The paper's claim is that a multimodal LLM can capture all those strengths at once by distilling each expert's features into lightweight proxies over one shared backbone, then fusing the proxies with weights read from the question itself. Across VSI-Bench and five ScanNet-series benchmarks, that combination outperforms every single-expert prior and prior spatial MLLMs, including 63.8% versus 57.2% on VSI-Bench. The practical stake is that spatial reasoning does not require choosing a single winning expert; a model can be taught to select the relevant prior per query at nearly the inference cost of one expert.

Core claim

The paper first establishes complementarity empirically: across object counting, distance/size estimation, relative relations, route planning, 3D visual grounding, captioning, and question answering, the best foundation-model prior changes from task to task—the geometric, depth, motion-tracking, video-generation, and generic-visual models each win somewhere. Its central proposal is ViPS, which starts from one base encoder and K tiny MLP proxies, each trained with an L2 loss to reproduce the actual features of a different foundation model, so K priors cost roughly one forward pass instead of K. A Dynamic Prior Fusion module then takes the final token of the question, computes K softmax weight

What carries the argument

Efficient Prior Proxy—one base encoder (VGGT by default) plus K lightweight MLPs trained with L2 alignment against the real feature outputs of K distinct foundation models, letting the model produce K priors at roughly the cost of one. Dynamic Prior Fusion—the last question token is projected through an MLP to K softmax weights; each proxy branch first passes through a zero-initialized convolution; the weighted sum is added element-wise to projected image tokens at five layers. The zero-initialized convolutions make injection progressive: at initialization the added signal is exactly zero, so heterogeneous prior distributions do not destabilize pretrained representations.

Load-bearing premise

The reported VSI-Bench gains assume the evaluation videos come from indoor scenes disjoint from the ScanNet/ScanNet++ scenes used in training; if evaluation and training scenes overlap, part of the 63.8% edge over VLM-3R could be memorization.

What would settle it

Run VSI-Bench with a strict scene-level split: train on ScanNet/ScanNet++ scans and evaluate only on videos from buildings never seen in any training stage; if the average drops from 63.8% toward VLM-3R's 57.2%, the headline gain is largely training-set overlap.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Multi-prior harmonization sets new best results on VSI-Bench and on the ScanNet-series grounding, captioning, and QA benchmarks.
  • Adding more expert priors costs only another lightweight proxy, not another full foundation-model forward pass.
  • The fusion weights adapt per task and per question type, so the model can favor a geometric prior for route planning and a generic visual prior for counting.
  • Zero-initialized injection is load-bearing: randomly initialized convolutions or simple averaged addition both degrade performance.
  • No single foundation model dominates, so future spatial MLLMs should treat prior selection as a per-query decision.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A strict scene-level split between ViPS's ScanNet/ScanNet++ training data and VSI-Bench evaluation would determine whether the 63.8% average reflects transferable spatial reasoning or memorization of familiar scan environments.
  • The ablation showing that dropping the alignment loss still helps suggests the proxies may act as diverse learned projections of the shared base; if so, the source models' true specializations may matter less than the number and parameterization of the proxy branches.
  • The same proxy-plus-dynamic-weight recipe could transfer to other modalities or to per-layer fusion weights, rather than the fixed uniform five-layer injection used here.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes ViPS, a framework that injects multiple foundation-model visual priors into an MLLM for spatial understanding. ViPS uses an Efficient Prior Proxy, which shares a single base encoder and learns lightweight MLPs to approximate the prior features of five foundation models, supervised by an L2 alignment loss against the ground-truth prior features (Eq. 4). It then applies Dynamic Prior Fusion: query-dependent weights (Eq. 5), zero-initialized convolutions per branch, and a weighted sum (Eq. 7) that is added to the MLLM's visual tokens across five layers. Experiments are reported on VSI-Bench and five ScanNet-series benchmarks, with average VSI-Bench accuracy 63.8% versus 57.2% for VLM-3R and top ScanRefer / ScanQA scores of 64.6% Acc@0.25 and 107.9 CIDEr. Ablations show that each prior contributes, that zero-init and dynamic weighting help, and that the proxy performs close to the ground-truth-prior upper bound.

Significance. If the results hold, ViPS provides a useful and practical recipe for combining complementary visual priors from multiple foundation models without paying the inference cost of running all models. The paper's strengths include a clear motivation supported by an empirical study, a sensible proxy-then-fuse design, ablations for each main component, a comparison against an upper bound with ground-truth priors, and a robustness check across different base models. The central claim is not forced by construction: the proxy outputs are supervised by external foundation-model features, and the w/GT-Priors ablation anchors the proxy against an external upper bound. However, the headline SOTA claim on VSI-Bench depends on an unverified scene-level train/test split, and the reported margins on several ScanNet-series benchmarks are small and shown without error bars or released artifacts.

major comments (2)
  1. [Appendix A.2 / Table 1] The VSI-Bench SOTA claim (Table 1: 63.8% average vs. 57.2% for VLM-3R) is based on training on 207,658 QA pairs from ScanNet++ and ScanNet listed in Appendix A.2. The paper never states that these training scenes are disjoint from the VSI-Bench test videos. Since VSI-Bench is drawn from the same indoor-scan ecosystem, this is a load-bearing external validity condition: if test scenes overlap with the 135,119 ScanNet++ or 51,630 ScanNet training pairs, the 6.6-point margin could be inflated by memorization of specific layouts and objects. Please state explicitly whether the training scenes are excluded from VSI-Bench test, cite the official split or VLM-3R's composition if it guarantees disjointness, or report a scene-disjoint evaluation.
  2. [Section 3.3 / Tables 1-5] All reported results appear to be from a single run with no error bars, and no code or checkpoints are provided. This is especially important because several ScanNet-series margins are small: ScanRefer 64.6 vs. 63.2 for Vega-3D, ScanQA CIDEr 107.9 vs. 106.3, and Scan2Cap trailing 3DRS. Without multiple seeds or released artifacts, the reader cannot assess whether these differences are significant or reproducible. Please report mean and standard deviation over at least three seeds for the main tables, or justify why this is infeasible, and release code/checkpoints to support the SOTA claims.
minor comments (5)
  1. [Appendix A.3] The empirical study in Section 2.1 is run on a 10% subset of the training data. The rankings in Figure 3 may not reflect full-data behavior; this qualification should be stated where the key finding is introduced.
  2. [Table 1] Some rows in Table 1 have missing entries or misaligned cells (e.g., VLM-3R's Appearance Order value and several '–' entries). Please ensure all numbers align with their column headers and that missing values are explicitly marked.
  3. [Table 5] The 'w/o L_alignment' row contains a formatting artifact: '1x 1x-' in the Efficiency columns. Also, the alignment loss in Eq. 4 is L2, while the reported proxy error is cosine distance; clarify how the cosine error relates to the training loss.
  4. [Section 2.2] The text says the dynamic weights are derived from the 'final token of the input instruction' and applied at five layers. It would be helpful to specify whether this is the token embedding before the LLM, the hidden state at each injection layer, or a shared representation, since this affects the claimed context-awareness.
  5. [Conclusion] The wording 'new state-of-the-art performance' is used in several places despite the small margins and absent error bars. Consider softening these statements or explicitly tying them to the reproducibility conditions.

Circularity Check

0 steps flagged

No substantive circularity; central prior-fusion result is externally supervised; score 2 for minor self-citation only.

full rationale

ViPS's derivation is not circular. The Efficient Prior Proxy is trained with Eq. (4), an L2 loss against features F^k_gt = E_k(V) extracted by five external foundation models (VGGT, DepthAnything3, TraceAnything, Wan2.1, RADIO); these targets are independent of the downstream benchmarks, and Table 5's w/GT Priors row is an external upper bound rather than a fitted prediction. The Dynamic Prior Fusion (Eqs. 5–7) is a learned gating function of the query token and is ablated in Table 4, so the claimed benefit of context-aware weighting is an empirical result, not a tautology. The complementary-prior finding (Figure 3) is produced by independently injecting each external prior and comparing on public benchmarks; it is not read off from the fusion weights. The only self-citation of note is the 3DRS training recipe used in Appendix A.1/A.3 ('our training protocol aligns with the configurations established by previous works such as Video-3D LLM and 3DRS'), but this sets the data mix and is not load-bearing for the central claim: the SOTA numbers in Tables 1–2 are measured against external published methods. Appendix F's limitation ('extracting spatial awareness from external visual priors is ultimately a sub-optimal workaround') concerns scalability and is not an admission of circularity. The VSI-Bench concern—that Appendix A.2 lists 207,658 ScanNet/ScanNet++ training pairs without demonstrating scene-level disjointness from VSI-Bench test videos—is a data-leakage/correctness risk external to the derivation, not a circularity. Overall: no self-definition, no fitted-input-called-prediction, no imported uniqueness theorem, no ansatz-by-citation, and no renaming; score 2 only for the minor non-load-bearing self-citation.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The framework rests on several domain assumptions that are partially tested but not proven: one base encoder can carry all prior information through MLP proxies, L2 feature alignment preserves semantics, the last query token encodes task context, and zero-init convs provide stable injection. No new physical entities are invented; the two architectural components are named and ablated. Main free choices are K=5 priors, five injection layers, and LoRA capacity.

free parameters (4)
  • K = number of integrated prior models = 5 (VGGT, DepthAnything3, TraceAnything, Wan2.1, RADIO)
    Hand-chosen set of foundation models; no selection criterion or ablation over other foundation models.
  • Injection layers for Dynamic Prior Fusion = 5 LLM layers
    Uniformly applied across five layers; no ablation over layer count or locations.
  • LoRA rank/alpha = rank 512/alpha 1024 (Qwen2-VL); rank 256/alpha 256 (Qwen3-VL)
    Adapter capacity chosen without reported sensitivity analysis.
  • Proxy MLP architecture
    Paper describes a 'simple MLP' without widths/depths; capacity directly controls fidelity of prior estimation.
axioms (5)
  • domain assumption One robust base encoder carries sufficient shared low/mid-level visual structure to reconstruct all K diverse priors with lightweight MLPs.
    Sec. 2.2, Efficient Prior Proxy: 'different foundation models often share common low- and mid-level visual and geometric features.' Load-bearing for the efficiency claim.
  • domain assumption L2 distance between proxy outputs and ground-truth prior features is an adequate fidelity measure.
    Eq. (4) uses L2 alignment; if feature distances do not correspond to semantic prior differences, the alignment loss is misdirected.
  • domain assumption The final token of the text instruction summarizes the task context needed to choose priors.
    Sec. 2.2, Eq. (5): 'final token … naturally encapsulates the aggregated semantics of the entire instruction due to the causal mechanism of the LLM.'
  • ad hoc to paper Zero-initialized convolutions prevent early-training disruption without limiting final expressiveness.
    Sec. 3.4 ablation 'w/o Zero-init' supports this empirically; no theoretical guarantee is given.
  • domain assumption VSI-Bench test scenes are disjoint from the ScanNet/ScanNet++ training data used in Appendix A.2.
    No explicit scene-level split statement; if violated, Table 1 SOTA could be inflated by memorization.
invented entities (2)
  • Efficient Prior Proxy independent evidence
    purpose: Estimates K foundation-model priors from one base encoder via lightweight MLPs, avoiding K independent forward passes during inference.
    Falsifiable via proxy-vs-GT cosine error (0.252) and the w/GT-priors upper-bound ablation in Table 5.
  • Dynamic Prior Fusion independent evidence
    purpose: Computes query-conditioned softmax weights over zero-init-conv-transformed priors and injects the weighted sum into five LLM layers.
    Falsifiable via ablations: 'w/o Zero-init' and 'Vanilla Addition' degrade specific benchmark metrics (Tables 4–5).

pith-pipeline@v1.3.0-alltime-deepseek · 16345 in / 15247 out tokens · 158211 ms · 2026-08-02T00:16:38.961401+00:00 · methodology

0 comments
read the original abstract

Multimodal Large Language Models (MLLMs) have demonstrated substantial promise in spatial understanding. Existing works typically incorporate prior knowledge extracted from a pre-trained foundation model to further enhance the spatial awareness of MLLMs. In this paper, we first reveal that when integrating diverse foundation models into MLLMs, different models provide complementary spatial priors that benefit different tasks. Motivated by this, we propose $\textbf{ViPS}$, a novel multi-model prior framework designed to fully unleash the potential of incorporating multiple $\textbf{Vi}$sual $\textbf{P}$riors from diverse models into MLLMs for $\textbf{S}$patial understanding. Specifically, ViPS introduces an Efficient Prior Proxy to generate multiple foundational priors with minimal inference overhead, and a Dynamic Prior Fusion mechanism to achieve harmonious and context-aware prior fusion and injection from the prior proxies. Extensive experiments demonstrate that ViPS successfully harmonizes diverse visual priors, establishing new state-of-the-art performance across multiple complex spatial reasoning and 3D spatial understanding benchmarks. Project page: https://visual-ai.github.io/vips

Figures

Figures reproduced from arXiv: 2607.15054 by Kai Han, Xiaohu Huang, Xiao Lin.

Figure 1
Figure 1. Figure 1: Comparison of Existing Single-Expert Paradigms and Our Multi-Prior Framework. Top-left: Existing paradigms typically rely on a single external encoder (e.g., VGGT) to provide visual priors for MLLMs. Bottom-left: In contrast, our approach integrates diverse knowledge from multiple expert models into the MLLM. Middle and right: Extensive evaluations demonstrate that our method achieves state-of-the-art perf… view at source ↗
Figure 2
Figure 2. Figure 2: Prior Analysis of Diverse Foundation Models. Left: The t-SNE of visual prior features extracted from different foundation models. Right: Spatial heatmaps for different models 2 Method 2.1 Empirical Study on Diverse Model Priors Preliminary. Our work focuses on spatial understanding based on MLLMs. Formally, given a visual input comprising a sequence of frames (e.g., a video clip or multi-view images of a s… view at source ↗
Figure 3
Figure 3. Figure 3: Relative Performance of Diverse Foundation Models. We evaluate the performance of various foundation models when serving as priors for MLLMs across a wide range of spatial understanding tasks. Notably, no single model dominates all metrics, thoroughly motivating the need for the multi-model prior integration. 3 [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overview of the Proposed ViPS Framework. The framework integrates distinct prior knowledge from multiple foundation models via the Efficient Prior Proxy and coordinates them using Dynamic Prior Fusion for comprehensive spatial reasoning. semantics) from the shared foundational feature Fbase. The representation for the k-th prior is uniformly formulated as: F k prior = ϕk(Fbase) ∈ R S×Dprior , (3) where Dk … view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of Dynamic Prior Weights. This figure illustrates the distribution of the learned fusion weights (wk in Equation 7) assigned to different foundation models. Left: The weight distribution on the test sets of ScanQA, ScanRefer, Multi3DRefer, SQA3D, and Scan2Cap. Right: The weight distribution across different question types in VSI-Bench. predominantly focused on: (i) more efficient mechanisms fo… view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative Visualization on VSI-Bench. We compare our ViPS framework against the baseline, which is obtained by fine-tuning Qwen2-VL on our identical training dataset, showing that ViPS produces more accurate responses to complex spatial questions. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 23 linked inside Pith

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [4]

    Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning

    Sijin Chen, Xin Chen, Chi Zhang, et al. Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. InCVPR, 2024a. Yilun Chen, Shuai Yang, Haifeng Huang, et al. Grounded 3d-llm with referent tokens.arXiv preprint arXiv:2405.10370, 2024b. Yukang Chen, Fuzhao Xue, Dacheng Li, et al. Longvila: Scaling long-context visual la...

    Pith/arXiv arXiv

  3. [6]

    Scene-llm: Extending language model for 3d visual understanding and reasoning.arXiv preprint arXiv:2403.11401,

    Rao Fu, Jingyu Liu, Xilun Chen, et al. Scene-llm: Extending language model for 3d visual understanding and reasoning.arXiv preprint arXiv:2403.11401,

    Pith/arXiv arXiv

  4. [8]

    Chat-3d v2: Bridging 3d scene and large language models with object identifiers.arXiv preprint arXiv:2312.08168,

    Haifeng Huang, Zehan Wang, Rongjie Huang, et al. Chat-3d v2: Bridging 3d scene and large language models with object identifiers.arXiv preprint arXiv:2312.08168,

    Pith/arXiv arXiv

  5. [10]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    Aaron Hurst, Adam Lerer, Adam P Goucher, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    Pith/arXiv arXiv

  6. [11]

    Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980,

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980,

    Pith/arXiv arXiv

  7. [13]

    Thinking with geometry: Active geometry integration for spatial reasoning.arXiv preprint arXiv:2602.06037,

    Haoyuan Li, Qihang Cao, Tao Tang, et al. Thinking with geometry: Active geometry integration for spatial reasoning.arXiv preprint arXiv:2602.06037,

    Pith/arXiv arXiv

  8. [15]

    Depth anything 3: Recovering the visual space from any views

    Haotong Lin, Sili Chen, Junhao Liew, et al. Depth anything 3: Recovering the visual space from any views. arXiv preprint arXiv:2511.10647,

    Pith/arXiv arXiv

  9. [16]

    Trace anything: Representing any video in 4d via trajectory fields.arXiv preprint arXiv:2510.13802, 2025a

    Xinhang Liu, Yuxi Xiao, Donny Y Chen, et al. Trace anything: Representing any video in 4d via trajectory fields.arXiv preprint arXiv:2510.13802, 2025a. Zhijian Liu, Ligeng Zhu, Baifeng Shi, et al. Nvila: Efficient frontier visual language models. InCVPR, 2025b. Xiaojian Ma, Silong Yong, Zilong Zheng, et al. Sqa3d: Situated question answering in 3d scenes....

    arXiv

  10. [17]

    Spacer: Reinforcing mllms in video spatial reasoning.arXiv preprint arXiv:2504.01805,

    Kun Ouyang, Yuanxin Liu, Haoning Wu, et al. Spacer: Reinforcing mllms in video spatial reasoning.arXiv preprint arXiv:2504.01805,

    Pith/arXiv arXiv

  11. [18]

    Gpt4scene: Understand 3d scenes from videos with vision- language models.arXiv preprint arXiv:2501.01428,

    Zhangyang Qi, Zhixiong Zhang, Ye Fang, et al. Gpt4scene: Understand 3d scenes from videos with vision- language models.arXiv preprint arXiv:2501.01428,

    Pith/arXiv arXiv

  12. [19]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530,

    Gemini Team, Petko Georgiev, Ving Ian Lei, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530,

    Pith/arXiv arXiv

  13. [20]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,

    Pith/arXiv arXiv

  14. [21]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314,

    Team Wan, Ang Wang, Baole Ai, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314,

    Pith/arXiv arXiv

  15. [22]

    Ross3d: Reconstructive visual instruction tuning with 3d-awareness

    Haochen Wang, Yucheng Zhao, Tiancai Wang, et al. Ross3d: Reconstructive visual instruction tuning with 3d-awareness. InICCV, 2025a. Jianyuan Wang, Minghao Chen, Nikita Karaev, et al. Vggt: Visual geometry grounded transformer. InCVPR, 2025b. Peng Wang, Shuai Bai, Sinan Tan, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any ...

    Pith/arXiv arXiv

  16. [23]

    Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes.arXiv preprint arXiv:2308.08769,

    Zehan Wang, Haifeng Huang, Yang Zhao, et al. Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes.arXiv preprint arXiv:2308.08769,

    Pith/arXiv arXiv

  17. [24]

    Generation models know space: Unleashing implicit 3d priors for scene understanding.arXiv preprint arXiv:2603.19235,

    Xianjin Wu, Dingkang Liang, Tianrui Feng, et al. Generation models know space: Unleashing implicit 3d priors for scene understanding.arXiv preprint arXiv:2603.19235,

    Pith/arXiv arXiv

  18. [25]

    From flatland to space: Teaching vision-language models to perceive and reason in 3d.arXiv preprint arXiv:2503.22976,

    Jiahui Zhang, Yurui Chen, Yanpeng Zhou, et al. From flatland to space: Teaching vision-language models to perceive and reason in 3d.arXiv preprint arXiv:2503.22976,

    arXiv

  19. [26]

    Spatialstack: Layered geometry-language fusion for 3d vlm spatial reasoning.arXiv preprint arXiv:2603.27437,

    12 Jiang Zhang, Shijie Zhou, Bangya Liu, et al. Spatialstack: Layered geometry-language fusion for 3d vlm spatial reasoning.arXiv preprint arXiv:2603.27437,

    Pith/arXiv arXiv

  20. [27]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. InICCV, 2023a. Peiyuan Zhang, Kaichen Zhang, Bo Li, et al. Long context transfer from language to vision.arXiv preprint arXiv:2406.16852,

    Pith/arXiv arXiv

  21. [28]

    Multi3drefer: Grounding text description to multiple 3d objects

    Yiming Zhang, ZeMing Gong, and Angel X Chang. Multi3drefer: Grounding text description to multiple 3d objects. InICCV, 2023b. Duo Zheng, Shijia Huang, Yanyang Li, and Liwei Wang. Learning from videos for 3d world: Enhancing mllms with 3d vision geometry priors.arXiv preprint arXiv:2505.24625, 2025a. Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d llm: L...

    arXiv

  22. [29]

    Unifying 3d vision-language understanding via promptable queries

    Ziyu Zhu, Zhuofan Zhang, Xiaojian Ma, et al. Unifying 3d vision-language understanding via promptable queries. InECCV, 2024b. 13 Appendix A Detailed Training Dataset Description A.1 ScanNet-series Dataset For the 3D spatial understanding tasks, our training protocol aligns with the configurations established by previous works such as Video-3D LLM and 3DRS...

    2025

  23. [2014]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326,

    Bo Li, Yuanhan Zhang, Dong Guo, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326,

    Pith/arXiv arXiv

  24. [2020]

    Seeing through imagination: Learning scene geometry via implicit spatial world modeling.arXiv preprint arXiv:2512.01821,

    Meng Cao, Haokun Lin, Haoyuan Li, et al. Seeing through imagination: Learning scene geometry via implicit spatial world modeling.arXiv preprint arXiv:2512.01821,

    arXiv

  25. [2022]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631,

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631,

    Pith/arXiv arXiv

  26. [2023]

    3drs: Mllms need 3d-aware representation supervision for scene understanding.arXiv preprint arXiv:2506.01946,

    Xiaohu Huang, Jingjing Wu, Qunyi Xie, and Kai Han. 3drs: Mllms need 3d-aware representation supervision for scene understanding.arXiv preprint arXiv:2506.01946,

    arXiv

  27. [2024]

    Point-bind & point-llm: Aligning point cloud with multi- modality for 3d understanding, generation, and instruction following.arXiv preprint arXiv:2309.00615,

    Ziyu Guo, Renrui Zhang, Xiangyang Zhu, et al. Point-bind & point-llm: Aligning point cloud with multi- modality for 3d understanding, generation, and instruction following.arXiv preprint arXiv:2309.00615,

    Pith/arXiv arXiv

  28. [2025]

    Vlm-3r: Vision-language models augmented with instruction-aligned 3d reconstruction.arXiv preprint arXiv:2505.20279,

    Zhiwen Fan, Jian Zhang, Renjie Li, et al. Vlm-3r: Vision-language models augmented with instruction-aligned 3d reconstruction.arXiv preprint arXiv:2505.20279,

    Pith/arXiv arXiv

  29. [2026]

    Improved visual-spatial reasoning via r1-zero-like training

    11 Zhenyi Liao, Qingsong Xie, Yanhao Zhang, et al. Improved visual-spatial reasoning via r1-zero-like training. arXiv preprint arXiv:2504.00883,

    Pith/arXiv arXiv