Computer Vision and Pattern Recognition · arXiv:2610.24676 v1 · submitted 17 Oct 2026 · 90 references
From Views to Worlds: Active Exploration over 3D Worlds for Vision-Language Models
The paper says
“Vision-language models (VLMs) have achieved remarkable progress in 2D visual understanding tasks, yet the passive view-centric perception paradigm limits their capacity for 3D spatial reasoning.”
more of the abstract
Such reasoning often requires actively acquiring spatial evidence beyond the currently visible views, such as camera poses and cross-view spatial relationships, to infer 3D relationships that are not explicitly represented in 2D images. However, existing methods either lack active exploration or merely provide additional views during inference, without explicitly acquiring such spatial evidence beyond 2D images. In this paper, we propose View-to-World (V2W), an active exploration framework that enables VLMs to actively explore 3D worlds and lifts VLMs from passive view-centric perception to active world-centric spatial reasoning. Given a spatial reasoning task with multi-view images, V2W first reconstructs an explicit 3D world with geometry, camera poses, and language-grounded semantics, and exposes it to VLMs through visual and linguistic interfaces. Through multi-turn interaction with these interfaces, VLMs iteratively acquire spatial evidence from the constructed 3D world, enabling active world-centric spatial reasoning. V2W can be integrated with existing VLMs in a training-free manner and can be further enhanced by optimizing the exploration policy via reinforcement learning. Experiments on spatial mental modeling benchmarks demonstrate that V2W substantially improves VLM baselines in the training-free setting and achieves state-of-the-art performance with the learned exploration policy, highlighting the importance of active exploration for spatial intelligence. Project page: r̆lhttps://fangzhou2000.github.io/projects/v2w.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 20 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
The passes disagreed. Reader: holds up. Skeptic: agrees with the reader. Referee: major revision.
V2W is a well-engineered framework that lifts VLMs from passive 2D viewing to active 3D exploration, with strong empirical results—but the black-box 3D reconstruction pipeline is not validated against ground truth geometry, which tempers confidence in the explanation of why it works. This line is wrong
3 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 90 references say.
The paper says, at §3.2, Appendix A.6
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The paper does not provide quantitative validation of the reconstructed 3D world's geometric accuracy against ground truth. The entire spatial reasoning pipeline depends on the correctness of depth maps, camera poses, and object masks produced by Depth Anything 3 and SAM 3.
more of objection 1
The SAT benchmark is procedurally generated and therefore provides ground-truth camera poses, object positions, and semantics. The authors should report metrics such as mean camera pose error, object localization error, or mask accuracy on a subset of SAT scenes. The ablation with VGGT (Table 5) shows robustness to the choice of 3D model but does not measure absolute accuracy. Without such validation, it is unclear whether reconstruction errors are negligible or systematically degrade spatial reasoning, making this a load-bearing gap.
The paper says, at §4.1 (Checklist §7)
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.93):
“However, due to the substantial experimental cost of VLM training and evaluation, and following common practice in recent VLM research, we do not report multi-seed results, error bars, or formal statistical significance tests.”
Pith says: Objection 2 of 3 · major
The paper acknowledges that it does not report error bars or statistical significance tests. Given that the MindCube-Tiny evaluation set contains only 1,050 questions (with sub-categories as small as 200 questions), single-run accuracy may be sensitive to randomness. The authors should provide either multiple-seed results, bootstrapped confidence intervals, or an analysis of variance across inference runs for the main claims (Tables 1 and 2). This is particularly important for the smaller sub-category results (e.g., Rotation, 200 questions).
The paper says, at §4.2, Table 1 (training-based comparison)
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The training-based V2W is compared to 3DThinker and SpatialDreamer using results taken from their original papers. It is not stated whether all methods were trained on the same data splits of MindCube's 10K training samples. Since the RL training in V2W uses ground-truth answer rewards, the comparison should explicitly confirm that no privileged information (e.g., ground-truth camera poses from the procedural generation process) was used beyond what is available from the reconstructed 3D world. The authors should clarify this.
5 minor objections
The paper says, at Abstract
Pith says: minor objection 1 of 5
The phrase 'training-free manner' could be clarified: no additional fine-tuning of the VLM weights is performed, but the 3D world construction pipeline still requires off-the-shelf foundation models and per-scene preprocessing.
The paper says, at Figure 2
Pith says: minor objection 2 of 5
The figure is dense; consider adding a legend or numbered steps to separate the 3D world construction phase (top) from the multi-turn exploration loop (bottom).
show 3 more
The paper says, at §3.2, Linguistic Interface
Pith says: minor objection 3 of 5
Provide example text outputs for visible object descriptions and camera pose descriptions (e.g., 'The movement from image 1 to image 2 is diagonally forward and right.' is shown in a qualitative figure, but a formal specification in the appendix would help reproducibility).
The paper says, at Table 3, row 11
Pith says: minor objection 4 of 5
The abbreviation 'w/o SR' is used; please spell it out in the caption or add a footnote (e.g., 'SR: step-wise reward').
The paper says, at §2.1
Pith says: minor objection 5 of 5
The distinction between 'VLM' and 'LLM' in the related work section is sometimes blurred; consider using 'VLM' consistently when referring to multimodal models.
What would settle it
For SAT-val (procedurally generated, ground-truth 3D geometry available), re-run V2W (Qwen2.5-VL-7B, training-free) under two conditions: (A) standard reconstructed 3D world from Depth Anything 3 + SAM 3; (B) ground-truth world (perfect camera poses, object positions, semantics from the simulator). Compare overall accuracy and per-category accuracy. If gap ≤2 points, reconstruction is sufficient; if gap >5 points, reconstruction errors are a significant bottleneck and the central claim must be qualified.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 20 Oct 2026
- Reader holds up
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
The paper's core idea is genuinely new: instead of feeding VLMs more 2D views or baking 3D priors into weights, V2W builds an explicit 3D world from sparse multi-view images and lets the VLM actively query it through visual renders and textual spatial cues. That is a clean architectural contribution. The engineering is solid—Depth Anything 3 + SAM 3 + a lightweight VLM for object naming, exposed through tool calls. The results are striking: training-free gains of 10–20 points across several VLMs on MindCube-Tiny, and 90.8% with RL training, beating dedicated spatial models and GPT-4o by a wide margin.
What the paper does well: it tests across multiple base VLMs (including GPT-4.1-mini), ablates visual vs. linguistic interfaces, compares against CoT prompting to rule out trivial explanations, and includes a robustness experiment swapping the 3D reconstruction model (Depth Anything 3 → VGGT). The RL training details are sensible, with step-wise rewards to shape exploration. The writing is clear and the limitations are honestly acknowledged in Appendix A.6.
The main soft spot is one the authors themselves flag but do not fully address: the 3D world is constructed by off-the-shelf models whose geometric accuracy is never quantitatively validated. The stress-test note is right—SAT is procedurally generated and ground-truth camera poses, object positions, and semantics are available. Reporting a reconstruction error metric (e.g., mean Chamfer distance on object centroids, pose error vs. ground truth) would directly answer whether spatial reasoning improvements come from geometric fidelity or from the linguistic interface's coarse cues being robust to noisy geometry. The VGGT ablation shows the framework survives model swaps but does not measure absolute accuracy. Without this validation, it is hard to know how the pipeline degrades under domain shift (low light, reflective surfaces).
A secondary concern: no error bars or multi-seed runs are reported. The authors cite cost, which is fair, but the main result tables present single numbers. Given the margins (e.g., 90.8% vs. 84.9%), the headline claims would likely hold, but a reader cannot assess variance.
Overall, this is a useful paper for the 3D vision and spatial reasoning communities. The framework is modular and the interface design may inspire follow-up work. I would bring it to a reading group and cite it. For peer review, it deserves a serious referee: the central claim is supported, the novelty is real, and the soft spots are fixable rather than fatal.
The strongest objection
The V2W framework's effectiveness hinges on the quality of the 3D world constructed by Depth Anything 3 (monocular depth + pose) and SAM 3 (segmentation). The paper acknowledges in Appendix A.6 that these models may fail under challenging conditions (low light, reflective objects), leading to inaccurate reconstructions. However, it provides no quantitative validation of reconstruction accuracy against ground-truth geometry, even though the SAT benchmark is procedurally generated and thus ground-truth 3D information (camera poses, object positions, semantics) is available. Without such validation, it is unknown whether reconstruction errors are negligible or systematically degrade spatial reasoning. The paper's ablation with a different 3D foundation model (VGGT) shows robustness across models, but this does not measure absolute accuracy. If reconstruction errors are significant, the reported gains may partly stem from the linguistic interface's coarse spatial cues (which are robust to moderate errors) rather than from geometrically reliable active exploration. The concern is load-bearing because if the 3D world is inaccurate, the visual and linguistic interfaces could provide misleading evidence, undermining the central claim that active exploration over explicit 3D worlds drives spatial reasoning improvement.
The reader identified the 3D world reconstruction accuracy as the weakest assumption, which I agree with. I narrow this to a more specific, testable concern: the lack of ground-truth validation despite the availability of ground-truth geometry for SAT. The proposed concrete test would directly quantify the impact of reconstruction errors on the reported results. If the gap is small, it strengthens the claim; if large, it reveals a critical limitation. The paper's own admission of reconstruction failures makes this an unresolved empirical question. The test is feasible and does not require retraining or new data--only the use of ground-truth already inherent in the SAT procedural generation. The paper's overall contribution remains strong; thus the reader's ACCEPT verdict is appropriate, but this validation would further solidify it.
The referee's report
If the reported results hold, V2W represents a significant step toward enabling VLMs to perform world-centric spatial reasoning rather than passive view-centric perception. The framework is model-agnostic, can be applied without training, and the learned exploration policy yields large absolute gains (e.g., 90.8% vs. 34.5% baseline on MindCube-Tiny). The paper provides extensive ablation studies isolating the contributions of visual and linguistic interfaces, and demonstrates robustness across different base VLMs and 3D foundation models. The approach is well-motivated and the experimental design is thorough, including both training-free and training-based evaluations.
Pith's reading of the paper (a machine's summary, not the authors')
The paper argues that vision-language models (VLMs) fail at spatial reasoning because they passively receive 2D images, which lack explicit 3D geometry and cross-view relationships. The proposed V2W framework first reconstructs an explicit 3D world from sparse multi-view images using off-the-shelf foundation models (Depth Anything 3 for geometry, SAM 3 for semantics), then exposes it through a visual interface (novel-view renders) and a linguistic interface (textual cues for visible objects and camera poses). During inference, the VLM actively explores this world by calling functions (turn, move, zoom, query) over multiple turns, progressively collecting spatial evidence. The framework works training-free with large VLMs and, when the exploration policy is further optimized via supervised fine-tuning and reinforcement learning (GRPO), achieves state-of-the-art accuracy on MindCube-Tiny (90.8%) and SAT-val (95.5%), substantially outperforming both proprietary models and prior specialized methods.
What carries the argument. The VLM-compatible 3D world representation, built from multi-view images by off-the-shelf depth estimation (Depth Anything 3) and segmentation (SAM 3). This world is exposed through a visual interface (rendering novel-view images from queried camera poses) and a linguistic interface (returning text descriptions of visible objects and relative camera poses). The VLM interacts with these interfaces via function calls (turn-left, move-forward, zoom-in, camera-pose-query, visible-object-description) in a multi-turn loop. The exploration policy can be trained by supervised fine-tuning on teacher demonstrations (from Qwen3-VL-235B) and then optimized with Group Relative Policy Optimization (GRPO), using a reward combining trajectory-level (format, answer, exploration) and step-wise (matching to preferred actions, penalty for uninformative actions) signals.
Core claim. The central claim is that spatial reasoning in VLMs can be substantially improved by shifting from passive view-centric perception (in which the model sees only fixed 2D views) to active world-centric reasoning over an explicit 3D world representation. The paper demonstrates that constructing a VLM-compatible 3D world that integrates geometry, camera poses, and language-grounded semantics, and then allowing the VLM to interact with that world through rendered images and textual spatial cues, enables iterative collection of evidence that is not available in the original 2D images. When the exploration policy is further optimized via reinforcement learning, the resulting system markedly outperforms both generalist VLMs (including GPT-4o, by 48.5 percentage points on MindCube-Tiny) and specialized spatial reasoning models (3DThinker, SpatialDreamer), establishing a new state of the art on two benchmarks for spatial mental modeling.
The premise it rests on. The 3D world reconstructed by off-the-shelf depth and segmentation models (Depth Anything 3 and SAM 3) is accurate enough in geometry, camera poses, and semantics to support reliable spatial reasoning.
If the paper is right (Pith's inferences, not the authors')
If other VLMs adopt active world exploration, their spatial reasoning performance should improve, especially on tasks that require inference beyond the visible frame, such as mental rotation, perspective taking, and what-if simulations.
The RL-trained exploration policy suggests that effective strategies for gathering 3D evidence can be learned independently of the VLM's core visual-linguistic reasoning, potentially enabling smaller VLMs to match larger ones when given a good exploration policy.
Decoupling 3D world construction (off-the-shelf) from reasoning (VLM) means that improvements in depth estimation, segmentation, or camera pose estimation directly translate into better spatial reasoning without retraining the VLM.
The training-free gains (9–18 percentage points on various base models) show that VLMs, including proprietary models, already have a latent capacity for active exploration when provided with appropriate interfaces, suggesting a general mechanism for enhancing spatial intelligence.
The paradigm extends beyond static scene understanding to embodied tasks such as robotic navigation, manipulation, and goal-oriented exploration, where active evidence gathering is essential.
Because the 3D reconstruction relies entirely on the accuracy of Depth Anything 3 and SAM 3, failures in low-light, reflective, or visually homogeneous scenes could produce a misleading world and cause the VLM to arrive at wrong conclusions. The paper acknowledges this limitation and proposes a VLM-based verifier as future work.
The multi-turn function-calling loop is essentially an agentic interaction. Future work could extend it beyond greedy per-turn decisions to include memory of past explorations, explicit planning (e.g., graph search over viewpoints), or risk-aware exploration under reconstruction uncertainty.
The large gain over GPT-4o (48.5 points on MindCube-Tiny) suggests that even the strongest generalist VLMs are bottlenecked not by reasoning capability but by access to structured 3D information. Simply giving them more 2D views (as done by CoT or additional images) is insufficient; they need explicit geometric and spatial cues.
An explicit 3D world opens the door to hybrid systems that combine VLM reasoning with classical computational geometry: for example, the VLM could query exact distances between objects or execute path-planning algorithms on the reconstructed mesh, enabling precise quantitative spatial reasoning.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Vision-language models (VLMs) have achieved remarkable progress in 2D visual understanding tasks, yet the passive view-centric perception paradigm limits their capacity for 3D spatial reasoning. Such reasoning often requires actively acquiring spatial evidence beyond the currently visible views, such as camera poses and cross-view spatial relationships, to infer 3D relationships that are not explicitly represented in 2D images. However, existing methods either lack active exploration or merely provide additional views during inference, without explicitly acquiring such spatial evidence beyond 2D images. In this paper, we propose View-to-World (V2W), an active exploration framework that enables VLMs to actively explore 3D worlds and lifts VLMs from passive view-centric perception to active world-centric spatial reasoning. Given a spatial reasoning task with multi-view images, V2W first reconstructs an explicit 3D world with geometry, camera poses, and language-grounded semantics, and exposes it to VLMs through visual and linguistic interfaces. Through multi-turn interaction with these interfaces, VLMs iteratively acquire spatial evidence from the constructed 3D world, enabling active world-centric spatial reasoning. V2W can be integrated with existing VLMs in a training-free manner and can be further enhanced by optimizing the exploration policy via reinforcement learning. Experiments on spatial mental modeling benchmarks demonstrate that V2W substantially improves VLM baselines in the training-free setting and achieves state-of-the-art performance with the learned exploration policy, highlighting the importance of active exploration for spatial intelligence. Project page: r̆lhttps://fangzhou2000.github.io/projects/v2w.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-20
The authors' main claim, as the classifier picked it out 0.99
“Experiments on spatial mental modeling benchmarks demonstrate that V2W substantially improves VLM baselines in the training-free setting and achieves state-of-the-art performance with the learned exploration policy, highlighting the importance of active exploration for spatial intelligence.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.14 · states limitation 0.10 · human participants 0.03 · controlled comparison 0.84 · randomized assignment 0.02 · states quantitative result 0.05 · compares with prior results 0.93 · mentions code or data availability 0.35
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Vision-language models (VLMs) have achieved remarkable progress in 2D visual understanding tasks, yet the passive view-centric perception paradigm limits their capacity for 3D spatial reasoning. | background 0.96 | 0.06 | 0.03 | asserted 1.00 |
| Such reasoning often requires actively acquiring spatial evidence beyond the currently visible views, such as camera poses and cross-view spatial relationships, to infer 3D relationships that are not explicitly represented in 2D images. | background 0.99 | 0.07 | 0.04 | asserted 0.62 |
| However, existing methods either lack active exploration or merely provide additional views during inference, without explicitly acquiring such spatial evidence beyond 2D images. | gap 0.99 | 0.04 | 0.03 | asserted 1.00 |
| In this paper, we propose View-to-World (V2W), an active exploration framework that enables VLMs to actively explore 3D worlds and lifts VLMs from passive view-centric perception to active world-centric spatial reasoning. | contribution 1.00 | 0.87 | 0.20 | asserted 1.00 |
| Given a spatial reasoning task with multi-view images, V2W first reconstructs an explicit 3D world with geometry, camera poses, and language-grounded semantics, and exposes it to VLMs through visual and linguistic interfaces. | method 0.98 | 0.83 | 0.05 | asserted 1.00 |
| Through multi-turn interaction with these interfaces, VLMs iteratively acquire spatial evidence from the constructed 3D world, enabling active world-centric spatial reasoning. | method 0.99 | 0.58 | 0.08 | asserted 1.00 |
| V2W can be integrated with existing VLMs in a training-free manner and can be further enhanced by optimizing the exploration policy via reinforcement learning. | method 0.73 | 0.60 | 0.06 | asserted 0.71 |
| Experiments on spatial mental modeling benchmarks demonstrate that V2W substantially improves VLM baselines in the training-free setting and achieves state-of-the-art performance with the learned exploration policy, highlighting the importance of active exploration for spatial intelligence. | result 0.98 | 0.96 | 0.81 | asserted 1.00 |
| Project page: \url{https://fangzhou2000.github.io/projects/v2w}. | future work 0.55 | 0.14 | 0.02 | unclear 0.98 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-20: 1 request, 7,116 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (6)
Figures are the authors', reproduced from the paper.
References (90)
- [1] D. Azuma, T. Miyanishi, S. Kurita, and M. Kawanabe. Scanqa: 3d question answering for spatial scene understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19129–19139, 2022 on Pith
- [2] S. Bai, Y . Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025 on PithPith's reviewarXiv
- [3] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025 on PithPith's reviewarXiv
- [4] D. Bucciarelli, N. Moratelli, M. Cornia, L. Baraldi, and R. Cucchiara. Personalizing multimodal large language models for image captioning: an experimental analysis. InEuropean Conference on Computer Vision (ECCV) Workshops, pages 351–368, 2024 on Pith
- [5] M. Cao, X. Li, X. Liu, I. Reid, and X. Liang. Spatialdreamer: Incentivizing spatial reasoning via active mental imagery.arXiv preprint arXiv:2512.07733, 2025 on PitharXiv
- [6] M. Cao, H. Lin, H. Li, H. Tang, R. Xu, D. An, X. Liu, I. Reid, and X. Liang. Seeing through imagination: Learning scene geometry via implicit spatial world modeling.arXiv preprint arXiv:2512.01821, 2025. 10 on PitharXiv
- [7] N. Carion, L. Gustafson, Y .-T. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V . Alwala, H. Khedr, A. Huang, et al. Sam 3: Segment anything with concepts.arXiv preprint arXiv:2511.16719, 2025 on PithPith's reviewarXiv
- [8] B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14455–14465, 2024 on Pith
- [9] D. Z. Chen, A. X. Chang, and M. Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. InEuropean Conference on Computer Vision (ECCV), pages 202–221, 2020 on Pith
- [10] Z. Chen, A. Gholami, M. Nießner, and A. X. Chang. Scan2cap: Context-aware dense captioning in rgb-d scans. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3193–3203, 2021 on Pith
- [11] Z. Chen, X. Lu, Z. Zheng, P. Li, L. He, Y . Zhou, J. Shao, B. Zhuang, and L. Sheng. Geometrically-constrained agent for spatial reasoning.arXiv preprint arXiv:2511.22659, 2025 on PitharXiv
- [12] Z. Chen, M. Zhang, X. Yu, X. Luo, M. Sun, Z. Pan, Y . Feng, P. Pei, X. Cai, and R. Huang. Think with 3d: Geometric imagination grounded spatial reasoning from limited views.arXiv preprint arXiv:2510.18632, 2025 on PitharXiv
- [13] A.-C. Cheng, H. Yin, Y . Fu, Q. Guo, R. Yang, J. Kautz, X. Wang, and S. Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models.Advances in Neural Information Processing Systems (NeurIPS), 37:135062–135093, 2024 on Pith
- [14] K. Cheng, W. Song, J. Fan, Z. Ma, Q. Sun, F. Xu, C. Yan, N. Chen, J. Zhang, and J. Chen. Caparena: Benchmarking and analyzing detailed image captioning in the llm era. InFindings of the Association for Computational Linguistics: ACL 2025, pages 14077–14094, 2025 on Pith
- [15] W. Dai, J. Li, D. Li, A. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning.Advances in Neural Information Processing Systems (NeurIPS), 36:49250–49267, 2023 on Pith
- [16] W. Deng, X. Zhang, and M. Qi. Active exploring like a pigeon: Reinforcing spatial reasoning via agentic vision-language models. InInternational Conference on Machine Learning (ICML), pages 24407–24417, 2026 on Pith
- [17] Z. Fan, J. Zhang, R. Li, J. Zhang, R. Chen, H. Hu, K. Wang, H. Qu, D. Wang, Z. Yan, et al. Vlm-3r: Vision-language models augmented with instruction-aligned 3d reconstruction.arXiv preprint arXiv:2505.20279, 2025 on PithPith's reviewarXiv
- [18] M. A. Fischler and R. C. Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.Communications of the ACM, 24(6): 381–395, 1981 on Pith
- [19] X. Gao, Z. Zhang, D. Z. Chen, S. Xu, L. Quan, E. Pérez-Pellitero, and Y . Jang. Map2thought: Explicit 3d spatial reasoning via metric cognitive maps.arXiv preprint arXiv:2601.11442, 2026 on PitharXiv
- [20] Y . Guo, Y . Hou, W. Ma, M. Tang, and M.-H. Yang. Pursuing minimal sufficiency in spatial reasoning.arXiv preprint arXiv:2510.16688, 2025 on PitharXiv
show 70 more
- [21] T. Gupta and A. Kembhavi. Visual programming: Compositional visual reasoning without train- ing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14953–14962, 2023 on Pith
- [22] D. Harris. Spatial reasoning in context: bridging cognitive and educational perspectives of spatial-mathematics relations. InFrontiers in Education, volume 8, page 1302099. Frontiers Media SA, 2023 on Pith
- [23] J. Huang, B. Jia, Y . Wang, Z. Zhu, X. Linghu, Q. Li, S.-C. Zhu, and S. Huang. Unveiling the mist over 3d vision-language understanding: Object-centric evaluation with chain-of-analysis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)... on Pith
- [24] J. Huang, X. Ma, X. Linghu, J. He, Q. Li, S.-C. Zhu, Y . Chen, B. Jia, and S. Huang. Leo- vl: Efficient scene representation for scalable 3d vision-language learning.arXiv preprint arXiv:2506.09935, 2025 on PitharXiv
- [25] X. Huang, J. Wu, Q. Xie, and K. Han. 3drs: Mllms need 3d-aware representation supervision for scene understanding.Advances in Neural Information Processing Systems (NeurIPS), 38: 67961–67988, 2025 on Pith
- [26] Y . Ji, H. Tan, J. Shi, X. Hao, Y . Zhang, H. Zhang, P. Wang, M. Zhao, Y . Mu, P. An, et al. Robobrain: A unified brain model for robotic manipulation from abstract to concrete. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages ... on Pith
- [27] P. N. Johnson-Laird. Mental models in cognitive science.Cognitive science, 4(1):71–115, 1980 on Pith
- [28] P. N. Johnson-Laird.Mental models: Towards a cognitive science of language, inference, and consciousness. Harvard University Press, 1983 on Pith
- [29] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, et al. Segment anything. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4015–4026, 2023 on Pith
- [30] V . Leroy, Y . Cabon, and J. Revaud. Grounding image matching in 3d with mast3r. InEuropean Conference on Computer Vision (ECCV), pages 71–91, 2024 on Pith
- [31] B. Li, X. Sun, J. Liu, Z. Wang, J. Wu, X. Yu, H. Chen, E. Barsoum, M. Chen, and Z. Liu. Latent visual reasoning.arXiv preprint arXiv:2509.24251, 2025 on PithPith's reviewarXiv
- [32] H. Li, Z. Zou, F. Liu, X. Zhang, F. Hong, Y . Cao, Y . Lan, M. Zhang, G. Yu, D. Zhang, et al. Iggt: Instance-grounded geometry transformer for semantic 3d reconstruction.arXiv preprint arXiv:2510.22706, 2025 on PitharXiv
- [33] H. Li, Q. Cao, T. Tang, K. Xiang, Z. Guo, J. Han, H. Xu, and X. Liang. Thinking with geometry: Active geometry integration for spatial reasoning.arXiv preprint arXiv:2602.06037, 2026 on PithPith's reviewarXiv
- [34] J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational Conference on Machine Learning (ICML), pages 19730–19742. PMLR, 2023 on Pith
- [35] H. Lin, S. Chen, J. Liew, D. Y . Chen, Z. Li, G. Shi, J. Feng, and B. Kang. Depth anything 3: Recovering the visual space from any views.arXiv preprint arXiv:2511.10647, 2025 on PithPith's reviewarXiv
- [36] H. Liu, C. Li, Y . Li, and Y . J. Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306, 2024 on Pith
- [37] P. Lu, B. Peng, H. Cheng, M. Galley, K.-W. Chang, Y . N. Wu, S.-C. Zhu, and J. Gao. Chameleon: Plug-and-play compositional reasoning with large language models.Advances in Neural Information Processing Systems (NeurIPS), 36:43447–43478, 2023 on Pith
- [38] Z. Luo, C. Zhang, S. Yong, C. Dai, Q. Wang, H. Ran, G. Shi, K. P. Sycara, and Y . Xie. pyspatial: Generating 3d visual programs for zero-shot spatial reasoning. InInternational Conference on Learning Representations (ICLR), 2026 on Pith
- [39] D. Marsili, R. Agrawal, Y . Yue, and G. Gkioxari. Visual agentic ai for spatial reasoning with a dynamic api. InProceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 19446–19455, 2025 on Pith
- [40] OpenAI. GPT-4V(ision) System Card, Sept. 2023. URL https://openai.com/index/ gpt-4v-system-card/ on Pith
- [41] OpenAI. GPT-4o System Card, Aug. 2024. URL https://openai.com/index/ gpt-4o-system-card/. 12 on Pith
- [42] OpenAI. OpenAI o1 System Card, Dec. 2024. URL https://openai.com/index/ openai-o1-system-card/ on Pith
- [43] OpenAI. Introducing GPT-4.1 in the API, Apr. 2025. URL https://openai.com/index/ gpt-4-1/ on Pith
- [44] Z. Peng, W. Wang, L. Dong, Y . Hao, S. Huang, S. Ma, and F. Wei. Kosmos-2: Grounding multimodal large language models to the world.arXiv preprint arXiv:2306.14824, 2023 on PithPith's reviewarXiv
- [45] Z. Qi, Z. Zhang, Y . Fang, J. Wang, and H. Zhao. Gpt4scene: Understand 3d scenes from videos with vision-language models.arXiv preprint arXiv:2501.01428, 2025 on PithPith's reviewarXiv
- [46] Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https: //qwen.ai/blog?id=qwen3.5 on Pith
- [47] N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos. InInternational Conference on Learning Representations (ICLR), 2025 on Pith
- [48] A. Ray, J. Duan, E. L. Brown II, R. Tan, D. Bashkirova, R. Hendrix, K. Ehsani, A. Kembhavi, B. A. Plummer, R. Krishna, et al. Sat: Dynamic spatial aptitude training for multimodal language models. InConference on Language Modeling (CoLM), 2025 on Pith
- [49] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024 on PithPith's reviewarXiv
- [50] Y . Shen, K. Song, X. Tan, D. Li, W. Lu, and Y . Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face.Advances in Neural Information Processing Systems (NeurIPS), 36:38154–38180, 2023 on Pith
- [51] G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y . Peng, H. Lin, and C. Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024 on PithPith's reviewarXiv
- [52] Z. Shi, K. Mei, Y . Quan, D. N. Metaxas, and R. Tang. Improving visual reasoning with iterative evidence refinement.arXiv preprint arXiv:2603.14117, 2026 on PitharXiv
- [53] Z. Su, L. Li, M. Song, Y . Hao, Z. Yang, J. Zhang, G. Chen, J. Gu, J. Li, X. Qu, et al. Open- thinkimg: Learning to think with images via visual tool reinforcement learning.arXiv preprint arXiv:2505.08617, 2025 on PithPith's reviewarXiv
- [54] D. Surís, S. Menon, and C. V ondrick. Vipergpt: Visual inference via python execution for reasoning. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11888–11898, 2023 on Pith
- [55] X. Tan, B. Liu, Y . Bao, Q. Tian, Z. Gao, X. Wu, Z. Luo, S. Wang, Y . Zhang, X. Wang, et al. Towards safe and trustworthy embodied ai: foundations, status, and prospects.Shanghai AI Lab., Shanghai, China, Tech. Rep., 2025 on Pith
- [56] Q. Tian, X. Tan, J. Gong, Y . Xie, and L. Ma. Uniforward: Unified 3d scene and semantic field reconstruction via feed-forward gaussian splatting from only sparse-view images.arXiv preprint arXiv:2506.09378, 2025 on PithPith's reviewarXiv
- [57] Q. Tian, X. Tan, Y . Xie, and L. Ma. Drivingforward: Feed-forward 3d gaussian splatting for driving scene reconstruction from flexible surround-view input. InProceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 39, pages 7374–7382, 2025 on Pith
- [58] Q. Tian, Z. Wu, X. Wang, L. Ma, and X. Tan. Groupforward: Building referable 3d scenes via instance-grouped feed-forward gaussian splatting.arXiv preprint arXiv:2608.17535, 2026 on PithPith's reviewarXiv
- [59] Q. Tian, J. Ying, X. Wang, Y . Xie, L. Ma, and X. Tan. FLEG: feed-forward language embedded gaussian splatting from any views via compact semantic representation. InEuropean Conference on Computer Vision (ECCV), volume 17016, pages 318–335, 2026. 13 on Pith
- [60] H. Wang, Y . Zhao, T. Wang, H. Fan, X. Zhang, and Z. Zhang. Ross3d: Reconstructive visual instruction tuning with 3d-awareness. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9275–9286, 2025 on Pith
- [61] J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and D. Novotny. Vggt: Visual geometry grounded transformer. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025 on Pith
- [62] S. Wang, V . Leroy, Y . Cabon, B. Chidlovskii, and J. Revaud. Dust3r: Geometric 3d vision made easy. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20697–20709, 2024 on Pith
- [63] Y . Wang, J. Zhou, H. Zhu, W. Chang, Y . Zhou, Z. Li, J. Chen, J. Pang, C. Shen, and T. He.π3: Permutation-equivariant visual geometry learning.arXiv preprint arXiv:2507.13347, 2025 on PithPith's reviewarXiv
- [64] Z. Wang, C. Chen, F. Luo, Y . Dong, Y . Zhang, Y . Xu, X. Wang, P. Li, and Y . Liu. Actiview: Evaluating active perception ability for multimodal large language models. InProceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 7605–7633, 2025 on Pith
- [65] D. Wu, F. Liu, Y .-H. Hung, and Y . Duan. Spatial-mllm: Boosting mllm capabilities in visual- based spatial intelligence. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS), 2025 on Pith
- [66] J. Wu, J. Guan, K. Feng, Q. Liu, S. Wu, L. Wang, W. Wu, and T. Tan. Reinforcing spatial reasoning in vision-language models with interwoven thinking and visual drawing. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS), 2025 on Pith
- [67] M. Wu, J. Yang, J. Jiang, M. Li, K. Yan, H. Yu, M. Zhang, C. Zhai, and K. Nahrstedt. Vtool-r1: Vlms learn to think with images via reinforcement learning on multimodal tool use.arXiv preprint arXiv:2505.19255, 2025 on PitharXiv
- [68] P. Xu, S. Wang, Y . Zhu, J. Li, and Y . Zhang. Spatialbench: Benchmarking multimodal large language models for spatial cognition.arXiv preprint arXiv:2511.21471, 2025 on PithPith's reviewarXiv
- [69] J. Yang, S. Yang, A. W. Gupta, R. Han, L. Fei-Fei, and S. Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. InProceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 10632–10643, 2025 on Pith
- [70] Y . Yang, J. Liu, Z. Zhang, S. Zhou, R. Tan, J. Yang, Y . Du, and C. Gan. Mindjourney: Test-time scaling with world models for spatial reasoning. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS), 2025 on Pith
- [71] Z. Yang, L. Li, J. Wang, K. Lin, E. Azarnasab, F. Ahmed, Z. Liu, C. Liu, M. Zeng, and L. Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action.arXiv preprint arXiv:2303.11381, 2023 on PithPith's reviewarXiv
- [72] Z. Yang, X. Yu, D. Chen, M. Shen, and C. Gan. Machine mental imagery: Empower multimodal reasoning with latent visual tokens.arXiv preprint arXiv:2506.17218, 2025 on PithPith's reviewarXiv
- [73] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao. React: Synergizing rea- soning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023 on Pith
- [74] B. Yin, Q. Wang, P. Zhang, J. Zhang, K. Wang, Z. Wang, J. Zhang, K. Chandrasegaran, H. Liu, R. Krishna, et al. Spatial mental modeling from limited views.arXiv preprint arXiv:2506.21458, 2025 on PitharXiv
- [75] J. Ying, Q. Tian, R. Xu, X. Zhu, D. Dong, J. Xu, and X. Tan. Exemplar2vqa: A scalable exemplar-driven visual question answering generation framework via multi-agent coding.arXiv preprint arXiv:2609.37655, 2026 on PithPith's reviewarXiv
- [76] W. Yuan, J. Duan, V . Blukis, W. Pumacay, R. Krishna, A. Murali, A. Mousavian, and D. Fox. Robopoint: A vision-language model for spatial affordance prediction in robotics. InConference on Robot Learning (CoRL), pages 4005–4020. PMLR, 2025. 14 on Pith
- [77] X. Yue, Y . Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y . Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni... on Pith
- [78] J. Zhang, T. Hu, X. Huang, Y . Gong, and D. Zeng. 3dbench: a scalable 3d benchmark and instruction-tuning dataset. InProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI), pages 1706–1714, 2024 on Pith
- [79] J. Zhang, Y . Chen, Y . Xu, Z. Huang, J. Mei, J. Chen, Y . Zhou, Y .-J. Yuan, X. Cai, G. Huang, et al. From flatland to space: Teaching vision-language models to perceive and reason in 3d. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets ... on Pith
- [80] J. Zhang, J. Jiang, H. Li, Y . Chen, K. Jiang, and D. Z. Chen. Gap-mllm: Geometry-aligned pre-training for activating 3d spatial perception in multimodal large language models.arXiv preprint arXiv:2603.16461, 2026 on PithPith's reviewarXiv
- [81] Y . Zhang, Z. Gong, and A. X. Chang. Multi3drefer: Grounding text description to multiple 3d objects. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15225–15236, 2023 on Pith
- [82] Y . Zhang, J. Wu, W. Li, B. Li, Z. Ma, Z. Liu, and C. Li. Llava-video: Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024 on PithPith's reviewarXiv
- [83] Z. Zhang, Y . Wu, L. Jia, Y . Wang, Z. Zhang, Y . Li, B. Ran, F. Zhang, Z. Sun, Z. Yin, et al. Think3d: Thinking with space for spatial reasoning.arXiv preprint arXiv:2601.13029, 2026 on PithPith's reviewarXiv
- [84] D. Zheng, S. Huang, Y . Li, and L. Wang. Learning from videos for 3d world: Enhancing mllms with 3d vision geometry priors. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS), 2025 on Pith
- [85] D. Zheng, S. Huang, and L. Wang. Video-3d llm: Learning position-aware video representation for 3d scene understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8995–9006, 2025 on Pith
- [86] Z. Zheng, M. Yang, J. Hong, C. Zhao, G. Xu, L. Yang, C. Shen, and X. Yu. Deepeyes: Incen- tivizing" thinking with images" via reinforcement learning.arXiv preprint arXiv:2505.14362, 2025 on PithPith's reviewarXiv
- [87] C. Zhou, R. Wang, F. Luo, M. D. Pesé, Z. Fan, Y . Zhong, and S. Huang. Ff3r: Feedforward feature 3d reconstruction from unconstrained views.arXiv preprint arXiv:2604.09862, 2026 on PithPith's reviewarXiv
- [88] C. Zhu, T. Wang, W. Zhang, J. Pang, and X. Liu. Llava-3d: A simple yet effective pathway to empowering lmms with 3d capabilities. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4295–4305, 2025 on Pith
- [89] M. Zhu, H. Zhong, C. Zhao, Z. Du, Z. Huang, M. Liu, H. Chen, C. Zou, J. Chen, M. Yang, et al. Active-o3: Empowering multimodal large language models with active perception via grpo. arXiv preprint arXiv:2505.21457, 2025. 15 A Appendix A.1 Ablation Study Across Different Base M... on PithPith's reviewarXiv
- [90] Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals... on Pith
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment