Pith. sign in

REVIEW 4 major objections 7 minor 77 references

City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read City-VLM claims a single outdoor vision-language model beats prior LVLMs by 18.14% on average across three scene-understanding QA benchmarks.

desk verdict SVM-City is a genuinely useful dataset idea, but the 18.14% claim is not checkable and the evaluation likely leaks training scenes into the test benchmarks. read the letter →

arxiv 2507.12795 v1 pith:5OYH6QAM submitted 2025-07-17 cs.CV cs.AI

Penglei Sun , Yaoxian Song , Xiangru Zhu , Xiang Liu , Qiang Wang , Yue Liu , Changqun Xia , Tiefeng Li
show 2 more authors
Yang Yang Xiaowen Chu
This is my paper
classification cs.CVcs.AI
keywords outdoorsceneunderstandinglargevision-languagemodelincompletemultimodallearningquestionanswering2D-3Dfusioninstructiontuningdatasetcity-scaleperceptionpointcloud
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

City-VLM claims that outdoor scene understanding requires a vision-language model that handles multiple scales, viewpoints, and sensor modalities at once, something current LVLMs do not do well because they focus on single-modality indoor scenes. To support this, the paper builds SVM-City, an instruction-tuning dataset of 567k question-answer pairs from vehicle, drone, aerial, and satellite sources, and trains City-VLM on it. Rather than concatenating 2D image and 3D point-cloud features, City-VLM maps whatever modalities are available into a shared Gaussian latent space, which is meant to keep the model working even when one modality is missing. The paper reports an average 18.14% improvement over existing LVLMs on EarthVQA, City-3DQA, and Nuscenes-QA, and claims this shows pragmatic and generalization performance across multiple outdoor scenes.

What carries the argument

The key mechanism is the Incomplete Multimodal Fusion (IMF) module, a VAE-style fusion block that turns available 2D and 3D features into $\mu$ and $\log\sigma$, samples $z = \mu + \epsilon\sigma$ with $\epsilon \sim \mathcal{N}(0, I)$ under a KL-divergence penalty to $\mathcal{N}(0, I)$, and uses $\mu$ at inference. Missing modalities are zero-padded, so the module always sees a fixed-shape input but must learn to represent an absent modality as uncertainty in the Gaussian latent. This replaces explicit fusion operations and carries the paper's claim that a shared probabilistic embedding degrades gracefully when part of the sensor input is unavailable.

What would settle it

Compare the city identities, scene IDs, and point-cloud or image sources in SVM-City against the test splits of EarthVQA, City-3DQA, and Nuscenes-QA; if any test scene or city block appears in the training set, the 18.14% average improvement would reflect memorization rather than generalization, and an evaluation on explicitly disjoint held-out scenes would settle the claim.

Watch

Extended reading notes

Core claim

The central claim is that treating 2D and 3D outdoor inputs as samples from a joint probabilistic distribution, rather than fusing them by concatenation, MLPs, or cross-attention, makes an LVLM robust to missing sensor modalities and improves question answering across scene scales. The model encodes images and point clouds with separate encoders, converts their combined features into the mean and log-variance of a Gaussian, samples a fused latent during training with a KL penalty toward a standard normal, and uses the mean latent at inference. Trained on SVM-City, it reaches 78.84% on EarthVQA, 67.30% and 64.70% on the sentence-wise and city-wise splits of City-3DQA, and 61.6% on Nuscenes-QA, outperforming both specialist models and prior LVLMs by the reported margins.

Load-bearing premise

The central claim stands on the assumption that the SVM-City training scenes do not overlap with the EarthVQA, City-3DQA, and Nuscenes-QA test scenes, but because SVM-City is assembled from the same source datasets, the paper does not describe any split or exclusion that guarantees this.

Editorial extensions

If this is right

  • If the reported gains hold, City-VLM beats the best specialist model on EarthVQA without using semantic segmentation features as auxiliary input.
  • On Nuscenes-QA, the two-modality model reaches 61.6%, 1.2 points above the prior specialist fusion baseline, suggesting probabilistic fusion can compete with explicit cross-modal detection pipelines.
  • The IMF module outperforms both MLP-based and cross-attention-based fusion under zero-padded missing inputs on all three benchmarks, implying a probabilistic latent is a stronger way to tolerate incomplete modalities.
  • Low-altitude drone-scene QA improves by roughly 30% over general LVLMs, indicating that the largest gap is in views and scales that current LVLMs were never trained on.
  • Because inference uses the distribution mean, one model can answer with image only, point cloud only, or both, which is the practical configuration for heterogeneous outdoor sensors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the reported 18.14% average gain is evidence of generalization only if the SVM-City training set is disjoint from the EarthVQA, City-3DQA, and Nuscenes-QA test sets; since SVM-City is assembled from the same source datasets, a shared-city or shared-scene check would settle this directly.
  • Editorial inference: the IMF module's robustness to missing modalities is demonstrated through fusion-module ablations, not through random sensor dropout at evaluation time, so a test with modalities dropped during inference would clarify whether the learned robustness is real.
  • Editorial inference: because the evaluation uses an LLM-based judge to decide whether model answers match ground truth, some of the measured gain may reflect the judge's tolerance, and a human or exact-match replication on a held-out subset would quantify that dependence.
  • Editorial inference: if probabilistic latent fusion is the active ingredient, the same module should transfer to other incomplete multimodal settings such as robot navigation with intermittent LiDAR or satellite imaging with cloud occlusion, but the paper does not test those settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces SVM-City, an outdoor multisource instruction-tuning dataset constructed from NuScenes, LoveDA, EarthExplorer, UrbanBIS, and SensatUrban, containing 420k images, 4,811M point-cloud points, and 567k QA pairs across terrestrial, low-altitude, and high-altitude viewpoints. It also proposes City-VLM, an LVLM whose Incomplete Multimodal Fusion Module (IMF) uses a VAE-style probabilistic embedding to enable fusion when 2D or 3D input is missing. The paper reports experiments on EarthVQA, City-3DQA, and Nuscenes-QA and claims an average improvement of 18.14% over existing LVLMs, along with ablations showing that IMF outperforms MLP and attention fusion.

Significance. If the empirical claims were validated, the paper would offer a useful large-scale dataset and a plausible mechanism for probabilistic 2D/3D fusion under missing modalities. The scale and multi-viewpoint design of SVM-City are genuine strengths, and the IMF module is a reasonable transfer of incomplete-multimodal learning to LVLMs. However, the current evaluation does not establish the headline improvement: the paper does not document that SVM-City excludes the benchmark test scenes, and the 18.14% aggregate is not defined or reproducible. These are load-bearing issues because the paper's main contribution is empirical.

major comments (4)
  1. [3.1 and 5.1-5.4] The training distribution and the evaluation benchmarks appear to share source data. Section 3.1 states that SVM-City is derived from NuScenes, LoveDA, EarthExplorer, UrbanBIS, and SensatUrban, while the three evaluation benchmarks are Nuscenes-QA (built on NuScenes), EarthVQA (built on LoveDA remote-sensing imagery), and City-3DQA (built on UrbanBIS/SensatUrban point clouds and described in the authors' own prior work [55]). Section 5.1 says only that 'Our City-VLM is trained on the SVM-City dataset'; no train/test exclusion, split statistics, or dataset release are reported. Because the LVLM baselines in Tables 2-4 were not trained on SVM-City, any scene-level overlap would give City-VLM an asymmetric advantage. The authors must either document an explicit exclusion of all benchmark test scenes from SVM-City or re-run the evaluation with a released disjoint split.
  2. [Abstract, Introduction, Conclusion] The headline number '18.14%' is never defined. No averaging rule over Tables 2-4 is given, and natural computations do not reproduce it. For example, comparing City-VLM w/ IMF to the best LVLM baseline in each table gives roughly 3.59 percentage points on EarthVQA (vs. Instruct-BLIP), 46.70 percentage points on City-3DQA sentence-wise (vs. LLaVA), and 13.0-14.2 percentage points on Nuscenes-QA (vs. LidarLLM or LLaVA); the simple mean is about 20-21 points. The authors should state the exact formula, including which baselines are included and whether the average is over tasks or over all comparisons, or remove the aggregate claim.
  3. [5.1] Section 5.1 lacks the training details needed to interpret the benchmark comparisons. The paper does not report the number of training steps or epochs, LoRA rank and target modules, IMF latent dimension and network widths, or whether City-VLM was further fine-tuned on each benchmark's training split or evaluated zero-shot after SVM-City training. The baselines are described as fine-tuned on the benchmark training sets, so this information is necessary to know what is being compared and to reproduce Tables 2-4.
  4. [5.6] The ablation study in Section 5.6 does not control for model capacity or training budget. It is not stated whether the MLP and attention variants have the same number of parameters and the same optimization schedule as the IMF variant. Without such controls, the reported gains of IMF over the alternatives (e.g., 2.79 percentage points sentence-wise on City-3DQA) could be due to capacity or tuning rather than the probabilistic fusion mechanism.
minor comments (7)
  1. [5.1] Please report the GPT-4 judge version, temperature, and number of runs; the current description is not sufficient for reproducibility.
  2. [5.1] There is a typo in 'and we the evaluation code in this link'; it should be 'and we provide the evaluation code in this link.'
  3. [3.1 and Table 1] The source dataset is spelled both 'Earthexplorer' and 'EarthExplorer'; use one consistent spelling throughout.
  4. [Table 3] The heading 'Comparsion' should be 'Comparison.'
  5. [5.6] A case study is described under the Ablation Study heading; consider moving it to its own subsection.
  6. [5.2] The phrase 'an improvement of 1.83%, 0.7% over the best specialist models' is ambiguous; clarify whether 1.83% is a relative improvement and 0.7% is a percentage-point improvement.
  7. [Introduction and Conclusion] The paper repeatedly claims to be the 'first' to address this setting; please temper or substantiate these claims with a more systematic review of concurrent outdoor scene-understanding benchmarks.

Circularity Check

1 steps flagged · score 6.0 of 10

Evaluation is in-distribution: SVM-City is built from the same source datasets as the EarthVQA, City-3DQA and Nuscenes-QA test sets, and no split excluding them is reported.

  1. other [Section 3.1 (Data Generation); Section 5.1 (Implementation Details); Sections 5.2-5.4 (Evaluation Tasks)]
    "Specifically, the Nuscenes dataset [9] was acquired from vehicle-mounted sensors, while the LoveDA dataset [61] consists of spaceborne RS imagery. The Earthexplorer dataset [59] contains aerial orthophotos. Additionally, both the UrbanBIS [65] and SensatUrban [27] datasets are obtained from low-altitude drones. ... Our City-VLM is trained on the SVM-City dataset. ... We evaluate our method on three QA tasks covering high-altitude, low-altitude and terrestrial view, EarthVQA [60], City-3DQA [55] and Nuscenes-QA [47]."

    SVM-City is assembled from exactly the source datasets that underlie the three evaluation benchmarks: EarthVQA builds on LoveDA remote-sensing imagery, City-3DQA builds on UrbanBIS/SensatUrban point clouds, and Nuscenes-QA builds on NuScenes. The paper reports no train/test exclusion, no split statistics, and no disjointness check between SVM-City and the benchmark test sets; Section 5.1 only states that City-VLM is trained on SVM-City. The reported 'prediction' on these benchmarks is therefore not an independent test of generalization: the model may have been trained on the same city-scale scenes it is then evaluated on, so scene-specific cues (layouts, colors, counts) can inflate accuracy through memorization rather than reasoning.

full rationale

This is an empirical dataset/model paper, not a mathematical derivation, so the audit focuses on whether the evaluation is external to the training distribution. The IMF module is a standard VAE with reparameterization and KL regularization; there is no equation-level circularity in Sections 4.1-4.2. The circularity problem is in the evaluation protocol: Section 3.1 creates SVM-City from NuScenes, LoveDA, EarthExplorer, UrbanBIS, and SensatUrban, while Sections 5.2-5.4 test on EarthVQA, City-3DQA, and Nuscenes-QA, which are built from those same sources. No split or exclusion is documented, so the claimed 18.14% average improvement over LVLMs is not demonstrated to be out-of-distribution generalization. This missing control is a partial circularity of the evaluation rather than a definitional identity, so the score is 6 rather than higher. The aggregate 18.14% is also not reproducible from Tables 2-4 without an averaging rule, but that is a reporting issue, not circularity. If the authors release SVM-City with a verified exclusion of all benchmark test scenes, the leakage concern would be resolved and the score would drop substantially.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central empirical result rests on four assumptions: standard VAE math, zero-padding for missing modalities, correctness of the auto-generated dataset, and disjointness of SVM-City from the evaluation benchmarks. The last assumption is unstated and is the main threat. No free parameters are fitted in an analytic derivation, but several training and evaluation choices that affect the headline numbers are unreported.

free parameters (5)
  • Unreported number of training epochs/steps
    Section 5.1 gives optimizer settings (lr 1e-3, weight decay 5e-4, batch size 4, LoRA) but no duration; final accuracy depends on this.
  • LoRA rank and target modules
    Section 5.1 says 'in the LoRA setting' but does not specify rank or which modules are adapted; directly affects model capacity.
  • IMF latent dimension and network widths
    Section 4.2 defines mu and sigma through f_mu/f_sigma but never states the dimension of z or the architecture of these functions.
  • Downstream fine-tuning protocol for EarthVQA, City-3DQA, Nuscenes-QA
    Section 5.1 says only 'trained on SVM-City'; whether City-VLM is zero-shot or fine-tuned on each benchmark is not stated, which changes the comparison.
  • GPT-4 judge version and temperature
    Section 5.1 evaluates with a GPT-4 text judge but does not report model version, temperature, or number of judge runs; the metric is stochastic.
assumptions (4)
  • standard math Reparameterization trick and KL divergence to N(0,I) are valid and unbiased as used in Eqs. (6)-(7).
    Standard VAE machinery from Kingma [35]; used in Section 4.2.
  • domain assumption Zero-padding a missing modality is a valid way to train a missing-modal fusion module.
    Adopted from Pan et al. [45] and Wei et al. [62] in Section 4.2; no comparison to alternatives such as learned masks.
  • domain assumption Combining NuScenes, LoveDA, EarthExplorer, UrbanBIS and SensatUrban with ChatGPT-generated QA yields a consistent, correct instruction-tuning dataset.
    Section 3.1 describes annotation via segmentation and ChatGPT; correctness is asserted without human verification or quality metrics.
  • domain assumption The EarthVQA, City-3DQA and Nuscenes-QA test sets are disjoint from the SVM-City training data.
    Never stated or enforced in Sections 3.1 and 5.1; the source datasets are the same families as the benchmarks. This is the load-bearing assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning." pith.science (2026). https://pith.science/paper/5OYH6QAM

@misc{pith2026250712795,
  author       = {Pith},
  title        = {Pith review of: City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5OYH6QAM}},
  note         = {Machine review of arXiv:2507.12795}
}
abstract

Scene understanding enables intelligent agents to interpret and comprehend their environment. While existing large vision-language models (LVLMs) for scene understanding have primarily focused on indoor household tasks, they face two significant limitations when applied to outdoor large-scale scene understanding. First, outdoor scenarios typically encompass larger-scale environments observed through various sensors from multiple viewpoints (e.g., bird view and terrestrial view), while existing indoor LVLMs mainly analyze single visual modalities within building-scale contexts from humanoid viewpoints. Second, existing LVLMs suffer from missing multidomain perception outdoor data and struggle to effectively integrate 2D and 3D visual information. To address the aforementioned limitations, we build the first multidomain perception outdoor scene understanding dataset, named \textbf{\underline{SVM-City}}, deriving from multi\textbf{\underline{S}}cale scenarios with multi\textbf{\underline{V}}iew and multi\textbf{\underline{M}}odal instruction tuning data. It contains $420$k images and $4, 811$M point clouds with $567$k question-answering pairs from vehicles, low-altitude drones, high-altitude aerial planes, and satellite. To effectively fuse the multimodal data in the absence of one modality, we introduce incomplete multimodal learning to model outdoor scene understanding and design the LVLM named \textbf{\underline{City-VLM}}. Multimodal fusion is realized by constructing a joint probabilistic distribution space rather than implementing directly explicit fusion operations (e.g., concatenation). Experimental results on three typical outdoor scene understanding tasks show City-VLM achieves $18.14 \%$ performance surpassing existing LVLMs in question-answering tasks averagely. Our method demonstrates pragmatic and generalization performance across multiple outdoor scenes.

Figures

Figures reproduced from arXiv: 2507.12795 by the authors.

Figure 1
Figure 1. Various large vision-language models (LVLMs) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overview of SVM-City: (a) the visual semantic from SVM-City and (b) the data annotation process applied to the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The statistics of SVM-City. imagery. The Earthexplorer dataset [59] contains aerial orthopho￾tos. Additionally, both the UrbanBIS [65] and SensatUrban [27] datasets are obtained from low-altitude drones. Question Template Taxonomy. Based on the taxonomy of spatial questions proposed in cognition [24], we propose the following question templates for applications in outdoor scenes. • Localization. These questions aim … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The architecture of City-VLM. The left (a) is the overview of City-VLM and the right (b) is the Incomplete Multimodal Fusion Module (IMF) Module. to form the input sequence for the large language model (LLM), which autoregressively generates the output sequence a: P(a|…
Figure 5
Figure 5. Figure 5: In this case studies, we compare the performance [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

77 extracted references · 52 canonical work pages

  1. [55]

    Penglei Sun, Yaoxian Song, Xiang Liu, Xiaofei Yang, Qiang Wang, YANG Yang, Xiaowen Chu, et al. 2024. 3D Question Answering for City Scene Understanding. In ACM Multimedia 2024

    work page 2024

  2. [1]

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. 2020. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16 . Springer, 422–440

    2020

  3. [2]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al

  4. [3]

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition . 6077–6086

    work page 2018

  5. [4]

    Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 2016. 3d semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1534–1543

    work page 2016

  6. [5]

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. 2022. ScanQA: 3D question answering for spatial scene understanding. Inproceedings of the IEEE/CVF conference on computer vision and pattern recognition . 19129–19139

    work page 2022

  7. [6]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  8. [7]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond.arXiv preprint arXiv:2308.12966 (2023)

    arXiv 2023

Show all 77 references
  1. [8]

    Yakoub Bazi, Laila Bashmal, Mohamad Mahmoud Al Rahhal, Riccardo Ricci, and Farid Melgani. 2024. Rs-llava: A large vision-language model for joint captioning and question answering in remote sensing imagery. Remote Sensing 16, 9 (2024), 1477

    2024

  2. [9]

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. 2020. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and patt...

    2020

  3. [10]

    Xu Cao, Tong Zhou, Yunsheng Ma, Wenqian Ye, Can Cui, Kun Tang, Zhipeng Cao, Kaizhao Liang, Ziran Wang, James M Rehg, et al. 2024. MAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understand- ing. In Proceedings of the IEEE/CVF Conference on C...

    2024

  4. [11]

    Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matterport3D: Learning from RGB-D Data in Indoor Environments. In International Conference on 3D Vision (3DV)

    2017

  5. [12]

    Jie Chang, Zhonghao Lan, Changmao Cheng, and Yichen Wei. 2020. Data uncer- tainty learning in face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 5710–5719

    2020

  6. [13]

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. 2020. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision. Springer, 202–221

    2020

  7. [14]

    Runnan Chen, Youquan Liu, Lingdong Kong, Nenglun Chen, Xinge Zhu, Yuexin Ma, Tongliang Liu, and Wenping Wang. 2024. Towards label-free scene under- standing by vision foundation models. Advances in Neural Information Processing Systems 36 (2024)

    2024

  8. [15]

    Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. 2024. LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding Reasoning and Planning. InProceedings of the IEEE/CVF Conference on Computer Vision and Patter...

    2024

  9. [16]

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 20...

    2023

  10. [17]

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 2017. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5828–5839

    2017

  11. [18]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv:2305.06500 [cs.CV]

    2023 arXiv

  12. [19]

    Thierry Deruyttere, Simon Vandenhende, Dusan Grujicic, Luc Van Gool, and Marie Francine Moens. 2019. Talk2Car: Taking Control of Your Self-Driving Car. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint C...

    2019

  13. [20]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  14. [21]

    Nanyi Fei, Zhiwu Lu, Yizhao Gao, Guoxing Yang, Yuqi Huo, Jingyuan Wen, Haoyu Lu, Ruihua Song, Xin Gao, Tao Xiang, et al . 2022. Towards artificial general intelligence via a multimodal foundation model. Nature Communications 13, 1 (2022), 3094

    2022

  15. [22]

    Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. 2024. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11401 (2024)

    2024 arXiv

  16. [23]

    Andreas Geiger, Philip Lenz, and Raquel Urtasun. 2012. Are we ready for au- tonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition . IEEE, 3354–3361

    2012

  17. [24]

    William G Hayward and Michael J Tarr. 1995. Spatial language and spatial representation. Cognition 55, 1 (1995), 39–84

    1995

  18. [25]

    Yining Hong, Zishuo Zheng, Peihao Chen, Yian Wang, Junyan Li, and Chuang Gan. 2024. Multiply: A multisensory object-centric embodied large language model in 3d world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 26406–26416

    2024

  19. [26]

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations

    2021

  20. [27]

    Qingyong Hu, Bo Yang, Sheikh Khalid, Wen Xiao, Niki Trigoni, and Andrew Markham. 2022. Sensaturban: Learning semantics from urban-scale photogram- metric point clouds. International Journal of Computer Vision 130, 2 (2022), 316–343

    2022

  21. [28]

    Haifeng Huang, Zehan Wang, Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. 2023. Chat-3d v2: Bridging 3d scene and large language models with object identifiers. arXiv preprint arXiv:2312.08168 (2023)

    2023 arXiv

  22. [29]

    Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du. 2021. Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. arXiv preprint arXiv:2112.11790 (2021)

    2021 arXiv

  23. [30]

    Drew A Hudson and Christopher D Manning. 2018. Compositional Attention Networks for Machine Reasoning. In International Conference on Learning Repre- sentations

    2018

  24. [31]

    Yang Jiao, Zequn Jie, Shaoxiang Chen, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. 2023. Msmdfusion: Fusing lidar and camera at multiple scales with multi- depth seeds for 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . ...

    2023

  25. [32]

    Zhao Jin, Munawar Hayat, Yuwei Yang, Yulan Guo, and Yinjie Lei. 2023. Context- aware alignment and mutual masking for 3d-language pre-training. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 10984– 10994

    2023

  26. [33]

    Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. 2018. Bilinear attention networks. Advances in neural information processing systems 31 (2018)

    2018

  27. [34]

    Sungkyung Kim, Adam Lee, Junyoung Park, Andrew Chung, Jusang Oh, and Jay- Yoon Lee. 2024. Towards Efficient Visual-Language Alignment of the Q-Former for Visual Reasoning Tasks. In Findings of the Association for Computational Linguistics: EMNLP 2024. 15155–15165

    2024

  28. [35]

    Diederik P Kingma. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)

    2013 arXiv

  29. [36]

    Manuel Kolmet, Qunjie Zhou, Aljoša Ošep, and Laura Leal-Taixé. 2022. Text2pos: Text-to-point-cloud cross-modal localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 6687–6696

    2022

  30. [37]

    Kartik Kuckreja, Muhammad Sohail Danish, Muzammal Naseer, Abhijit Das, Salman Khan, and Fahad Shahbaz Khan. 2024. Geochat: Grounded large vision- language model for remote sensing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 27831–27840

    2024

  31. [38]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning . PMLR, 19730–19742

    2023

  32. [39]

    Zeju Li, Chao Zhang, Xiaoyan Wang, Ruilong Ren, Yifan Xu, Ruifei Ma, Xiangde Liu, and Rong Wei. 2024. 3dmit: 3d multi-modal instruction tuning for scene understanding. In 2024 IEEE International Conference on Multimedia and Expo Workshops (ICMEW). IEEE, 1–5

    2024

  33. [40]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 26296–26306

    2024

  34. [41]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruc- tion tuning. Advances in neural information processing systems 36 (2024)

    2024

  35. [42]

    Sylvain Lobry, Diego Marcos, Jesse Murray, and Devis Tuia. 2020. RSVQA: Visual question answering for remote sensing data. IEEE Transactions on Geoscience and Remote Sensing 58, 12 (2020), 8555–8566

    2020

  36. [43]

    Gordon MacLeod and Mark Goodwin. 1999. Space, scale and state strategy: rethinking urban and regional governance. Progress in human geography 23, 4 (1999), 503–527

    1999

  37. [44]

    Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and An- drew Y Ng. 2011. Multimodal deep learning. InProceedings of the 28th international conference on machine learning (ICML-11) . 689–696

    2011

  38. [45]

    Yongsheng Pan, Mingxia Liu, Yong Xia, and Dinggang Shen. 2021. Disease-image- specific learning for diagnosis-oriented neuroimage synthesis with incomplete multi-modality data. IEEE transactions on pattern analysis and machine intelligence 44, 10 (2021), 6839–6853

    2021

  39. [46]

    Maria Parelli, Alexandros Delitzas, Nikolas Hars, Georgios Vlassis, Sotirios Anag- nostidis, Gregor Bachmann, and Thomas Hofmann. 2023. Clip-guided vision- language pre-training for question answering in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision a...

    2023

  40. [47]

    Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang

  41. [48]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  42. [49]

    Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, et al. 2021. Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied A...

    2021

  43. [50]

    Gabriel Sarch, Yue Wu, Michael Tarr, and Katerina Fragkiadaki. 2023. Open- Ended Instructable Embodied Agents with Memory-Augmented Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2023 . 3468–3500

    2023

  44. [51]

    Alaia Sola, Cristina Corchero, Jaume Salom, and Manel Sanmarti. 2020. Multi- domain urban-scale energy modelling tools: A review. Sustainable Cities and Society 54 (2020), 101872

    2020

  45. [52]

    Yaoxian Song, Penglei Sun, Pengfei Fang, Linyi Yang, Yanghua Xiao, and Yue Zhang. 2022. Human-in-the-loop Robotic Grasping Using BERT Scene Repre- sentation. In Proceedings of the 29th International Conference on Computational Linguistics. 2992–3006

    2022

  46. [53]

    Yaoxian Song, Penglei Sun, Haoyu Liu, Zhixu Li, Wei Song, Yanghua Xiao, and Xiaofang Zhou. 2024. Scene-Driven Multimodal Knowledge Graph Construction for Embodied AI. IEEE Transactions on Knowledge and Data Engineering (2024)

    2024

  47. [54]

    Yu Su, Yanfei Zhong, Qiqi Zhu, and Ji Zhao. 2021. Urban scene understanding based on semantic and socioeconomic features: From high-resolution remote sensing imagery to multi-source geographic datasets.ISPRS Journal of Photogram- metry and Remote Sensing 179 (2021), 50–65

    2021

  48. [56]

    Penglei Sun, Yaoxian Song, Xinglin Pan, Peijie Dong, Xiaofei Yang, Qiang Wang, Zhixu Li, Tiefeng Li, and Xiaowen Chu. 2024. Multi-Task Domain Adaptation for Language Grounding with 3D Objects. In European Conference on Computer Vision. Springer, 387–404

    2024

  49. [57]

    Michael Tanner, Pedro Pinies, Lina Maria Paz, Ştefan Săftescu, Alex Bewley, Emil Jonasson, and Paul Newman. 2022. Large-scale outdoor scene reconstruction and correction with vision. The International Journal of Robotics Research 41, 6 (2022), 637–663

    2022

  50. [58]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

    2023 arXiv

  51. [59]

    Receivables Management Section U.S Geological Survey. [n. d.]. EarthExplorer. https://earthexplorer.usgs.gov/

  52. [60]

    Junjue Wang, Zhuo Zheng, Zihang Chen, Ailong Ma, and Yanfei Zhong. 2024. Earthvqa: Towards queryable earth via relational reasoning-based remote sensing visual question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 5481–5489

    2024

  53. [61]

    Junjue Wang, Zhuo Zheng, Xiaoyan Lu, and Yanfei Zhong. 2021. LoveDA: A Re- mote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)

    2021

  54. [62]

    Shicai Wei, Yang Luo, Yuji Wang, and Chunbo Luo. 2024. Robust Multimodal Learning via Representation Decoupling. arXiv preprint arXiv:2407.04458 (2024)

    2024 arXiv

  55. [63]

    Zhiquan Wen, Guanghui Xu, Mingkui Tan, Qingyao Wu, and Qi Wu. 2021. Debi- ased visual question answering from feature and sample perspectives. Advances in Neural Information Processing Systems 34 (2021), 3784–3796

    2021

  56. [64]

    Jiannan Xiang, Xin Wang, and William Yang Wang. 2020. Learning to Stop: A Simple yet Effective Approach to Urban Vision-Language Navigation. InFindings of the Association for Computational Linguistics: EMNLP 2020 . 699–707

    2020

  57. [65]

    Guoqing Yang, Fuyou Xue, Qi Zhang, Ke Xie, Chi-Wing Fu, and Hui Huang. 2023. UrbanBIS: a large-scale benchmark for fine-grained urban building instance City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning ACM MM, 2025, Dublin, Ireland...

    2023

  58. [66]

    Senqiao Yang, Jiaming Liu, Ray Zhang, Mingjie Pan, Zoey Guo, Xiaoqi Li, Zehui Chen, Peng Gao, Yandong Guo, and Shanghang Zhang. 2023. Lidar-llm: Exploring the potential of large language models for 3d lidar understanding. arXiv preprint arXiv:2312.14074 (2023)

    2023 arXiv

  59. [67]

    Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. 2016. Stacked attention networks for image question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition . 21–29

    2016

  60. [68]

    Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. 2021. Center-based 3d object detection and tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 11784–11793

    2021

  61. [69]

    Fangwen Yu, Yujie Wu, Songchen Ma, Mingkun Xu, Hongyi Li, Huanyu Qu, Chenhang Song, Taoyi Wang, Rong Zhao, and Luping Shi. 2023. Brain-inspired multimodal hybrid neural network for robot place recognition. Science Robotics 8, 78 (2023), eabm6996

    2023

  62. [70]

    Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. 2019. Deep modular co- attention networks for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 6281–6290

    2019

  63. [71]

    Haihan Zhang, Chun Xie, Hisatoshi Toriya, Hidehiko Shishido, and Itaru Kitahara

  64. [72]

    Xiangtao Zheng, Binqiang Wang, Xingqian Du, and Xiaoqiang Lu. 2021. Mutual attention inception network for remote sensing visual question answering. IEEE Transactions on Geoscience and Remote Sensing 60 (2021), 1–14

    2021

  65. [73]

    Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. 2024. Uni3d: Exploring unified 3d representation at scale. In International Conference on Learning Representations (ICLR)

    2024

  66. [74]

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 2023. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 2911– 2921

    2023

  67. [2022]

    Advances in neural information processing systems 35 (2022), 23716–23736

    Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35 (2022), 23716–23736

    2022

  68. [2023]

    Remote Sensing 15, 15 (2023), 3871

    Vehicle Localization in a Completed City-Scale 3D Scene Using Aerial Images and an On-Board Stereo Camera. Remote Sensing 15, 15 (2023), 3871

    2023

  69. [2024]

    In Proceedings of the AAAI Conference on Artificial Intelligence, Vol

    Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 4542–4550

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.