Pith. sign in

REVIEW 3 major objections 5 minor 66 references

UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence with Spatial Reasoning and Understanding

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a single 8B open vision-language model, fine-tuned on a synthetic curriculum of urban instruction data, outperforms the compared general-purpose multi-modal LLMs—including GPT-4o on most tasks—across 12 urban tasks…

desk verdict Solid urban MLLM engineering with a plausible central result, but the benchmark is too entangled with the training pipeline and the 'all tasks' claim is contradicted by the paper's own detailed tables. read the letter →

arxiv 2506.23219 v1 pith:OZR2SKS4 submitted 2025-06-29 cs.CV cs.AIcs.CL

Jie Feng , Shengyuan Wang , Tianhui Liu , Yanxin Xi , Yong Li This is my paper
classification cs.CVcs.AIcs.CL
keywords urbanintelligencemulti-modallargelanguagemodelspatialreasoninginstructiontuningstreetviewimagerysatellitetrajectorypredictiongeospatialquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UrbanLLaVA tries to show that a single relatively small open vision-language model can become a general urban model, rather than needing one specialist per data type. The authors build UData, an urban instruction dataset that spans a location view (street views and map data), a trajectory view (routes and navigation), and a global view (satellite imagery), and train an 8B model with a three-stage pipeline. They report that the resulting model outperforms all compared baselines on a 12-task UBench across Beijing, London, and New York, including improvements of 31.47% to 375.38% over the base model on Beijing tasks, and beats GPT-4o on most tasks. The reason a reader would care is that a successful unified urban model could replace dozens of task-specific urban AI systems with one open, cheaply trained model.

What carries the argument

The load-bearing mechanism is UData, a synthetic instruction-data pipeline that fuses four urban data types—structured geospatial map data, public trajectory data, street-view images, and satellite images—into question–answer pairs ordered from location view to trajectory view to global view. On top of it, UTrain is a three-stage recipe (task alignment → knowledge learning → mixture tuning) that explicitly decouples learning task formats and reasoning skills from learning urban domain knowledge; the authors show the order matters and that this combination gives stable training and balanced task performance. UBench, the evaluation benchmark, extends prior benchmarks with six new tasks, including street-view address and landmark prediction, satellite address and land-use inference, and two multi-image tasks, providing the measurement apparatus for the claim.

What would settle it

Build a fresh held-out UBench for the same three cities using unreleased street-view and satellite patches, rephrased question templates, and POI lists that were not used in UData, then compare UrbanLLaVA with VILA1.5-8B and GPT-4o. If the margin over the base model collapses to near zero on these fresh items while still being large on the original split, the central claim fails; if the margin holds, the claim stands.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that fine-tuning a general MLLM on carefully ordered urban instruction data yields a model whose urban spatial reasoning is far stronger than the base model's and competitive with much larger proprietary systems. The training data is organized as a progression from local views (single street-view images plus structured geospatial knowledge), through trajectory views (route texts, real trajectories, visual navigation), to global views (satellite images, multi-image comparisons, cross-view alignment). The three-stage UTrain pipeline—task alignment first, then knowledge learning, then mixture tuning—lets the model learn task formats, urban domain knowledge, and mixed skills without the instability seen in one-stage training. In the strongest reported result, UrbanLLaVA-VILA1.5-8B improves on its VILA1.5-8B base by 31.47% to 375.38% across Beijing UBench tasks, outperforms every baseline on all Beijing tasks, and also transfers: a Beijing-trained model improves performance on London and New York benchmarks. The paper frames this as evidence for a unified multi-modal urban foundation model.

Load-bearing premise

The load-bearing premise is that the UBench validation tasks measure general urban ability rather than overlap with the UData training data, since four of the six new single-image tasks are explicitly aligned with the instruction data and are separated only by a train/validation split of the same original dataset; if validation items reuse templates, POI lists, or nearby image patches, the reported margins partly reflect memorization.

Editorial extensions

If this is right

  • If the results hold, an 8B open model can serve as one unified urban assistant handling geospatial QA, trajectory prediction, street-view address/landmark questions, satellite land-use questions, and cross-view retrieval/localization.
  • The three-stage training recipe (task alignment before knowledge learning, then mixture tuning) would be a reusable template for injecting domain knowledge into general MLLMs without hurting their general performance.
  • Cross-city transfer results imply that the learned spatial reasoning is not city-specific; a model trained with Beijing data can boost performance in London and New York.
  • The roughly 10.7-hour training time on 4 A100 GPUs for Beijing suggests specialized urban capability can be added cheaply to an existing open model.
  • General benchmark scores (RealWorldQA, LLaVA-Bench, MM-Vet) staying roughly level means the urban specialization does not obviously erase broader visual understanding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A cautious extension of the paper's logic: the largest gains (up to roughly 375%) come on tasks whose questions are generated from templates, so a focused audit should check whether validation examples share templates, POI lists, or nearby image patches with training data before the absolute numbers are read as calibrated.
  • If the leakage concern is resolved, the paper's cross-city results suggest the next high-value experiment is a fully unseen-city evaluation, since it would separate city-agnostic spatial reasoning from city-specific memorization; the authors' Beijing-trained model tested on London and New York is close to this but the benchmark tasks still come from the same task families.
  • The success of a synthetic, view-ordered curriculum implies a general recipe for other geo-domains (rural, maritime, indoor) that the paper does not test.
  • A testable extension the authors do not run: train the same UData/UTrain recipe on a different base MLLM and measure whether the relative gains transfer, which would show the method is about the data curriculum rather than the base model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper introduces UrbanLLaVA, a multi-modal large language model for urban tasks that processes street-view images, satellite images, geospatial data, and trajectory data. The authors construct UData, a synthetic instruction dataset spanning local, trajectory, and global views; propose UTrain, a three-stage fine-tuning pipeline (task alignment, knowledge learning, mixture tuning) built on VILA1.5-8B; and extend existing benchmarks into UBench with 12 tasks across three cities. The main empirical claim is that UrbanLLaVA outperforms open-source MLLMs and GPT-4o on most UBench tasks, with large relative gains over its base model, while retaining general capabilities and transferring from Beijing to London and New York.

Significance. The paper addresses a real gap: current MLLMs for urban data tend to be modality-specific, and there is no unified benchmark. The strengths of this manuscript are the open release of code and data, a systematic data-generation pipeline with quality checks, a careful ablation of data components (Table 4) and training strategies (Fig. 5), and the demonstration of cross-city generalization (Fig. 6). If the benchmark results are robust, an 8B model that substantially outperforms GPT-4o on multiple urban multimodal tasks would be a practically significant contribution to urban computing and CV. However, the current evidence is weakened by the benchmark's overlap with the training pipeline and the lack of statistical rigor, so the significance is conditional on a decontamination analysis.

major comments (3)
  1. [Section 2.3 and Section 2.1] The four new single-image UBench tasks (STV-Address, STV-Landmark, SAT-Address, SAT-Landuse) are generated by the same UData pipeline that produces the training instructions, using the same templates, option formats, and geospatial/POI sources. The only protection described in Section 2.3 is a random partition of 'the original dataset' into training and validation sets, which prevents image-level duplication but not template-, option-, or database-level leakage. If validation options are drawn from the same POI lists and address strings that appear in training, a model can learn cue associations that do not generalize to genuinely new urban queries. The cross-city results (Fig. 6) reduce the risk of exact-address memorization, but because the London and New York benchmarks are constructed with the same pipeline, format-level overfitting remains possible. Please provide a decontamination analysis—for example, held-out cities with independently regenerated templates and options, or a fine-grained analysis of whether the model's errors correlate with training-set nearest neighbors—and re-state the headline gains as conditional on this analysis.
  2. [Section 3.2, Table 2, Tables 7–8] The claim that 'UrbanLLaVA shows marked improvements over all baselines across all tasks in UBench' is not supported by the paper's own tables. According to Table 2, UrbanLLaVA@London is 0.73% below GPT-4o on GeoQA and UrbanLLaVA@NewYork is 7.37% below; the detailed Tables 7 and 8 show further per-task deficits against the best baseline: TrajPredict (London −12.73%, New York −24.32%), CameraLoc (London −8.77%), and SceneComp (New York −10.22%). The aggregated 'vs. Best Baseline' row in Table 2 therefore obscures the per-task picture. Please replace the 'all tasks' phrasing with an accurate per-task summary and ensure the main text does not overstate the result.
  3. [Section 3.2 and Table 1] No error bars, confidence intervals, or significance tests are reported for any UBench comparison, and several tasks have very small sample sizes. Navigation has only 50 examples per city, and most other tasks have 200; a binary success-rate difference of 0.34 vs. 0.06 (Beijing Navigation) has a large confidence interval. Given that the paper's most dramatic claims are relative gains of up to 375% (Table 2, Geo+Traj), the absence of variance estimates makes it impossible to distinguish systematic improvement from noise. Please report bootstrap confidence intervals or standard errors for the main comparisons, and ideally repeat the fine-tuning with multiple seeds for the central results.
minor comments (5)
  1. [Table 8] The row label 'CityGPT-V-VILA1.5-8B' should be 'UrbanLLaVA-VILA1.5-8B' for consistency with Tables 6 and 7.
  2. [Section 3.2] The text uses 'LLama3.2' while the rest of the paper uses 'LLaMA3.2'; please unify the spelling.
  3. [Supplementary Figure 17] The explanation for SceneComp appears to be copied from Figure 8 (STV-Outlier) and references a bike lane, which does not match the question about counting buildings; replace it with the correct explanation.
  4. [Figure 2] Figure 2 is extremely dense and the fonts are very small; splitting it into separate data, training, and benchmark diagrams would improve readability.
  5. [Section 6] The limitation section states that experiments focus on the 8B model, but Table 9 and Figure 12 report results for Qwen2.5-VL-7B and VILA1.5-3B/13B; please make the limitation statement more precise.

Circularity Check

1 steps flagged · score 6.0 of 10

UBench's new single-image tasks are generated from the same UData pipeline and only randomly split, so the headline gains partly measure training-distribution fit rather than independent urban ability; cross-city results provide partial external support, so circularity is partial.

  1. fitted input called prediction [Section 2.3 (UBench), Section 2.1 (UData), Table 1]
    "Four of these tasks are designed for single street view and satellite images, including address inference for both image types, landmark recognition for street view images, and land use inference for satellite images. These single-image tasks are aligned with the urban instruction data, and we partition the original dataset into training and validation sets to prevent potential data leakage."

    UBench tasks STV-Address, STV-Landmark, SAT-Address and SAT-Landuse (Table 1) mirror UData's Location Address, Landmark Details and Landuse Inference instruction samples (Section 2.1, Figures 30-33). The paper explicitly says these evaluation tasks are 'aligned with the urban instruction data' and that the only protection is a random train/validation split of the same generated dataset. Since UrbanLLaVA is fine-tuned on UData, its scores on these tasks partly measure memorization of the same templates, option formats, POI lists, and data sources that baselines never saw. The London and New York versions are produced by the same pipeline, so cross-city transfer reduces content-level leakage but not template- or format-level overlap.

full rationale

The central empirical claim is that UrbanLLaVA outperforms all baselines across UBench, and the paper attributes this to UData and UTrain. The key weakness is that four of the six new UBench tasks are stated to be 'aligned with the urban instruction data' and are split from the same dataset that was used for training. For these tasks, the evaluation is not independent of the training distribution: the model has been optimized on the same template-and-option structure, so large relative gains over untrained baselines are expected and do not by themselves establish general urban ability. The cross-city experiments (trained on Beijing, tested on London and New York) provide meaningful partial external support, and the general benchmarks in Table 3 show the model retains general competence, which prevents a higher circularity score. The paper also contains an overstatement in Section 3.2 ('marked improvements over all baselines across all tasks') that its own Tables 7-8 contradict for London/NY GeoQA, but that is a consistency/correctness issue rather than a circularity issue. No machine-checked or parameter-free external theorem is invoked, and the self-citations to CityGPT and CityBench are not load-bearing for the derivation chain; the central circularity is the benchmark/training overlap. Overall, the reported benchmark superiority is partially circular because the new evaluation tasks are generated from the same pipeline as the training data, warranting a score of 6 rather than a clean non-finding.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The empirical claims rest on data-generation and evaluation assumptions rather than equations: raw platform data is accurate, instruction templates produce correct labels, a train/validation split is sufficient to prevent leakage, and general benchmarks can be judged by GPT4o. No free parameters beyond training hyperparameters, and no invented entities.

free parameters (3)
  • learning_rate = 1e-5
    Section 3.3 and Supplementary Figure 10a identify learning rate as the key stability parameter; 1e-5 is chosen over the VILA default 1e-4 based on loss smoothness and UBench performance.
  • training_pipeline_order = Task Alignment -> Knowledge -> Mixture
    Section 3.3 Figure 5 shows the three-stage order is selected based on UBench task performance across different orderings.
  • mixture_data_ratio = 1/3 domain data plus 1/3 general text data
    Section 2.2 states that mixture tuning resamples 1/3 domain specific data from the first two stages and 1/3 general textual instruction data; the exact composition of the remaining third is not fully specified or ablated.
assumptions (4)
  • domain assumption UBench validation samples are not contaminated by UData training samples beyond the stated random split.
    Section 2.3 says UBench tasks are 'aligned with the urban instruction data' and only partitioned into training and validation; no leakage analysis is provided for the four new tasks or for CityBench and UrBench tasks.
  • domain assumption OpenStreetMap, Foursquare, Google Earth, and Baidu or Google street view data provide reliable ground truth for synthetic QA labels.
    Section 2.1 builds instruction data by transforming raw platform data into questions and answers; errors in raw data or templates propagate into both training and evaluation.
  • domain assumption GPT4o ratings are valid for LLaVA-Bench and MM-Vet comparison.
    Section 3.1 says ratings are produced by GPT4o as judge; judge bias or variance is not quantified.
  • domain assumption VILA1.5-8B remains a suitable base after multi-stage fine-tuning.
    Main results use VILA1.5-8B; general benchmark scores in Table 3 show a small drop on LLaVA-Bench and MM-Vet, which is assumed acceptable for the urban gains.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence with Spatial Reasoning and Understanding." pith.science (2026). https://pith.science/paper/OZR2SKS4

@misc{pith2026250623219,
  author       = {Pith},
  title        = {Pith review of: UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence with Spatial Reasoning and Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OZR2SKS4}},
  note         = {Machine review of arXiv:2506.23219}
}
abstract

Urban research involves a wide range of scenarios and tasks that require the understanding of multi-modal data. Current methods often focus on specific data types and lack a unified framework in urban field for processing them comprehensively. The recent success of multi-modal large language models (MLLMs) presents a promising opportunity to overcome this limitation. In this paper, we introduce $\textit{UrbanLLaVA}$, a multi-modal large language model designed to process these four types of data simultaneously and achieve strong performance across diverse urban tasks compared with general MLLMs. In $\textit{UrbanLLaVA}$, we first curate a diverse urban instruction dataset encompassing both single-modal and cross-modal urban data, spanning from location view to global view of urban environment. Additionally, we propose a multi-stage training framework that decouples spatial reasoning enhancement from domain knowledge learning, thereby improving the compatibility and downstream performance of $\textit{UrbanLLaVA}$ across diverse urban tasks. Finally, we also extend existing benchmark for urban research to assess the performance of MLLMs across a wide range of urban tasks. Experimental results from three cities demonstrate that $\textit{UrbanLLaVA}$ outperforms open-source and proprietary MLLMs in both single-modal tasks and complex cross-modal tasks and shows robust generalization abilities across cities. Source codes and data are openly accessible to the research community via https://github.com/tsinghua-fib-lab/UrbanLLaVA.

Figures

Figures reproduced from arXiv: 2506.23219 by the authors.

Figure 1
Figure 1. Existing works vs. our UrbanLLaVA in urban research. tial data [1, 2], and spatiotemporal series data [20, 28]. To￾gether, these data types capture the multi-faceted nature of urban environments, representing a wide range of spatial information and urban knowledge [35, 46, 57]. Integrat￾ing these multi-modal data into a cohesive framework is essential for developing a systematic understanding of ur￾ban spaces and ad… view at source ↗
Figure 2
Figure 2. The framework of UrbanLLaVA, including UData, UTrain and UBench. build-in common sense and reasoning abilities as a cen￾tral component for unifying the processing data across various modalities, such as images [31], speech [19], and time series [25]. For example, Ma et al. [34] develop a vision-language model as a conversational assistant for autonomous driving, Brohan et al. [4] introduce RT-2, a vision language mo… view at source ↗
Figure 3
Figure 3. The thorough composition of UData in Beijing. in MLLMs. Finally, we extend existing urban benchmarks to build a systematic urban benchmark UBench for evalu￾ating the capabilities of MLLMs in tackling diverse urban tasks. In summary, our contributions are as follows, • UrbanLLaVA is the first MLLM designed for the unified modeling of four major types of urban data, with the goal of fostering comprehensive understandi… view at source ↗
Figures from the paper (29 more)
Figure 4
Figure 4. Figure 4: UTrain: three-stage training pipeline. Training UrbanLLaVA presents significant challenges due to the heterogeneity of multi-modal urban instruction data and the diversity of urban tasks. Achieving stable train￾ing and balancing performance across various tasks is no￾t…
Figure 5
Figure 5. Figure 5: Performance of different training strategies. ‘K’ refers to [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Learning from one city (Beijing) can be directly generalized to other cities (London and New York). In this figure, Baseline is [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: An example of the SceneFunc task, where correct an [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 9
Figure 9. Figure 9: Input data examples for three-stage training. [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: (a) illustrates that the training loss is smoother and lower when the learning rate is [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 12
Figure 12. Figure 12: Results on UrbanLLaVA with different model sizes. age, satisfactory instruction following, and urban knowl￾edge mastering. STV-Landmark. A representative example is in Fig￾ure 14. In this task, models are required to find out the closest landmark feature to a given st…
Figure 13
Figure 13. Figure 13: An example of the SAT-LandUse task. The correct [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: An example of the STV-Landmark task. The correct [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: Example of a SAT-Address task. Prompt: The following is a multiple-choice question about selecting the most appropriate address for a street view image. A. Bajiajiayuan, Xisanqi, Houbajia, Haidian District, Beijing, 100192, China B. Beichen West Road, Chaoyang Distric…
Figure 16
Figure 16. Figure 16: Example of a STV-Address task. representative examples of our urban instruction data, as shown in [PITH_FULL_IMAGE:figures/full_fig_p015_16.png]
Figure 19
Figure 19. Figure 19: An example of a CameraLoc task. 4 [PITH_FULL_IMAGE:figures/full_fig_p015_19.png]
Figure 20
Figure 20. Figure 20: An example of global view training instances of Image Content. [PITH_FULL_IMAGE:figures/full_fig_p016_20.png]
Figure 21
Figure 21. Figure 21: An example of global view training instances of Landuse Inference. [PITH_FULL_IMAGE:figures/full_fig_p017_21.png]
Figure 22
Figure 22. Figure 22: An example of local view training instances of Location Address. [PITH_FULL_IMAGE:figures/full_fig_p017_22.png]
Figure 23
Figure 23. Figure 23: An example of local view training instances of Image Description. [PITH_FULL_IMAGE:figures/full_fig_p018_23.png]
Figure 24
Figure 24. Figure 24: An example of local view training instances of Landmark Details. [PITH_FULL_IMAGE:figures/full_fig_p018_24.png]
Figure 25
Figure 25. Figure 25: An example of local view training instances of Cross Modality Reasoning. [PITH_FULL_IMAGE:figures/full_fig_p019_25.png]
Figure 26
Figure 26. Figure 26: An example of global view training instances of Cross Modality Reasoning. [PITH_FULL_IMAGE:figures/full_fig_p020_26.png]
Figure 27
Figure 27. Figure 27: An example of global view training instances of Image Content. [PITH_FULL_IMAGE:figures/full_fig_p021_27.png]
Figure 28
Figure 28. Figure 28: An example of global view training instances of Cross Modality Reasoning. [PITH_FULL_IMAGE:figures/full_fig_p022_28.png]
Figure 29
Figure 29. Figure 29: An example of local view training instances of Cross Modality Reasoning. [PITH_FULL_IMAGE:figures/full_fig_p023_29.png]
Figure 30
Figure 30. Figure 30: An example of global view training instances of Location Address. [PITH_FULL_IMAGE:figures/full_fig_p024_30.png]
Figure 31
Figure 31. Figure 31: An example of global view training instances of Landuse Inference. [PITH_FULL_IMAGE:figures/full_fig_p025_31.png]
Figure 32
Figure 32. Figure 32: An example of local view training instances of Location Address. [PITH_FULL_IMAGE:figures/full_fig_p026_32.png]
Figure 33
Figure 33. Figure 33: An example of local view training instances of Landmark Details. [PITH_FULL_IMAGE:figures/full_fig_p026_33.png]
Figure 34
Figure 34. Figure 34: An example of global view training instances of Cross View Data. [PITH_FULL_IMAGE:figures/full_fig_p027_34.png]
Figure 35
Figure 35. Figure 35: An example of global view training instances of Multiple SAT Comparison. [PITH_FULL_IMAGE:figures/full_fig_p028_35.png]
Figure 36
Figure 36. Figure 36: Maps for Beijing, London and New York. 17 [PITH_FULL_IMAGE:figures/full_fig_p028_36.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

66 extracted references · 42 canonical work pages

  1. [1]

    Lamp: A language model on the map

    Pasquale Balsebre, Weiming Huang, and Gao Cong. Lamp: A language model on the map. arXiv preprint arXiv:2403.09059, 2024. 1, 3, 9

    arXiv 2024

  2. [2]

    City foundation models for learning general purpose rep- resentations from openstreetmap

    Pasquale Balsebre, Weiming Huang, Gao Cong, and Yi Li. City foundation models for learning general purpose rep- resentations from openstreetmap. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 87–97, 2024. 1, 9

    work page 2024

  3. [3]

    Street view imagery in urban analytics and gis: A review

    Filip Biljecki and Koichi Ito. Street view imagery in urban analytics and gis: A review. Landscape and Urban Planning, 215:104217, 2021. 9

    work page 2021

  4. [4]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. 2

    arXiv 2023

  5. [5]

    Touchdown: Natural language naviga- tion and spatial reasoning in visual street environments

    Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language naviga- tion and spatial reasoning in visual street environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12538–12547, 2019. 4

    work page 2019

  6. [6]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 3, 8

    arXiv 2023

  7. [7]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 5, 8

    arXiv 2024

  8. [8]

    Internvl: Scaling up vision founda- tion models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision founda- tion models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024. 5, 8 9

    work page 2024

Show all 66 references
  1. [9]

    Spatial- rgpt: Grounded spatial reasoning in vision language model

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Rui- han Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatial- rgpt: Grounded spatial reasoning in vision language model. arXiv preprint arXiv:2406.01584, 2024. 9

    2024 arXiv

  2. [10]

    Understanding world or predict- ing future? a comprehensive survey of world models

    Jingtao Ding, Yunke Zhang, Yu Shang, Yuheng Zhang, Ze- fang Zong, Jie Feng, Yuan Yuan, Hongyuan Su, Nian Li, Nicholas Sukiennik, et al. Understanding world or predict- ing future? a comprehensive survey of world models. arXiv preprint arXiv:2411.14499, 2024. 9

    2024

  3. [11]

    Enhancing chat language models by scal- ing high-quality instructional conversations

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scal- ing high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023. 5

    2023 arXiv

  4. [12]

    How abilities in large lan- guage models are affected by supervised fine-tuning data composition

    Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large lan- guage models are affected by supervised fine-tuning data composition. arXiv preprint arXiv:2310.05492, 2023. 4

    arXiv

  5. [13]

    Vlmevalkit: An open- source toolkit for evaluating large multi-modality models,

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. Vlmevalkit: An open- source toolkit for evaluating large multi-modality models,

  6. [14]

    Urban visual intelligence: Uncovering hidden city pro- files with street view images

    Zhuangyuan Fan, Fan Zhang, Becky PY Loo, and Carlo Ratti. Urban visual intelligence: Uncovering hidden city pro- files with street view images. Proceedings of the National Academy of Sciences, 120(27):e2220417120, 2023. 1, 9

    2023

  7. [15]

    Agent- move: A large language model based agentic framework for zero-shot next location prediction

    Jie Feng, Yuwei Du, Jie Zhao, and Yong Li. Agent- move: A large language model based agentic framework for zero-shot next location prediction. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human L...

    2025

  8. [16]

    Citygpt: Empowering urban spatial cognition of large language models

    Jie Feng, Tianhui Liu, Yuwei Du, Siqi Guo, Yuming Lin, and Yong Li. Citygpt: Empowering urban spatial cognition of large language models. In Proceedings of the 31th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2025. 2, 3, 9

    2025

  9. [17]

    A survey of large language model-powered spatial intelligence across scales: Advances in embodied agents, smart cities, and earth science

    Jie Feng, Jinwei Zeng, Qingyue Long, Hongyi Chen, Jie Zhao, Yanxin Xi, Zhilun Zhou, Yuan Yuan, Shengyuan Wang, Qingbin Zeng, et al. A survey of large language model-powered spatial intelligence across scales: Advances in embodied agents, smart cities, and earth science. arXiv ...

    2025 arXiv

  10. [18]

    City- bench: Evaluating the capabilities of large language models for urban tasks

    Jie Feng, Jun Zhang, Tianhui Liu, Xin Zhang, Tianjian Ouyang, Junbo Yan, Yuwei Du, Siqi Guo, and Yong Li. City- bench: Evaluating the capabilities of large language models for urban tasks. In Proceedings of the 31th ACM SIGKDD International Conference on Knowledge Discovery an...

    2025

  11. [19]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023. 2

    2023

  12. [20]

    Mobility-llm: Learning visiting intentions and travel prefer- ences from human mobility data with large language models

    Letian Gong, Yan Lin, Xinyue Zhang, Yiwen Lu, Xuedi Han, Yichen Liu, Shengnan Guo, Youfang Lin, and Huaiyu Wan. Mobility-llm: Learning visiting intentions and travel prefer- ences from human mobility data with large language models. arXiv preprint arXiv:2411.00823, 2024. 1, 2, 9

    2024 arXiv

  13. [21]

    Regiongpt: Towards region understanding vision lan- guage model

    Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, Yizhou Yu, Ping Luo, and Sifei Liu. Regiongpt: Towards region understanding vision lan- guage model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13796– 1380...

    2024

  14. [22]

    Urbanvlp: A multi- granularity vision-language pre-trained foundation model for urban indicator prediction

    Xixuan Hao, Wei Chen, Yibo Yan, Siru Zhong, Kun Wang, Qingsong Wen, and Yuxuan Liang. Urbanvlp: A multi- granularity vision-language pre-trained foundation model for urban indicator prediction. arXiv preprint arXiv:2403.16831,

    arXiv

  15. [23]

    Vision-language models for medical report generation and visual question answering: A review, 2024

    Iryna Hartsock and Ghulam Rasool. Vision-language models for medical report generation and visual question answering: A review, 2024. 9

    2024

  16. [24]

    Rsgpt: A remote sensing vision language model and benchmark

    Yuan Hu, Jianlong Yuan, Congcong Wen, Xiaonan Lu, and Xiang Li. Rsgpt: A remote sensing vision language model and benchmark. arXiv preprint arXiv:2307.15266, 2023. 9

    2023 arXiv

  17. [25]

    Time-llm: Time series forecasting by reprogramming large language models

    Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan- Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728, 2023. 2

    arXiv

  18. [26]

    Geochat: Grounded large vision-language model for remote sensing

    Kartik Kuckreja, Muhammad Sohail Danish, Muzammal Naseer, Abhijit Das, Salman Khan, and Fahad Shahbaz Khan. Geochat: Grounded large vision-language model for remote sensing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27831–27840...

    2024

  19. [27]

    Llava-med: Training a large language- and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems, 36, 2024

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language- and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  20. [28]

    Urbangpt: Spatio- temporal large language models

    Zhonghang Li, Lianghao Xia, Jiabin Tang, Yong Xu, Lei Shi, Long Xia, Dawei Yin, and Chao Huang. Urbangpt: Spatio- temporal large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5351–5362, 2024. 1, 9

    2024

  21. [29]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26689–26699, 2024. 4, 5, 8

    2024

  22. [30]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 5, 7, 8

    2024

  23. [31]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2, 8

    2024

  24. [32]

    Citylens: Bench- 10 marking large language-vision models for urban socioeco- nomic sensing

    Tianhui Liu, Jie Feng, Hetian Pang, Xin Zhang, Tianjian Ouyang, Zhiyuan Zhang, and Yong Li. Citylens: Bench- 10 marking large language-vision models for urban socioeco- nomic sensing. arXiv preprint arXiv:2506.00530, 2025. 9

    2025

  25. [33]

    Skysensegpt: A fine-grained in- struction tuning dataset and model for remote sensing vision- language understanding

    Junwei Luo, Zhen Pang, Yongjun Zhang, Tingzhu Wang, Linlin Wang, Bo Dang, Jiangwei Lao, Jian Wang, Jingdong Chen, Yihua Tan, et al. Skysensegpt: A fine-grained in- struction tuning dataset and model for remote sensing vision- language understanding. arXiv preprint arXiv:2406.10100,

    arXiv

  26. [34]

    Dolphins: Multimodal language model for driving

    Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Multimodal language model for driving. arXiv preprint arXiv:2312.00438, 2023. 2, 9

    2023 arXiv

  27. [35]

    On the opportunities and chal- lenges of foundation models for geoai (vision paper)

    Gengchen Mai, Weiming Huang, Jin Sun, Suhang Song, Deepak Mishra, Ninghao Liu, Song Gao, Tianming Liu, Gao Cong, Yingjie Hu, et al. On the opportunities and chal- lenges of foundation models for geoai (vision paper). ACM Transactions on Spatial Algorithms and Systems, 2024. 1, 9

    2024

  28. [36]

    LLaMA 3.2: Advancing Vision, Edge, and Mo- bile Devices

    Meta AI. LLaMA 3.2: Advancing Vision, Edge, and Mo- bile Devices. https://ai.meta.com/blog/llama- 3 - 2 - connect - 2024 - vision - edge - mobile - devices/, 2024. Accessed: 2024-11-01. 5

    2024

  29. [37]

    Lhrs-bot: Empowering remote sensing with vgi-enhanced large multimodal language model

    Dilxat Muhtar, Zhenshi Li, Feng Gu, Xueliang Zhang, and Pengfeng Xiao. Lhrs-bot: Empowering remote sensing with vgi-enhanced large multimodal language model. arXiv preprint arXiv:2402.02544, 2024. 9

    2024 arXiv

  30. [38]

    Introducing chatgpt

    OpenAI. Introducing chatgpt. https://openai.com/ blog/chatgpt/, 2022. 1

    2022

  31. [39]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. 2023. 8

    2023

  32. [40]

    Hello GPT-4

    OpenAI. Hello GPT-4. https : / / openai . com / index/hello-gpt-4o/, 2024. 5

    2024

  33. [41]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 5, 8

    2024 arXiv

  34. [42]

    Visionllm v2: An end-to-end general- ist multimodal large language model for hundreds of vision- language tasks

    Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Wenhai Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end general- ist multimodal large language model for hundreds of vision- language tasks. arXiv preprint arXiv:2406.08394, 2024. 9

    2024 arXiv

  35. [43]

    V*: Guided visual search as a core mechanism in multimodal llms

    Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13084–13094, 2024. 9

    2024

  36. [44]

    RealworldQA Dataset

    XAI Organization. RealworldQA Dataset. https : / / huggingface . co / datasets / xai - org/RealworldQA, 2024. Accessed: 2024-10-01. 5, 7

    2024

  37. [45]

    Analyz- ing large language models’ capability in location prediction

    Zhaomin Xiao, Eduardo Blanco, and Yan Huang. Analyz- ing large language models’ capability in location prediction. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 951–958, 2024. 1

    2024

  38. [46]

    Ur- ban generative intelligence (ugi): A foundational platform for agents in embodied city environment

    Fengli Xu, Jun Zhang, Chen Gao, Jie Feng, and Yong Li. Ur- ban generative intelligence (ugi): A foundational platform for agents in embodied city environment. arXiv preprint arXiv:2312.11813, 2023. 1

    2023 arXiv

  39. [47]

    Towards large rea- soning models: A survey of reinforced reasoning with large language models

    Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, et al. Towards large rea- soning models: A survey of reinforced reasoning with large language models. arXiv preprint arXiv:2501.09686, 2025. 9

    2025 arXiv

  40. [48]

    Par- ticipatory cultural mapping based on collective behavior data in location-based social networks

    Dingqi Yang, Daqing Zhang, and Bingqing Qu. Par- ticipatory cultural mapping based on collective behavior data in location-based social networks. ACM Transactions on Intelligent Systems and Technology (TIST), 7(3):1–23,

  41. [49]

    A survey on multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023. 1, 3, 8

    2023 arXiv

  42. [50]

    Mm-vet: Evaluating large multimodal models for inte- grated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for inte- grated capabilities. In International conference on machine learning. PMLR, 2024. 5, 7

    2024

  43. [51]

    Skyeyegpt: Unifying remote sensing vision-language tasks via instruc- tion tuning with large language model

    Yang Zhan, Zhitong Xiong, and Yuan Yuan. Skyeyegpt: Unifying remote sensing vision-language tasks via instruc- tion tuning with large language model. arXiv preprint arXiv:2401.09712, 2024. 9

    2024 arXiv

  44. [52]

    Earthgpt: A universal multi-modal large lan- guage model for multi-sensor image comprehension in re- mote sensing domain

    Wei Zhang, Miaoxin Cai, Tong Zhang, Yin Zhuang, and Xuerui Mao. Earthgpt: A universal multi-modal large lan- guage model for multi-sensor image comprehension in re- mote sensing domain. IEEE Transactions on Geoscience and Remote Sensing, 2024. 9

    2024

  45. [53]

    Urban foundation models: A survey

    Weijia Zhang, Jindong Han, Zhao Xu, Hang Ni, Hao Liu, and Hui Xiong. Urban foundation models: A survey. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6633–6643,

  46. [54]

    UrbanMLLM: Joint learning of cross-view imagery for urban understanding, 2025

    Xin Zhang, Tianjian Ouyang, Yu Shang, Qingmin Liao, and Yong Li. UrbanMLLM: Joint learning of cross-view imagery for urban understanding, 2025. 9

    2025

  47. [55]

    Per- ceiving urban inequality from imagery using visual language models with chain-of-thought reasoning

    Yunke Zhang, Ruolong Ma, Xin Zhang, and Yong Li. Per- ceiving urban inequality from imagery using visual language models with chain-of-thought reasoning. In Proceedings of the ACM on Web Conference 2025, pages 5342–5351, 2025. 9

    2025

  48. [56]

    Urbench: A comprehensive bench- mark for evaluating large multimodal models in multi-view urban scenarios

    Baichuan Zhou, Haote Yang, Dairong Chen, Junyan Ye, Tianyi Bai, Jinhua Yu, Songyang Zhang, Dahua Lin, Con- ghui He, and Weijia Li. Urbench: A comprehensive bench- mark for evaluating large multimodal models in multi-view urban scenarios. In Proceedings of the AAAI Conference o...

    2025

  49. [57]

    Deep learning for cross-domain data fu- sion in urban computing: Taxonomy, advances, and outlook

    Xingchen Zou, Yibo Yan, Xixuan Hao, Yuehong Hu, Haomin Wen, Erdong Liu, Junbo Zhang, Yong Li, Tianrui Li, Yu Zheng, et al. Deep learning for cross-domain data fu- sion in urban computing: Taxonomy, advances, and outlook. Information Fusion, 113:102606, 2025. 1, 2, 3, 9 11 Urba...

    2025

  50. [58]

    Figure 9

    Data Example for Three Stages Below, we provide examples for each stage. Figure 9. Input data examples for three-stage training

  51. [59]

    Comparing with models for single-modality urban tasks We focus on comparing our approach with these modality- specific models to better showcase its effectiveness, which are presented in following Table 5. Table 5. Comparing with models for single-modality urban tasks. GeoQA S...

  52. [60]

    Table 2 in Section 3.2 is the aggregated results of these three tables

    Additional Detailed Results of Three Cities The detailed results of UBench on three cities are presented in Table 6, Table 7 and Table 8. Table 2 in Section 3.2 is the aggregated results of these three tables. Urban- LLaVA@Beijing surpasses the baselines in all tasks, show- ca...

  53. [61]

    Additional Results for Training Strategies As a supplement to results in Section 3.3, we report results on learning rate, modality and trained components here. We first adjusted some experimental parameters to ex- plore their effects, and ultimately found that the learning rat...

  54. [62]

    11 presents training results with different amounts, ex- hibiting the high quality of UData

    Effects of Training Data Size Fig. 11 presents training results with different amounts, ex- hibiting the high quality of UData

  55. [63]

    Effects of Base Model Our method is model-agnostic and can be generalized to different MLLMs, e.g., Qwen2.5-VL-7B in Table 9

  56. [64]

    However, for certain tasks, models of different sizes exhibit similar capabilities

    Effects of Model Size As Figure 12 shows, performance generally improves with increasing parameter size for VILA 1.5 (from 3B to 13B). However, for certain tasks, models of different sizes exhibit similar capabilities. This occurs either because the tasks are inherently challe...

  57. [65]

    This task needs a model to speculate the land use type (commercial, residential, agricultural, etc.) based on a satellite image

    Additional Case Study SA T-LandUse. This task needs a model to speculate the land use type (commercial, residential, agricultural, etc.) based on a satellite image. One example is shown in Fig- ure 13. Our UrbanLLaVA can respond to this task precisely, showing the capability o...

    1920

  58. [66]

    Urban Instruction Data Table 10 provides detailed statistics of UData across three cities, while Table 11 presents the detailed statistics of the raw data used to construct UData. Additionally, we present 3 Prompt: The following is a multiple-choice question about selecting th...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.