Pith. sign in

REVIEW 5 major objections 5 minor 78 references

USTBench: Benchmarking and Dissecting Spatiotemporal Reasoning of LLMs as Urban Agents

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs can read a city but not think ahead in it: USTBench decomposes urban spatiotemporal reasoning into understanding, forecasting, planning, and reflection, and finds the higher-order abilities lagging.

desk verdict USTBench is a genuinely useful diagnostic resource, but its headline planning/reflection deficit rests on a simulator reward model that the paper never validates. read the letter →

arxiv 2505.17572 v1 pith:4I32KD7B submitted 2025-05-23 cs.AI

Siqi Lai , Yansong Ning , Zirui Yuan , Zhixi Chen , Hao Liu This is my paper
classification cs.AI
keywords spatiotemporalreasoningurbanagentslargelanguagemodelsbenchmarkplanningreflectionUAgentEnv
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

USTBench sets out to measure, for the first time, the spatiotemporal reasoning of large language models acting as urban agents, rather than only scoring the final outcome of urban tasks. The authors decompose this ability into four processes: understanding, forecasting, planning, and reflection with feedback, and build an interactive city environment, UAgentEnv, that generates 62,466 structured QA pairs from nine real-world urban tasks. Across thirteen leading LLMs, the benchmark finds that models are comparatively strong at understanding and near-term forecasting but consistently weak at long-horizon planning and at learning from environmental feedback. This matters because traffic control, route planning, and urban planning agents depend on exactly those higher-order abilities to operate in dynamic cities. The paper argues that this process-level view explains outcome-level failures that accuracy-based evaluations miss.

What carries the argument

The load-bearing object is UAgentEnv, an interactive city environment that verbalizes geospatial structures (road networks, adjacency, connectivity) and temporal dynamics (traffic flow, trajectories, socio-economic series) as textual observations with data schemas. On top of it, USTBench builds four QA families: understanding QAs over eight spatial and temporal pattern types; forecasting QAs from real-world prediction tasks; planning QAs whose ground truth is computed by a feedback-driven exploratory process that rolls out all action sequences over a horizon of $H=5$ with discount factor $\gamma=0.9$ and picks the action with the highest expected cumulative reward; and reflection QAs that pair a previous action or prediction with environmental feedback. This machinery converts outcome-level urban tasks into process-level diagnostics.

What would settle it

Recompute the planning ground truth with a one-step greedy policy ($H=1$) and with the full $H=5$ rollouts; if the chosen optimal actions coincide on most QA instances, the benchmark is not actually measuring long-horizon planning. Alternatively, replay the selected actions in a high-fidelity traffic simulator or on logged real-world data; if the $H=5$ choices are not better than simple heuristics on real outcomes, the planning scores are an artifact of UAgentEnv's reward model.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that LLM spatiotemporal reasoning is not a single capability but a stack with an uneven profile. When the four processes are measured separately, current LLMs reach above-chance and often high accuracy on spatial and temporal understanding and on short-horizon forecasting, but planning accuracy drops substantially and reflection-with-feedback accuracy stays below 50 percent for most models. The paper presents planning as a higher-order ability that presupposes understanding and forecasting, and reflection as the mechanism an agent needs to adapt as urban dynamics shift. It further reports that reasoning-specialized models such as DeepSeek-R1 do not consistently beat generalist LLMs on these urban tasks, which the authors read as evidence that general mathematical and logical post-training does not automatically transfer to urban spatiotemporal reasoning.

Load-bearing premise

The planning and reflection scores stand on UAgentEnv's simulator being a faithful stand-in for real city dynamics, because the ground-truth optimal actions come from rollouts in that simulator and would be biased if its transition and reward models do not match a real city.

Editorial extensions

If this is right

  • An urban LLM deployment that only checks final metrics may report success while missing systematic planning and reflection failures; USTBench's process QA can expose them.
  • Reasoning-intensive post-training on math and logic should not be assumed to improve urban agents; domain-specific spatiotemporal data is a more direct route, as the paper's post-training experiment on Qwen2.5-7B indicates.
  • Long-horizon planning is a bottleneck: models that forecast well still choose actions poorly when the objective spans multiple steps.
  • Reflection with feedback is measurable and currently weak; agents that cannot convert environmental feedback into corrected reasoning will not adapt to shifting urban conditions.
  • The 62,466 QA pairs and nine tasks give the community a standardized way to compare future urban LLM agents process by process.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The four-process decomposition is a general protocol: the same understanding, forecasting, planning, and reflection split could be applied to other sequential-decision settings such as delivery fleets or factory robotics, though the paper does not claim this.
  • A direct testable extension is that fine-tuning specifically on trend, periodicity, and chronology data, the weakest temporal skills, should improve forecasting more than equal-size general reasoning fine-tuning; the paper shows one supporting example but does not systematically compare.
  • The reflection results imply a production risk: adding reflection modules to agents with weak reflection ability can degrade performance, so such components should be gated or trained rather than always-on in deployed systems.
  • Because the planning ground truth comes from a simulator, benchmark scores upper-bound real planning skill only if the simulator reward matches real urban objectives; a real-world validation set would strengthen the result.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces USTBench, a benchmark for evaluating the spatiotemporal reasoning of LLMs as urban agents, together with an interactive city environment, UAgentEnv, that supports five decision-making and four prediction tasks. The benchmark decomposes reasoning into spatiotemporal understanding, forecasting, planning, and reflection with feedback, and evaluates thirteen LLMs on 62,466 (claimed) structured QA pairs as well as end-to-end task metrics. The central empirical claims are that LLMs perform reasonably on understanding and forecasting but struggle with long-horizon planning and reflective adaptation, and that general-purpose reasoning models do not consistently outperform non-reasoning models on urban tasks.

Significance. If validated, USTBench would be a useful contribution: it moves beyond outcome-only urban LLM evaluation, provides process-level diagnostics, covers multiple real-world task types, includes reflection with environment feedback, and reports results across a broad model suite. Strengths include the use of public data sources, explicit QA examples, confidence intervals for representative models, and a post-training experiment connecting understanding to downstream reasoning. The central finding that planning and reflection are bottlenecks is plausible and worth pursuing. However, the benchmark's validity rests on several methodological details that are currently underspecified or unvalidated, so the significance is conditional on those being resolved.

major comments (5)
  1. [Abstract, Section 4.1, Table 2] The stated total of 62,466 QA pairs is inconsistent with Table 2: the four category counts (27,000 + 15,336 + 15,000 + 8,130) sum to 65,466. Please correct the total or explain the discrepancy, since the benchmark scale is part of the paper's central claim.
  2. [Section 4.1.4, Eq. (2), Appendix E.2] The planning ground truth is computed as the action maximizing expected discounted reward in UAgentEnv with H=5 and gamma=0.9, but no sensitivity analysis or validation against an independent objective is reported. Because the headline planning and reflection deficits are measured against these labels, the paper should show that the rankings are stable under reasonable variations of H, gamma, and the reward model, and, if possible, that the proxy reward correlates with a task-level performance metric.
  3. [Section 4.1.2, Tables 14 and 16] The ground-truth labeling procedures for Trend and Periodicity QAs are not specified. Section 4.1.2 lists the pattern types but does not give the algorithm or criteria used to label a series as 'Increase first, then decrease' or to determine the period length, and the examples show options that are not obviously uniquely determined by the data. Please document the labeling protocol, including handling of ambiguous cases, and report label-quality checks such as multiple annotators or algorithmic verification.
  4. [Section 4.1.5] The construction and scoring of Reflection QA are underspecified. It is not stated whose previous action or prediction is used as the input, how the environmental feedback is generated for each QA, or how a model's reflection output is scored (e.g., whether both the is_correct judgment and the revised answer must be correct). This matters directly for the claimed reflection deficit, so the protocol needs to be defined precisely.
  5. [Section 5.3] The reported 'relative performance improvements of up to 337.31% in forecasting accuracy and 53.48% in decision outcomes' are not traced to a specific baseline or table row. Please provide the exact comparison points so readers can verify the magnitude of the claimed improvements.
minor comments (5)
  1. [Table 3] The heading 'DeepSeek-R1-0.4386' and the value '0.67172' appear to be formatting or typographical errors; please clean up the table.
  2. [Appendix G.4.3, Figures 8–10] Several example figures contain model responses copied from a different QA; for instance, Figure 8 shows a distance-ranking response under an adjacency question. These duplicated or mismatched examples should be replaced with correct outputs.
  3. [Appendix C] The runtime table mentions 'GPT-4o-mini' although the evaluated model list includes 'o4-mini'; please clarify which models were actually used.
  4. [Section 5.3] The phrases 'forecasting accuracy' and 'decision outcomes' should be defined with the exact metrics used, since Table 5 mixes MAPE, accuracy, cost, distance, service, and ecology scores.
  5. [Abstract and Section 1] Minor grammatical issues such as 'evaluating urban LLM agent' should be corrected throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: USTBench's reasoning QA labels come from real data and an explicit simulator rollout objective, not from model-fitted parameters; cited prior work is non-load-bearing.

full rationale

USTBench is an evaluation benchmark rather than a fitted derivation. The forecasting QA labels are read directly from real-world historical data (Section 4.1.3), the planning labels are produced by an explicit feedback-driven rollout over the UAgentEnv simulator with stated H=5 and gamma=0.9 (Section 4.1.4, Eq. 2), and the reflection labels pair prior model outputs with environmental feedback (Section 4.1.5). No parameter of UAgentEnv or any reward function is fitted to the thirteen LLMs being evaluated, so the reported planning and reflection deficits are measurements against an externally specified objective rather than reductions to the models' own outputs. The benchmark's self-citations (e.g., LLMLight [21] and CoLLMLight [63] for traffic-signal-control settings) are configuration references in the environment construction and are not load-bearing for the central claim about LLM reasoning abilities. The paper's own limitation statement that evaluations are conducted mainly in simulated environments and that real-world validation is needed (Section 7) is a validity caveat about the proxy reward model, not evidence of circularity. No load-bearing step reduces to its own inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the fidelity of the simulated environments, the correctness of the oracle-derived planning ground truths, and the validity of multiple-choice accuracy as a reasoning measure. These are domain assumptions rather than fitted parameters. No new physical or conceptual entities are postulated.

free parameters (4)
  • Exploration coefficient epsilon = 0.1
    Chosen to control diversity of decision-making observations in QA collection; affects scenario coverage but is not fitted to model outcomes.
  • Planning horizon H = 5
    Used in the oracle to compute ground-truth planning actions; hand-chosen and affects which actions are labeled optimal.
  • Discount factor gamma = 0.9
    Used in the oracle reward accumulation; hand-chosen and influences the relative weight of future rewards in planning ground truth.
  • Observation and prediction window sizes = task-dependent (e.g., 12-step traffic, 6-year GDP)
    Window lengths for forecasting QAs are chosen per task and affect what information the model sees and what prediction is asked.
assumptions (4)
  • domain assumption The UAgentEnv simulators (CityFlow, SUMO, etc.) faithfully represent real urban dynamics and reward functions.
    Planning and reflection ground truths are derived from these simulations; if the simulators are unrealistic, benchmark scores do not transfer to real cities.
  • domain assumption The ground-truth labels for Trend, Periodicity, and other pattern QAs are unambiguous and correctly derived from the raw time series.
    No formal labeling rule is specified in Section 4.1.2; subjective labels would undermine the understanding scores.
  • domain assumption Multiple-choice accuracy is a valid measure of reasoning ability.
    The entire process-based evaluation uses fixed-option questions, assuming that correct option selection reflects the underlying reasoning process.
  • domain assumption LLMs are evaluated under a fixed temperature and single prompt templates, so results are representative of each model's ability.
    Section G.2 specifies temperature 0.1; prompt sensitivity is not studied, so results may vary with different prompting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of USTBench: Benchmarking and Dissecting Spatiotemporal Reasoning of LLMs as Urban Agents." pith.science (2026). https://pith.science/paper/4I32KD7B

@misc{pith2026250517572,
  author       = {Pith},
  title        = {Pith review of: USTBench: Benchmarking and Dissecting Spatiotemporal Reasoning of LLMs as Urban Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4I32KD7B}},
  note         = {Machine review of arXiv:2505.17572}
}
read the original abstract

Large language models (LLMs) have shown emerging potential in spatiotemporal reasoning, making them promising candidates for building urban agents that support diverse urban downstream applications. Despite these benefits, existing studies primarily focus on evaluating urban LLM agent on outcome-level metrics (e.g., prediction accuracy, traffic efficiency), offering limited insight into their underlying reasoning processes. As a result, the strengths and limitations of urban LLM agents in spatiotemporal reasoning remain poorly understood. To this end, we introduce USTBench, the first benchmark to evaluate LLMs' spatiotemporal reasoning abilities as urban agents across four decomposed dimensions: spatiotemporal understanding, forecasting, planning, and reflection with feedback. Specifically, USTBench supports five diverse urban decision-making and four spatiotemporal prediction tasks, all running within our constructed interactive city environment UAgentEnv. The benchmark includes 62,466 structured QA pairs for process-level evaluation and standardized end-to-end task assessments, enabling fine-grained diagnostics and broad task-level comparison across diverse urban scenarios. Through extensive evaluation of thirteen leading LLMs, we reveal that although LLMs show promising potential across various urban downstream tasks, they still struggle in long-horizon planning and reflective adaptation in dynamic urban contexts. Notably, recent advanced reasoning models (e.g., DeepSeek-R1) trained on general logic or mathematical problems do not consistently outperform non-reasoning LLMs. This discrepancy highlights the need for domain-specialized adaptation methods to enhance urban spatiotemporal reasoning. Overall, USTBench provides a foundation to build more adaptive and effective LLM-based urban agents and broad smart city applications.

Figures

Figures reproduced from arXiv: 2505.17572 by the authors.

Figure 1
Figure 1. The comparison of outcome-based and process-based evaluations. In recent literature, many efforts have been made to evaluate the spatiotemporal reasoning ability of LLMs as urban agents. However, as sum￾marized in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The performance of leading LLMs in urban spatiotemporal reasoning. Using USTBench, we evaluate thirteen state￾of-the-art LLMs, covering both non-reasoning and reasoning models. Our key contributions and findings are summarized as: (1) We con￾struct USTBench, the first benchmark explicitly designed to evaluate the spatiotemporal reason￾ing capabilities of LLMs as intelligent urban agents. It combines both fine-graine… view at source ↗
Figure 3
Figure 3. The workflow of UAgentEnv environment [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: The performance of the model with enhanced spatiotemporal understanding abilities. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The performance of LLM agents with or without the reflection mechanism. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: The token usage vs. score. In our process-based spatiotemporal reasoning evaluation, we observe that the non-reasoning model GPT-4o is comparable, and in some cases surpasses, reasoning models. To further ex￾plore this, we conduct a cost-effectiveness anal￾ysis ( [PIT…
Figure 7
Figure 7. Figure 7: The LLMs’ reasoning on distance relation analysis. [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: The LLMs’ reasoning on adjacency relation analysis. [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 10
Figure 10. Figure 10: The LLMs’ reasoning on duration pattern analysis. [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 12
Figure 12. Figure 12: The LLMs’ reasoning on trend pattern analysis. [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 14
Figure 14. Figure 14: The LLMs’ reasoning on periodicity pattern analysis. [PITH_FULL_IMAGE:figures/full_fig_p027_14.png]
Figure 16
Figure 16. Figure 16: The LLMs’ reasoning on planning [PITH_FULL_IMAGE:figures/full_fig_p029_16.png]
Figure 17
Figure 17. Figure 17: The LLMs’ reasoning on reflection. G.4.4 Repetition Issue Road Planning But the question is about connecting the largest number of unconnected regions. So, building any of the roads that connect two unconnected regions would be better than building road 7. But the opt…
Figure 18
Figure 18. Figure 18: The repetition issues of DeepSeek-R1-Distill-Qwen-7B. [PITH_FULL_IMAGE:figures/full_fig_p031_18.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

78 extracted references · 42 canonical work pages

  1. [1]

    Smart sustainable cities of the future: An extensive interdisciplinary literature review.Sustainable cities and society, 31:183–212, 2017

    Simon Elias Bibri and John Krogstie. Smart sustainable cities of the future: An extensive interdisciplinary literature review.Sustainable cities and society, 31:183–212, 2017

    work page 2017

  2. [2]

    Box and jenkins: time series analysis, forecasting and control

    George Box. Box and jenkins: time series analysis, forecasting and control. InA Very British Affair: Six Britons and the Development of Time Series Analysis During the 20th Century, pages 161–215. Springer

    work page

  3. [3]

    TEMPO: Prompt-based generative pre-trained transformer for time series forecasting

    Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. TEMPO: Prompt-based generative pre-trained transformer for time series forecasting. InThe Twelfth International Conference on Learning Representations, 2024

    work page 2024

  4. [4]

    Agentboard: An analytical evaluation board of multi-turn llm agents.Advances in Neural Information Processing Systems, 37:74325–74362, 2024

    Ma Chang, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn llm agents.Advances in Neural Information Processing Systems, 37:74325–74362, 2024

    work page 2024

  5. [5]

    Graphwiz: An instruction-following language model for graph computational problems

    Nuo Chen, Yuhan Li, Jianheng Tang, and Jia Li. Graphwiz: An instruction-following language model for graph computational problems. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 353–364, 2024

    2024

  6. [6]

    Deeptransport: Learning spatial-temporal dependency for traffic condition forecasting

    Xingyi Cheng, Ruiqing Zhang, Jie Zhou, and Wei Xu. Deeptransport: Learning spatial-temporal dependency for traffic condition forecasting. In2018 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2018

    work page 2018

  7. [7]

    Timebench: A comprehensive evaluation of temporal reasoning abilities in large language models.arXiv preprint arXiv:2311.17667, 2023

    Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Haotian Wang, Ming Liu, and Bing Qin. Timebench: A comprehensive evaluation of temporal reasoning abilities in large language models.arXiv preprint arXiv:2311.17667, 2023

    arXiv 2023

  8. [8]

    On the evolution of random graphs.Publ

    Paul Erdos, Alfréd Rényi, et al. On the evolution of random graphs.Publ. math. inst. hung. acad. sci, 5(1):17–60, 1960

    work page 1960

Show all 78 references
  1. [9]

    Test of time: A benchmark for evaluating llms on temporal reasoning.arXiv preprint arXiv:2406.09170, 2024

    Bahare Fatemi, Mehran Kazemi, Anton Tsitsulin, Karishma Malkan, Jinyeong Yim, John Palowitch, Sungyong Seo, Jonathan Halcrow, and Bryan Perozzi. Test of time: A benchmark for evaluating llms on temporal reasoning.arXiv preprint arXiv:2406.09170, 2024

    2024 arXiv

  2. [10]

    Citygpt: Empowering urban spatial cognition of large language models.arXiv preprint arXiv:2406.13948, 2024

    Jie Feng, Yuwei Du, Tianhui Liu, Siqi Guo, Yuming Lin, and Yong Li. Citygpt: Empowering urban spatial cognition of large language models.arXiv preprint arXiv:2406.13948, 2024

    2024 arXiv

  3. [11]

    Citybench: Evaluating the capabilities of large language model as world model.arXiv preprint arXiv:2406.13945, 2024

    Jie Feng, Jun Zhang, Junbo Yan, Xin Zhang, Tianjian Ouyang, Tianhui Liu, Yuwei Du, Siqi Guo, and Yong Li. Citybench: Evaluating the capabilities of large language model as world model.arXiv preprint arXiv:2406.13945, 2024

    2024 arXiv

  4. [12]

    Pygad: An intuitive genetic algorithm python library.Multimedia tools and applications, 83(20):58029–58042, 2024

    Ahmed Fawzy Gad. Pygad: An intuitive genetic algorithm python library.Multimedia tools and applications, 83(20):58029–58042, 2024

    2024

  5. [13]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools.arXiv preprint arXiv:2406.12793, 2024

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools.arXiv preprint arXiv:2406.12793, 2024

    2024 arXiv

  6. [14]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  7. [15]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  8. [16]

    Language models represent space and time.arXiv preprint arXiv:2310.02207, 2023

    Wes Gurnee and Max Tegmark. Language models represent space and time.arXiv preprint arXiv:2310.02207, 2023

    2023 arXiv

  9. [17]

    The scoot on-line traffic signal optimisation technique.Traffic Engineering & Control, 23(4), 1982

    PB Hunt, DI Robertson, RD Bretherton, and M Cr Royle. The scoot on-line traffic signal optimisation technique.Traffic Engineering & Control, 23(4), 1982. 10

    1982

  10. [18]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  11. [19]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  12. [20]

    Towards mitigating LLM hallucination via self reflection

    Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating LLM hallucination via self reflection. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, Decembe...

    2023

  13. [21]

    Llmlight: Large language models as traffic signal control agents

    Siqi Lai, Zhao Xu, Weijia Zhang, Hao Liu, and Hui Xiong. Llmlight: Large language models as traffic signal control agents. InProceedings of the 31th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2025

    2025

  14. [22]

    Reframing spatial reasoning evaluation in language models: A real-world simulation benchmark for qualitative reasoning.arXiv preprint arXiv:2405.15064, 2024

    Fangjun Li, David C Hogg, and Anthony G Cohn. Reframing spatial reasoning evaluation in language models: A real-world simulation benchmark for qualitative reasoning.arXiv preprint arXiv:2405.15064, 2024

    2024 arXiv

  15. [23]

    Repetition in repetition out: Towards understanding neural text degeneration from the data perspective.Advances in Neural Information Processing Systems, 36:72888–72903, 2023

    Huayang Li, Tian Lan, Zihao Fu, Deng Cai, Lemao Liu, Nigel Collier, Taro Watanabe, and Yixuan Su. Repetition in repetition out: Towards understanding neural text degeneration from the data perspective.Advances in Neural Information Processing Systems, 36:72888–72903, 2023

    2023

  16. [24]

    Towards alleviating traffic congestion: Optimal route planning for massive-scale trips

    Ke Li, Lisi Chen, and Shuo Shang. Towards alleviating traffic congestion: Optimal route planning for massive-scale trips. InProceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 3400–3406, 2021

    2021

  17. [25]

    Stbench: Assessing the ability of large language models in spatio-temporal analysis.arXiv preprint arXiv:2406.19065, 2024

    Wenbin Li, Di Yao, Ruibo Zhao, Wenjie Chen, Zijie Xu, Chengxue Luo, Chang Gong, Quanliang Jing, Haining Tan, and Jingping Bi. Stbench: Assessing the ability of large language models in spatio-temporal analysis.arXiv preprint arXiv:2406.19065, 2024

    2024 arXiv

  18. [26]

    Urbangpt: Spatio-temporal large language models

    Zhonghang Li, Lianghao Xia, Jiabin Tang, Yong Xu, Lei Shi, Long Xia, Dawei Yin, and Chao Huang. Urbangpt: Spatio-temporal large language models. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5351–5362, 2024

    2024

  19. [27]

    Timecma: Towards llm-empowered multivariate time series forecasting via cross-modality alignment

    Chenxi Liu, Qianxiong Xu, Hao Miao, Sun Yang, Lingzheng Zhang, Cheng Long, Ziyue Li, and Rui Zhao. Timecma: Towards llm-empowered multivariate time series forecasting via cross-modality alignment. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, page...

    2025

  20. [28]

    Knowledge-infused contrastive learning for urban imagery-based socioeconomic prediction

    Yu Liu, Xin Zhang, Jingtao Ding, Yanxin Xi, and Yong Li. Knowledge-infused contrastive learning for urban imagery-based socioeconomic prediction. InProceedings of the ACM web conference 2023, pages 4150–4160, 2023

    2023

  21. [29]

    Simulation of urban mobility (sumo), February 4 2025

    Pablo Alvarez Lopez, Angelo Banse, Mirko Barthauer, Michael Behrisch, Benjamin Couéraud, Jakob Erdmann, Yun-Pang Flötteröd, Robert Hilbrich, Ronald Nippold, and Peter Wagner. Simulation of urban mobility (sumo), February 4 2025

    2025

  22. [30]

    Scats, sydney co-ordinated adaptive traffic system: A traffic responsive method of controlling urban traffic

    PR Lowrie. Scats, sydney co-ordinated adaptive traffic system: A traffic responsive method of controlling urban traffic. 1990

    1990

  23. [31]

    Spartqa:: A textual question answering benchmark for spatial reasoning.arXiv preprint arXiv:2104.05832, 2021

    Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjmashidi. Spartqa:: A textual question answering benchmark for spatial reasoning.arXiv preprint arXiv:2104.05832, 2021

    2021 arXiv

  24. [32]

    Transfer learning with synthetic corpora for spatial role labeling and reasoning.arXiv preprint arXiv:2210.16952, 2022

    Roshanak Mirzaee and Parisa Kordjamshidi. Transfer learning with synthetic corpora for spatial role labeling and reasoning.arXiv preprint arXiv:2210.16952, 2022

    2022 arXiv

  25. [33]

    Towards understanding the spatial literacy of chatgpt

    PETER MOONEY , WENCONG CUI, BOYUAN GUAN, and LEVENTE JUHÁSZ. Towards understanding the spatial literacy of chatgpt. InACM SIGSPATIAL International Conference, 2023. 11

    2023

  26. [34]

    Tlc trip record data, 2025

    New York City Taxi and Limousine Commission. Tlc trip record data, 2025. Accessed: 2025-05-08

    2025

  27. [35]

    Dima: An llm-powered ride-hailing assistant at didi.arXiv preprint arXiv:2503.04768, 2025

    Yansong Ning, Shuowei Cai, Wei Li, Jun Fang, Naiqiang Tan, Hua Chai, and Hao Liu. Dima: An llm-powered ride-hailing assistant at didi.arXiv preprint arXiv:2503.04768, 2025

    2025

  28. [36]

    Urbankgent: A unified large language model agent framework for urban knowledge graph construction.arXiv preprint arXiv:2402.06861, 2024

    Yansong Ning and Hao Liu. Urbankgent: A unified large language model agent framework for urban knowledge graph construction.arXiv preprint arXiv:2402.06861, 2024

    2024 arXiv

  29. [37]

    Openstreetmap planet data, 2025

    OpenStreetMap contributors. Openstreetmap planet data, 2025. Accessed: 2025-03-31

    2025

  30. [38]

    Self-reflection in llm agents: Effects on problem-solving performance.arXiv preprint arXiv:2405.06682, 2024

    Matthew Renze and Erhan Guven. Self-reflection in llm agents: Effects on problem-solving performance.arXiv preprint arXiv:2405.06682, 2024

    2024 arXiv

  31. [39]

    Sparc and sparp: Spatial reasoning char- acterization and path generation for understanding spatial reasoning capability of large language models

    Md Imbesat Rizvi, Xiaodan Zhu, and Iryna Gurevych. Sparc and sparp: Spatial reasoning char- acterization and path generation for understanding spatial reasoning capability of large language models. InProceedings of the 62nd Annual Meeting of the Association for Computational L...

    2024

  32. [40]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  33. [41]

    Stepgame: A new benchmark for robust multi- hop spatial reasoning in texts

    Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. Stepgame: A new benchmark for robust multi- hop spatial reasoning in texts. InProceedings of the AAAI conference on artificial intelligence, volume 36, pages 11321–11329, 2022

    2022

  34. [42]

    Towards benchmarking and improving the temporal reasoning capability of large language models.arXiv preprint arXiv:2306.08952, 2023

    Qingyu Tan, Hwee Tou Ng, and Lidong Bing. Towards benchmarking and improving the temporal reasoning capability of large language models.arXiv preprint arXiv:2306.08952, 2023

    2023 arXiv

  35. [43]

    Cityflow: A city-scale benchmark for multi-target multi-camera vehicle tracking and re-identification

    Zheng Tang, Milind Naphade, Ming-Yu Liu, Xiaodong Yang, Stan Birchfield, Shuo Wang, Ratnesh Kumar, David Anastasiu, and Jenq-Neng Hwang. Cityflow: A city-scale benchmark for multi-target multi-camera vehicle tracking and re-identification. InProceedings of the IEEE/CVF confere...

    2019

  36. [44]

    Qwq-32b: Embracing the power of reinforcement learning, 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, 2025. Accessed: 2025-03-26

    2025

  37. [45]

    Air quality prediction with physics-guided dual neural odes in open systems.ICLR, 2025

    Jindong Tian, Yuxuan Liang, Ronghui Xu, Peng Chen, Chenjuan Guo, Aoying Zhou, Lujia Pan, Zhongwen Rao, and Bin Yang. Air quality prediction with physics-guided dual neural odes in open systems.ICLR, 2025

    2025

  38. [46]

    Applications of artificial intelligence and machine learning in smart cities.Computer Communications, 154:313– 323, 2020

    Zaib Ullah, Fadi Al-Turjman, Leonardo Mostarda, and Roberto Gagliardi. Applications of artificial intelligence and machine learning in smart cities.Computer Communications, 154:313– 323, 2020

    2020

  39. [47]

    Robust extrema features for time-series data analysis.IEEE transactions on pattern analysis and machine intelligence, 35(6):1464–1479, 2012

    Pramod K Vemulapalli, Vishal Monga, and Sean N Brennan. Robust extrema features for time-series data analysis.IEEE transactions on pattern analysis and machine intelligence, 35(6):1464–1479, 2012

    2012

  40. [48]

    Reinforcement learning-based placement of charging stations in urban road networks

    Leonie von Wahl, Nicolas Tempelmeier, Ashutosh Sao, and Elena Demidova. Reinforcement learning-based placement of charging stations in urban road networks. InProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3992–4000, 2022

    2022

  41. [49]

    A survey on large language model based autonomous agents.Frontiers of Computer Science, 18(6):186345, 2024

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents.Frontiers of Computer Science, 18(6):186345, 2024

    2024

  42. [50]

    Global gridded gdp data set consistent with the shared socioeco- nomic pathways.Scientific data, 9(1):221, 2022

    Tingting Wang and Fubao Sun. Global gridded gdp data set consistent with the shared socioeco- nomic pathways.Scientific data, 9(1):221, 2022

    2022

  43. [51]

    Where would i go next? large language models as human mobility predictors.arXiv preprint arXiv:2308.15197, 2023

    Xinglei Wang, Meng Fang, Zichao Zeng, and Tao Cheng. Where would i go next? large language models as human mobility predictors.arXiv preprint arXiv:2308.15197, 2023. 12

    2023 arXiv

  44. [52]

    From news to forecast: Integrating event analysis in llm-based time series forecasting with reflection.Advances in Neural Information Processing Systems, 37:58118–58153, 2024

    Xinlei Wang, Maike Feng, Jing Qiu, Jinjin Gu, and Junhua Zhao. From news to forecast: Integrating event analysis in llm-based time series forecasting with reflection.Advances in Neural Information Processing Systems, 37:58118–58153, 2024

    2024

  45. [53]

    Tram: Benchmarking temporal reasoning for large language models

    Yuqing Wang and Yun Zhao. Tram: Benchmarking temporal reasoning for large language models. InFindings of the Association for Computational Linguistics ACL 2024, pages 6389– 6415, 2024

    2024

  46. [54]

    Colight: Learning network-level cooperation for traffic signal control

    Hua Wei, Nan Xu, Huichu Zhang, Guanjie Zheng, Xinshi Zang, Chacha Chen, Weinan Zhang, Yanmin Zhu, Kai Xu, and Zhenhui Li. Colight: Learning network-level cooperation for traffic signal control. InProceedings of the 28th ACM international conference on information and knowledge...

    1913

  47. [55]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  48. [56]

    Coverage location models: alternatives, approximation, and uncertainty.International Regional Science Review, 39(1):48–76, 2016

    Ran Wei. Coverage location models: alternatives, approximation, and uncertainty.International Regional Science Review, 39(1):48–76, 2016

    2016

  49. [57]

    Worldpop hub, 2025

    WorldPop. Worldpop hub, 2025

    2025

  50. [58]

    Large language models can learn temporal reasoning

    Siheng Xiong, Ali Payani, Ramana Kompella, and Faramarz Fekri. Large language models can learn temporal reasoning. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10452–10470, 2024

    2024

  51. [59]

    Evaluating spatial understanding of large language models.arXiv preprint arXiv:2310.14540, 2023

    Yutaro Yamada, Yihan Bao, Andrew K Lampinen, Jungo Kasai, and Ilker Yildirim. Evaluating spatial understanding of large language models.arXiv preprint arXiv:2310.14540, 2023

    2023 arXiv

  52. [60]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  53. [61]

    Foursquare dataset

    Dingqi Yang. Foursquare dataset. https://sites.google.com/site/yangdingqi/home/ foursquare-dataset, 2014. Accessed: 2025-05-19

    2014

  54. [62]

    Unist: A prompt-empowered universal model for urban spatio-temporal prediction

    Yuan Yuan, Jingtao Ding, Jie Feng, Depeng Jin, and Yong Li. Unist: A prompt-empowered universal model for urban spatio-temporal prediction. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4095–4106, 2024

    2024

  55. [63]

    Collmlight: Cooperative large language model agents for network-wide traffic signal control.arXiv preprint arXiv:2503.11739, 2025

    Zirui Yuan, Siqi Lai, and Hao Liu. Collmlight: Cooperative large language model agents for network-wide traffic signal control.arXiv preprint arXiv:2503.11739, 2025

    2025 arXiv

  56. [64]

    Agenttuning: Enabling generalized agent abilities for llms.arXiv preprint arXiv:2310.12823, 2023

    Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms.arXiv preprint arXiv:2310.12823, 2023

    2023 arXiv

  57. [65]

    Open3dvqa: A benchmark for comprehensive spatial reasoning with multimodal large language model in open space.arXiv preprint arXiv:2503.11094, 2025

    Weichen Zhan, Zile Zhou, Zhiheng Zheng, Chen Gao, Jinqiang Cui, Yong Li, Xinlei Chen, and Xiao-Ping Zhang. Open3dvqa: A benchmark for comprehensive spatial reasoning with multimodal large language model in open space.arXiv preprint arXiv:2503.11094, 2025

    2025

  58. [66]

    Reinforcement learning for traffic signal control

    Huichu Zhang, Siyuan Feng, Chang Liu, Yaoyao Ding, Yichen Zhu, Zihan Zhou, Weinan Zhang, Yong Yu, Haiming Jin, and Zhenhui Li. Reinforcement learning for traffic signal control. http://traffic-signal-control.github.io, 2023. Accessed on 1 December 2023

    2023

  59. [67]

    Urbanvideo-bench: Benchmarking vision- language models on embodied intelligence with video data in urban spaces.arXiv preprint arXiv:2503.06157, 2025

    Baining Zhao, Jianjie Fang, Zichao Dai, Ziyou Wang, Jirong Zha, Weichen Zhang, Chen Gao, Yue Wang, Jinqiang Cui, Xinlei Chen, et al. Urbanvideo-bench: Benchmarking vision- language models on embodied intelligence with video data in urban spaces.arXiv preprint arXiv:2503.06157, 2025

    2025

  60. [68]

    Where to go next: A spatio-temporal gated network for next poi recommendation.IEEE Transactions on Knowledge and Data Engineering, 34(5):2512–2524, 2020

    Pengpeng Zhao, Anjing Luo, Yanchi Liu, Jiajie Xu, Zhixu Li, Fuzhen Zhuang, Victor S Sheng, and Xiaofang Zhou. Where to go next: A spatio-temporal gated network for next poi recommendation.IEEE Transactions on Knowledge and Data Engineering, 34(5):2512–2524, 2020. 13

    2020

  61. [69]

    Cityeqa: A hierarchical llm agent on embodied question answering benchmark in city space.arXiv preprint arXiv:2502.12532, 2025

    Yong Zhao, Kai Xu, Zhengqiu Zhu, Yue Hu, Zhiheng Zheng, Yingfeng Chen, Yatai Ji, Chen Gao, Yong Li, and Jincai Huang. Cityeqa: A hierarchical llm agent on embodied question answering benchmark in city space.arXiv preprint arXiv:2502.12532, 2025

    2025 arXiv

  62. [70]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System ...

    2024

  63. [71]

    Spatial planning of urban communities via deep reinforcement learning.Nature Computational Science, 3(9):748– 762, 2023

    Yu Zheng, Yuming Lin, Liang Zhao, Tinghai Wu, Depeng Jin, and Yong Li. Spatial planning of urban communities via deep reinforcement learning.Nature Computational Science, 3(9):748– 762, 2023

    2023

  64. [72]

    Urbanplanbench: A comprehensive urban planning benchmark for evaluating large language models.arXiv preprint arXiv:2504.21027, 2025

    Yu Zheng, Longyi Liu, Yuming Lin, Jie Feng, Guozhen Zhang, Depeng Jin, and Yong Li. Urbanplanbench: A comprehensive urban planning benchmark for evaluating large language models.arXiv preprint arXiv:2504.21027, 2025

    2025 arXiv

  65. [73]

    Road planning for slums via deep reinforcement learning

    Yu Zheng, Hongyuan Su, Jingtao Ding, Depeng Jin, and Yong Li. Road planning for slums via deep reinforcement learning. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5695–5706, 2023

    2023

  66. [74]

    going on a vacation

    Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. “going on a vacation” takes longer than “going for a walk”: A study of temporal commonsense understanding. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Jo...

    2019

  67. [75]

    aha moments

    Zhilun Zhou, Yuming Lin, Depeng Jin, and Yong Li. Large language model for participatory urban planning.arXiv preprint arXiv:2402.17161, 2024. 14 A Limitations Exploration on Enhancement Methods: USTBench is designed primarily as a comprehensive benchmark to rigorously evaluat...

    2024 arXiv

  68. [76]

    answer":

    But even then, the total distance is way larger. The data doesn’t have any other connections from crosswalk 3 to other entities except school 1. Therefore, school 1 is directly adjacent and very close, making it the stronger candidate. Wait, but the problem is about which (sch...

  69. [77]

    Miscellaneous Shop

    is later. So the Day 30’s 6:02 PM is more recent than any entries in previous Sundays, because Day 30 is a later day. Therefore, the most recent event is Miscellaneous Shop 13944 at 06:02 PM on Sunday. So the answer would be "Miscellaneous Shop". Wait, in the data entry for Da...

  70. [78]

    answer":

    30.5 2. 30.25 (-0.25) 3. 29.8 (-0.45) 4. 29.7 (-0.1) 5. 29.45 (-0.25) 6. 28.2 (-1.25) 7. 27.05 (-1.15) 8. 26.65 (-0.4) 9. 26.4 (-0.25) 10. 25.9 (-0.5) 11. 25.35 (-0.55) 12. 24.6 (-0.75) So the rate of decrease varies. For instance, steps 6 and 7 have larger drops (like -1.25 a...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.