Pith. sign in

REVIEW 5 cited by

STBench: Assessing the Ability of Large Language Models in Spatio-Temporal Analysis

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.19065 v1 pith:MAXBGHBW submitted 2024-06-27 cs.CL

Wenbin Li , Di Yao , Ruibo Zhao , Wenjie Chen , Zijie Xu , Chengxue Luo , Chang Gong , Quanliang Jing
show 2 more authors
Haining Tan Jingping Bi
This is my paper
classification cs.CL
keywords spatio-temporalllmslanguagestbenchtasksknowledgemodelsassessing
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The rapid evolution of large language models (LLMs) holds promise for reforming the methodology of spatio-temporal data mining. However, current works for evaluating the spatio-temporal understanding capability of LLMs are somewhat limited and biased. These works either fail to incorporate the latest language models or only focus on assessing the memorized spatio-temporal knowledge. To address this gap, this paper dissects LLMs' capability of spatio-temporal data into four distinct dimensions: knowledge comprehension, spatio-temporal reasoning, accurate computation, and downstream applications. We curate several natural language question-answer tasks for each category and build the benchmark dataset, namely STBench, containing 13 distinct tasks and over 60,000 QA pairs. Moreover, we have assessed the capabilities of 13 LLMs, such as GPT-4o, Gemma and Mistral. Experimental results reveal that existing LLMs show remarkable performance on knowledge comprehension and spatio-temporal reasoning tasks, with potential for further enhancement on other tasks through in-context learning, chain-of-though prompting, and fine-tuning. The code and datasets of STBench are released on https://github.com/LwbXc/STBench.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Can LLMs Learn to Map the World from Local Descriptions?

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A 0.5B LLM trained on templated local descriptions from a synthetic grid city infers unseen distances and directions, encodes coordinates in its hidden states, and plans shortest paths, but fails under navigation pert...

  2. USTBench: Benchmarking and Dissecting Spatiotemporal Reasoning of LLMs as Urban Agents

    cs.AI 2025-05 conditional novelty 6.0 of 10

    USTBench is the first benchmark that decomposes urban spatiotemporal reasoning into understanding, forecasting, planning, and reflection, and shows LLMs struggle most with planning and reflection.

  3. A Modular Multitask Reasoning Framework Integrating Spatio-temporal Models and LLMs

    cs.CL 2025-06 conditional novelty 5.0 of 10

    STReason uses in-context learning to convert spatio-temporal queries into executable programs with specialized modules, outperforming plain LLMs on a new 150-query benchmark.

  4. Large Language Model Powered Intelligent Urban Agents: Concepts, Capabilities, and Applications

    cs.MA 2025-07 conditional novelty 4.0 of 10

    The paper defines urban LLM agents, surveys their sensing, memory, reasoning, execution, and learning workflows, and organizes their applications across planning, transportation, environment, safety, and society.

  5. Domain Specific Benchmarks for Evaluating Multimodal Large Language Models

    cs.LG 2025-06 conditional novelty 3.0 of 10

    A review paper that organizes domain-specific MLLM benchmarks into an eight-discipline taxonomy, with summary tables and performance highlights.

Pith tools