REVIEW 4 cited by
Can Large Language Models be Good Path Planners? A Benchmark and Investigation on Spatial-temporal Reasoning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Large language models (LLMs) have achieved remarkable success across a wide spectrum of tasks; however, they still face limitations in scenarios that demand long-term planning and spatial reasoning. To facilitate this line of research, in this work, we propose a new benchmark, termed $\textbf{P}$ath $\textbf{P}$lanning from $\textbf{N}$atural $\textbf{L}$anguage ($\textbf{PPNL}$). Our benchmark evaluates LLMs' spatial-temporal reasoning by formulating ''path planning'' tasks that require an LLM to navigate to target locations while avoiding obstacles and adhering to constraints. Leveraging this benchmark, we systematically investigate LLMs including GPT-4 via different few-shot prompting methodologies as well as BART and T5 of various sizes via fine-tuning. Our experimental results show the promise of few-shot GPT-4 in spatial reasoning, when it is prompted to reason and act interleavedly, although it still fails to perform long-term temporal reasoning. In contrast, while fine-tuned LLMs achieved impressive results on in-distribution reasoning tasks, they struggled to generalize to larger environments or environments with more obstacles.
Forward citations
Cited by 4 Pith papers
-
OmniRouting: A Semantic-Coupled Multimodal Benchmark for Constraint-Aware Spatial Reasoning in PCB Routing
The OmniRouting benchmark, with 1,681 PCB designs, shows current large multimodal models achieve under 13% clean net routability while humans reach about 94%, exposing major gaps in constraint-aware spatial reasoning.
-
MazeEval: A Benchmark for Testing Sequential Decision-Making in Language Models
A new maze-navigation benchmark claims LLM spatial reasoning is language-dependent, with O3 exceptional and other models failing by looping, but the looping result is an artifact of the termination rule.
-
Assessing the Value of Visual Input: A Benchmark of Multimodal Large Language Models for Robotic Path Planning
A benchmark of 15 multimodal LLMs on grid path planning reports modest success on 8x8 grids and near-failure on 20x20 grids, but its visual-vs-text comparison is confounded by prompt differences.
-
Application of LLMs to Multi-Robot Path Planning and Task Allocation
An LLM planner triggered by ensemble uncertainty improves a QMIX agent's performance in the SimpleSpread multi-agent task, though the supporting experiments lack error bars and quantitative evaluation.
Discussion (0). Sign in to comment.