Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios

T0 review · 4 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MobilityBench, a benchmark of 100,000 anonymized real mobility queries, shows that LLM route-planning agents handle basic queries well but fail far more often on preference-constrained routes.

desk verdict MobilityBench is a serious, potentially reusable benchmark with a sensible replay-sandbox design, but the missing FPR definition and unquantified fallback resolution make the exact results conditional until the artifacts are published. read the letter →

arxiv 2602.22638 v2 pith:B7MNYUT7 submitted 2026-02-26 cs.AI

Zhiheng Song , Jingshuai Zhang , Chuan Qin , Chao Wang , Chao Chen , Longfei Xu , Kaikui Liu , Xiangxiang Chu
show 1 more author
Hengshu Zhu
This is my paper
classification cs.AI
keywords LLMagentsrouteplanningbenchmarkAPIreplaysandboxpreference-constrainedroutingreproducibilitytoolusemobility
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MobilityBench is a benchmark for LLM-based route-planning agents, built from large-scale anonymized real user queries and designed to make evaluation reproducible despite live mapping services being non-deterministic. The paper's central claim is that, with a deterministic API-replay sandbox and a ground-truth protocol centered on outcome validity, agent performance can be measured fairly and decomposed into instruction understanding, planning, tool use, and decision-making. Using this setup, the paper evaluates multiple open- and closed-source LLMs under two agent frameworks and finds that models do well on basic information retrieval and basic route planning but struggle considerably with preference-constrained route planning. If this finding is right, the bottleneck for personalized mobility applications is not tool use or basic navigation but the ability to honor user constraints such as avoiding highways or minimizing transfers.

What carries the argument

The load-bearing mechanism is the deterministic API-replay sandbox. It intercepts every tool call an agent makes during evaluation and resolves it against a cache of responses captured when the ground truth was constructed, keyed by canonicalized arguments; exact hits are replayed, while misses fall back to fuzzy entity matching or nearest-neighbor spatial matching within a distance threshold. This freezes traffic, weather, and service state at data-collection time so that identical tool calls always yield identical outputs, removing environmental variance from live APIs. The second essential piece is the standard tool program — the minimal expert-defined sequence of tool calls that correctl

What would settle it

Pick the most time-sensitive scenarios (traffic-info query, arrival-time query) and replay the frozen cache months after collection; if an agent that trusts the cached 'current' traffic still passes while a freshly queried API would produce a materially different route, the sandbox is scoring state retrieval, not planning. Additionally, inspect cache-hit versus fallback rates per episode: an episode whose correct answer depends on a fuzzy match or nearest-neighbor fallback rather than an exact cached response is a direct place to test whether ground truth is truly grounded.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims to provide the first large-scale, reproducible benchmark for fine-grained route-planning agents in real-world mobility settings. Each episode is a four-tuple: an anonymized natural-language query, contextual information, a fixed replayable snapshot of relevant API responses, and a structured ground-truth annotation derived from an expert-defined standard tool program. The evaluation centers on Final Pass Rate — the share of queries for which an agent produces an executable solution satisfying all explicit and implicit constraints — and supplements it with metrics for intent detection, information extraction, task decomposition, tool selection, schema compli

Load-bearing premise

The benchmark's validity rests on the assumption that replaying API responses cached at data-collection time — with fuzzy or nearest-neighbor fallbacks for cache misses — yields the correct, contextually consistent answer for every episode; if a cached or fallback response is wrong for the query, the Final Pass Rate measures cache retrieval rather than route-planning ability.

Editorial extensions

If this is right

  • If MobilityBench is valid, route-planning agents can be compared fairly across LLMs and agent frameworks without confounding from live traffic or service variability.
  • The reported gap on preference-constrained route planning gives a concrete target: improving constraint honoring (avoid highways, minimize transfers, required waypoints) matters more than better retrieval or basic routing.
  • The metric decomposition allows researchers to see whether a failure is an intent-detection error, an information-extraction error, a planning error, a tool-call error, or a final-decision error, instead of a single pass/fail.
  • ReAct-style agents tended toward higher final pass rates than Plan-and-Execute, at roughly 35% higher input-token cost, implying a direct robustness-versus-cost trade-off for deployment.
  • Scaling model size and enabling thinking mode improved final pass rate in the paper's experiments, but both raised inference cost, so the paper points toward efficient small models as a practical need.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit: because cache misses are resolved by fuzzy or nearest-neighbor fallback, some 'successful' runs may be rewarded for retrieving a plausible cached response rather than for genuine route-planning skill; auditing fallback-match rates by scenario would test this.
  • The no-clarification assumption makes the benchmark intentionally harder than a production voice assistant; preference-constrained failure rates might shrink if agents were allowed one clarifying question.
  • The same replay-sandbox architecture could transfer to other non-deterministic API domains, such as flight, hotel, or ride-hail booking, where live prices and availability make reproducibility equally hard.
  • A natural extension is a time-shifted variant of the benchmark that changes the cached weather or traffic snapshot between query construction and evaluation, testing whether agents notice stale data rather than blindly trusting replayed responses.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces MobilityBench, a benchmark for evaluating LLM-based route-planning agents in real-world mobility scenarios. It is constructed from 100,000 anonymized real user queries from Amap, organized into 11 task scenarios across four intent families. To ensure reproducibility, the authors design a deterministic API-replay sandbox that caches responses from AMap Web Service APIs and serves them during evaluation. They propose a multi-dimensional evaluation protocol covering instruction understanding, planning, tool use, decision making, and efficiency, with a central outcome metric called Final Pass Rate (FPR). The paper evaluates a range of open- and closed-source LLMs under ReAct and Plan-and-Execute frameworks, reporting that current models perform well on basic tasks but struggle with preference-constrained route planning.

Significance. If the validity and reproducibility concerns are adequately addressed, MobilityBench would be a valuable contribution to the agent benchmarking literature. Its strengths include the large real-query scale, the explicit attempt to control environmental nondeterminism via a replay sandbox, the multi-dimensional evaluation protocol that goes beyond end-to-end success, and the stated public release of data, toolkit, and documentation. The empirical finding of a performance gap between basic and preference-constrained route planning is plausible and, if confirmed, useful for directing future research. However, the current manuscript omits key operational details that are load-bearing for interpreting the reported numbers, especially the definition and validation of the FPR metric and the behavior of the replay sandbox on cache misses.

major comments (4)
  1. [Section 3.2.4, Table 2] Final Pass Rate (FPR) is the central outcome metric for the paper's headline claims (e.g., 'struggle considerably with Preference-Constrained Route Planning'), yet it is never defined by an equation. The text states only that a solution is successful 'if it satisfies all user-specified explicit and implicit constraints.' This is insufficient: what constitutes 'satisfies'? Is it exact equality of route features, a tolerance-based comparison, or a semantic check? Without a formal definition, FPR is not reproducible, and the reported 60-70% numbers cannot be independently verified. Please provide the precise evaluation function, including how route plans, times, transfer counts, and constraint predicates are compared against ground truth.
  2. [Section 3.1.4; Section 4.2.1] The replay sandbox is the keystone of the benchmark's reproducibility claim, but the manuscript gives no statistics on how often exact cache hits occur versus fallback strategies (fuzzy matching, nearest-neighbor spatial matching) and no threshold values or validation that fallback responses are contextually equivalent to the frozen ground-truth responses. If a substantial fraction of agent calls are resolved via fallback, then the measured FPR conflates route-planning skill with whether the agent's tool-call syntax matches the SOP's canonicalization (e.g., using a POI name instead of coordinates, or a nonstandard time format). This concern is especially acute for Option-Constrained and Route-Constrained planning, where parameters are more complex. Please report cache-hit rates per scenario, the maximum distance threshold used, and ideally a human-validated sample of fallback resolutions
  3. [Section 3.2.1, Eq. (1); Section 3.2.2, Eq. (3); Section 3.2.3, Eq. (5)] Three evaluation indicators depend on unspecified components: the similarity threshold alpha_threshold in Eq. (1), the semantic match function f_DEC in Eq. (3), and the schema-compliance function f_SC in Eq. (5). None of these are defined or given values, and no reference is provided. The reproducibility claim of the paper is undermined because two independent implementations could choose different thresholds or matching functions and obtain different ID, DEC, and SC scores. Please specify these functions explicitly or, at minimum, provide the exact implementation used in the released toolkit.
  4. [Section 3.1.2; Section 3.1.3] The ground-truth construction pipeline uses Qwen-4B for open-set intent classification followed by 'expert adjudication,' and scenario-specific standard operating procedures (SOPs) are defined by domain experts. Yet no inter-annotator agreement, number of experts, or quality-assurance statistics are reported. The benchmark's validity rests on the assumption that the intent taxonomy is exhaustive and mutually exclusive and that the SOPs encode the 'correct' tool sequence. Please report the number of annotators, agreement measures, and the process for resolving disagreements. Without this, the reader cannot assess whether the 11-scenario taxonomy and the expert-defined ground truth are reliable.
minor comments (4)
  1. [Abstract and Section 1] The paper consistently uses 'route-planning' as a noun modifier; consider hyphenation consistency. Also, the example 'avoid tolls/highways' in Table 1 is listed under Option-Constrained, while the text uses 'Option-' and 'Route-Constrained' interchangeably in places; please unify terminology.
  2. [Section 3.2.4] The Delivery Rate (DR) is described as measuring 'complete and executable final output,' but it is not formally defined either. Please provide an equation or explicit criterion, especially since DR appears prominently in Table 2.
  3. [Section 4.2.3] The scaling effect analysis reports improvements of 0.91% and 5.43% without confidence intervals or statistical significance tests. Given the evaluation set size of 7,098 episodes, these differences may be within noise; please report variance or significance tests.
  4. [References and formatting] Several references are incomplete or inconsistently formatted (e.g., [7] 'EW DlJKSTRA', missing conference/journal details for some entries). Table 2 has missing spacing in '89.4896.58' and other cells. Please run a consistency pass.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; ground truth is external and the replay sandbox is a methodology choice, not an equation reducing to its inputs.

full rationale

MobilityBench's central claims are empirical comparisons of LLM route-planning agents, not derivations that reduce to their inputs. Ground truth is constructed from AMap Web Service API responses captured at collection time plus expert-defined standard operating procedures (Section 3.1.3), which are external to the models being evaluated. The deterministic replay sandbox (Section 3.1.4) is a reproducibility mechanism, not a fitted parameter or a prediction target. None of the evaluation metrics (ID, IE, DEC, TS, SC, DR, FPR) are shown to be equivalent by construction to the cached responses or to the SOP outputs; they compare agent behavior against a fixed, independently constructed reference. The paper cites two prior works by overlapping authors (INTSR [25], DSFNet [30]) in related work, but these citations are not load-bearing for the benchmark's validity or for the reported performance gaps. The potential concern that fuzzy/nearest-neighbor fallback in the sandbox could confound final-pass-rate with API-call canonicalization is a validity threat, not a circularity: it does not amount to a claim that a prediction is its own input. No self-definitional step, fitted-input-called-prediction, uniqueness-importation, or ansatz-smuggling was found.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on treating AMap's historical API responses as ground truth, on replay/fallback fidelity, on the representativeness of no-clarification filtered queries, and on the correctness of the Qwen-4B-plus-expert taxonomy. No new physical or metaphysical entities are postulated. The main unquantified free parameters are the intent-similarity threshold and several undefined matching functions.

free parameters (2)
  • alpha_threshold (intent-detection similarity threshold) = not reported
    Used in Eq. (1) to decide whether a predicted intent label matches the ground-truth label; the benchmark's ID scores depend on this hand-set threshold.
  • f_DEC and f_SC semantic match functions = not specified
    Equations (3) and (5) rely on undefined functions that determine whether two atomic actions or parameter sets match; scores change with the implementation of these functions.
assumptions (5)
  • domain assumption AMap Web Service API responses captured at collection time are correct ground truth.
    Sections 3.1.3-3.1.4: ground-truth annotations y are generated by executing expert-defined standard tool programs against this API and replaying cached responses.
  • domain assumption Replay fallbacks (fuzzy entity matching, nearest-neighbor spatial matching) preserve contextual consistency.
    Section 3.1.4: cache misses are resolved by approximation; if these approximations return wrong data, agent successes and failures are misattributed.
  • domain assumption Filtering out queries that require clarification leaves a representative sample of real mobility requests.
    Section 3.1.2: the strict no-clarification assumption removes ambiguous and underspecified queries, which may exclude a meaningful share of real-world usage.
  • ad hoc to paper Qwen-4B open-set labeling plus expert adjudication yields a correct, exhaustive intent taxonomy.
    Section 3.1.2: no inter-annotator agreement, human-validation statistics, or error analysis for the taxonomy are reported.
  • domain assumption Voice queries were transcribed accurately and anonymized.
    Section 3.1.2: transcription fidelity and PII removal are asserted, but no verification or privacy audit is described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios." pith.science (2026). https://pith.science/paper/B7MNYUT7

@misc{pith2026260222638,
  author       = {Pith},
  title        = {Pith review of: MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B7MNYUT7}},
  note         = {Machine review of arXiv:2602.22638}
}
read the original abstract

Route-planning agents powered by large language models (LLMs) have emerged as a promising paradigm for supporting everyday human mobility through natural language interaction and tool-mediated decision making. However, systematic evaluation in real-world mobility settings is hindered by diverse routing demands, non-deterministic mapping services, and limited reproducibility. In this study, we introduce MobilityBench, a scalable benchmark for evaluating LLM-based route-planning agents in real-world mobility scenarios. MobilityBench is constructed from large-scale, anonymized real user queries collected from Amap and covers a broad spectrum of route-planning intents across multiple cities worldwide. To enable reproducible, end-to-end evaluation, we design a deterministic API-replay sandbox that eliminates environmental variance from live services. We further propose a multi-dimensional evaluation protocol centered on outcome validity, complemented by assessments of instruction understanding, planning, tool use, and efficiency. Using MobilityBench, we evaluate multiple LLM-based route-planning agents across diverse real-world mobility scenarios and provide an in-depth analysis of their behaviors and performance. Our findings reveal that current models perform competently on Basic information retrieval and Route Planning tasks, yet struggle considerably with Preference-Constrained Route Planning, underscoring significant room for improvement in personalized mobility applications. We publicly release the benchmark data, evaluation toolkit, and documentation at https://github.com/AMAP-ML/MobilityBench.

Figures

Figures reproduced from arXiv: 2602.22638 by the authors.

Figure 1
Figure 1. Overview of MobilityBench, a systematic benchmark for evaluating route-planning agents. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Global coverage of MobilityBench Data. Instruction Understanding, Planning, Tool Use, and Decision Making, corresponding to the key stages of route-planning reasoning. Each capability is further quantified using a set of fine-grained indicators, enabling precise diagnosis of performance bottlenecks and failure modes that are invisible to end-to-end metrics. 3.2.1 Instruction Understanding. Since accurate interpretat… view at source ↗
Figure 3
Figure 3. Performance across four high-level task families. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Final pass rate comparison (Thinking vs. Non [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MapReason-OSM: Can Vision-Language Models Make Graph-Verifiable Mobility Decisions from Street Maps ?

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    MapReason-OSM supplies 6000 graph-verifiable instances across 12 mobility tasks on rendered OSM maps from 10 U.S. downtowns and shows that seven VLMs succeed at simple routing but perform near chance on cost-based fac...

  2. Large Language Models in Transportation Systems Management and Operations: From Text Reasoning to Multi-modal Decision Support

    cs.AI 2026-05 unverdicted novelty 2.0 of 10

    A survey synthesizing LLM and MM-LLM uses in transportation operations, mobility services, and decision support while noting challenges like data heterogeneity and real-time needs.

Reference graph

Works this paper leans on

37 extracted references · 13 linked inside Pith · cited by 2 Pith papers

  1. [1]

    ICMLT 2020: 2020 5th International Conference on Machine Learning Technolo- gies. 2020. Proceedings of the 2020 5th International Conference on Machine Learning Technologies

    2020

  2. [2]

    Palaash Agrawal, Shavak Vasania, and Cheston Tan. 2025. Can LLMs Perform Structured Graph Reasoning Tasks?. InInternational Conference on Pattern Recog- nition. Springer, 287–308

    2025

  3. [3]

    Soumyabrata Chaudhuri, Pranav Purkar, Ritwik Raghav, Shubhojit Mallick, Man- ish Gupta, Abhik Jana, and Shreya Ghosh. 2025. Tripcraft: A benchmark for spatio-temporally fine grained travel planning.arXiv preprint arXiv:2502.20508 (2025)

    arXiv 2025

  4. [4]

    Aili Chen, Xuyang Ge, Ziquan Fu, Yanghua Xiao, and Jiangjie Chen. 2024. Travelagent: An ai assistant for personalized travel planning.arXiv preprint arXiv:2409.08069(2024)

    arXiv 2024

  5. [5]

    Xiang Cheng, Yulan Hu, Xiangwen Zhang, Lu Xu, Zheng Pan, Xin Li, and Yong Liu. 2025. TravelBench: A Real-World Benchmark for Multi-Turn and Tool- Augmented Travel Planning.arXiv preprint arXiv:2512.22673(2025)

    arXiv 2025

  6. [6]

    Daniel Delling, Peter Sanders, Dominik Schultes, and Dorothea Wagner. 2009. Engineering route planning algorithms. InAlgorithmics of large and complex networks: design, analysis, and simulation. Springer, 117–139

    2009

  7. [7]

    EW DlJKSTRA. 1959. A Note on Two Problems in Connexion with Graphs. Numer. Math.50 (1959), 269–271

    1959

  8. [8]

    Peter E Hart, Nils J Nilsson, and Bertram Raphael. 1968. A formal basis for the heuristic determination of minimum cost paths.IEEE transactions on Systems Science and Cybernetics4, 2 (1968), 100–107

    1968

Show all 37 references
  1. [9]

    Siyuan Hu, Mingyu Ouyang, Difei Gao, and Mike Zheng Shou. 2024. The dawn of gui agent: A preliminary case study with claude 3.5 computer use.arXiv preprint arXiv:2411.10323(2024)

    2024 arXiv

  2. [10]

    Zhehui Huang, Guangyao Shi, and Gaurav S Sukhatme. 2024. Can Large Language Models Solve Robot Routing?arXiv preprint arXiv:2403.10795(2024)

    2024 arXiv

  3. [11]

    Mourad Jbene, Abdellah Chehri, Rachid Saadane, Smail Tigani, and Gwanggil Jeon. 2025. Intent detection for task-oriented conversational agents: A compara- tive study of recurrent neural networks and transformer models.Expert Systems 42, 2 (2025), e13712

    2025

  4. [12]

    Sehoon Kim, Suhong Moon, Ryan Tabrizi, Nicholas Lee, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. 2024. An llm compiler for parallel function calling. InForty-first International Conference on Machine Learning

    2024

  5. [13]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023. Agentbench: Evaluating llms as agents.ICLR(2023)

    2023

  6. [14]

    Junting Lu, Zhiyang Zhang, Fangkai Yang, Jue Zhang, Lu Wang, Chao Du, Qing- wei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. 2025. Axis: Efficient human-agent-computer interaction with api-first llm-based agents. InProceed- ings of the 63rd Annual Meeting of the Associa...

    2025

  7. [15]

    Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, Wenhao Yu, and Dong Yu. 2023. Laser: Llm agent with state-space exploration for web navigation. arXiv preprint arXiv:2309.08172(2023)

    2023 arXiv

  8. [16]

    Silin Meng, Yiwei Wang, Cheng-Fu Yang, Nanyun Peng, and Kai-Wei Chang

  9. [17]

    Yansong Ning, Rui Liu, Jun Wang, Kai Chen, Wei Li, Jun Fang, Kan Zheng, Naiqiang Tan, and Hao Liu. 2025. Deeptravel: An end-to-end agentic reinforce- ment learning framework for autonomous travel planning agents.arXiv preprint arXiv:2509.21842(2025)

    2025 arXiv

  10. [18]

    Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2024. Go- rilla: Large language model connected with massive apis.Advances in Neural Information Processing Systems37 (2024), 126544–126565

    2024

  11. [19]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789(2023)

    2023 arXiv

  12. [20]

    Yincen Qu, Huan Xiao, Feng Li, Gregory Li, Hui Zhou, Xiangying Dai, and Xiaoru Dai. 2025. TripScore: Benchmarking and rewarding real-world travel planning with fine-grained evaluation.arXiv preprint arXiv:2510.09011(2025)

    2025

  13. [21]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools.Advances in Neural Information Processing Systems36 (2023), 68539–68551

    2023

  14. [22]

    Junhong Shen, Atishay Jain, Zedian Xiao, Ishan Amlekar, Mouad Hadji, Aaron Podolny, and Ameet Talwalkar. 2025. WorkflowAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data. InICLR 2025 Workshop on Foundation Models in the Wild

    2025

  15. [23]

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan-and-solve prompting: Improving zero-shot chain- of-thought reasoning by large language models.arXiv preprint arXiv:2305.04091 (2023)

    2023 arXiv

  16. [24]

    Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. 2024. Travelplanner: A benchmark for real-world planning with language agents.arXiv preprint arXiv:2402.01622(2024)

    2024 arXiv

  17. [25]

    Huimin Yan, Longfei Xu, Junjie Sun, Ni Ou, Wei Luo, Xing Tan, Ran Cheng, Kaikui Liu, and Xiangxiang Chu. 2025. Intsr: An integrated generative framework for search and recommendation.arXiv preprint arXiv:2509.21179(2025)

    2025

  18. [26]

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems35 (2022), 20744–20757

    2022

  19. [27]

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. 𝜏- bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv preprint arXiv:2406.12045(2024)

    2024 arXiv

  20. [28]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems36 (2023), 11809–11822

    2023

  21. [29]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. InThe eleventh international conference on learning representations

    2022

  22. [30]

    Jiahao Yu, Yihai Duan, Longfei Xu, Chao Chen, Shuliang Liu, Kaikui Liu, Fan Yang, Xiangxiang Chu, and Ning Guo. 2025. DSFNet: Learning Disentangled Scenario Factorization for Multi-Scenario Route Ranking. InCompanion Proceedings of the ACM on Web Conference 2025. 567–576

    2025

  23. [31]

    Liangqi Yuan, Dong-Jun Han, Christopher G Brinton, and Sabine Brunswicker

  24. [32]

    Junlin Zeng, Xin Zhang, Xiang Zhao, and Yan Pan. 2025. A 1000× Faster LLM- enhanced Algorithm For Path Planning in Large-scale Grid Maps.arXiv preprint arXiv:2510.02716(2025)

    2025

  25. [33]

    Tao Zhe, Rui Liu, Fateme Memar, Xiao Luo, Wei Fan, Xinyue Ye, Zhongren Peng, and Dongjie Wang. 2025. Constraint-Aware Route Recommendation from Natural Language via Hierarchical LLM Agents.arXiv preprint arXiv:2510.06078(2025)

    2025

  26. [34]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information processing systems36 (2023), 46595–46623

    2023

  27. [35]

    Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. 2023. Language agent tree search unifies reasoning acting and planning in language models.arXiv preprint arXiv:2310.04406(2023). A Appendix A.1 MobilityBench Task Scenarios To facilitate a thorough...

    2023 arXiv

  28. [2024]

    Llm-a*: Large language model enhanced incremental heuristic search on path planning.arXiv preprint arXiv:2407.02511(2024)

    2024 arXiv

  29. [2025]

    LLMAP: LLM-Assisted Multi-Objective Route Planning with User Prefer- ences.arXiv preprint arXiv:2509.12273(2025)

    2025

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.