{"id":"5e82cac9-df6f-4aec-9a89-ae39a57f99fb","arxiv_id":"2602.22638","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"MobilityBench is a 100,000-episode benchmark with a replay sandbox for deterministic evaluation of LLM route-planning agents; current models score well on basic tasks but fail preference-constrained routing.","lead":"The paper introduces MobilityBench, a benchmark built from anonymized real user route-planning queries from Amap, with a deterministic API-replay sandbox for reproducible evaluation of LLM-based route-planning agents. It evaluates 11 LLMs and finds they handle basic routing well but struggle when user preferences and constraints are involved.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fallback resolution for cache misses may confound FPR with API-call canonicalization; needs validation.","rationale":"The central claim is that MobilityBench is a valid, reproducible measure of route-planning skill and that models perform poorly on preference-constrained tasks. This concern cuts directly at that claim: it identifies a concrete mechanism by which the evaluation could reward or punish agents based on tool-call syntax rather than spatial reasoning. The fallback is an explicit part of the design (Sec. 3.1.4), so this is not an outside-consensus objection but an internal consistency risk. The test is feasible with the released toolkit and would settle whether the confound is material. Secondary omissions the reader noted (FPR formula, thresholds) are also relevant, but the fallback behavior is the sharpest point of attack because it can be tested directly against the stated design. I agree with the reader's conditionality; the verdict remains CONDITIONAL pending these checks.","tokens_in":14857,"tokens_out":4716,"duration_ms":45883,"concrete_test":"Instrument the released sandbox to log for each intercepted tool call: exact cache hit, fuzzy hit, nearest-neighbor hit, or failure. Re-run the 7,098-episode evaluation for the 5 best and 5 worst models from Table 2, recording for each episode the fraction of calls resolved by fallback. Then compute FPR separately for episodes with all-exact hits and with any fallback. If the all-exact FPR exceeds the fallback-exposed FPR by more than 5 points, or if more than 20% of episodes use any fallback, FPR is substantially contaminated by cache canonicalization. Also sample 100 fallback-resolved calls and re-query the live AMap Web Service to check whether the fallback response matches the frozen ground-truth response; mismatches indicate the fallback breaks the frozen-state assumption.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's validity rests on the replay sandbox (Sec. 3.1.4). The cache is keyed by canonicalized arguments, and fallback uses fuzzy/nearest-neighbor matching when exact hits are unavailable. The ground truth for each episode is the output of the expert-defined 'standard tool program' executed at collection time, so the cache is complete only for the SOP's exact call sequences. An agent that resolves the same user intent through a different but equally correct tool sequence (e.g., POI query by canonical name vs. geocoded coordinates, or a slightly different departure-time format) may trigger a fallback rather than an exact hit. The fallback response is not guaranteed to equal the frozen response from the SOP; it may be spatially or semantically off, yielding wrong evidence and a false FPR failure. Conversely, fallback could coincidentally produce a correct-looking response and mask a planning error. Consequently, FPR conflates route-planning ability with how closely the agent's tool-call syntax matches the SOP's canonicalization. This is especially acute for Preference-Constrained Route Planning (Option-/Route-Constrained), where parameters like avoid_tolls or waypoint polylines are more complex and serialization variability is higher. If many evaluation episodes involve fallback resolution, the headline gap (Basic vs. Preference-Constrained) may reflect cache coverage and canonicalization strictness rather than skill. The manuscript provides no statistics on cache-hit vs. fallback rates, so this potential confound is unquantified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MobilityBench, a benchmark for evaluating LLM-based route-planning agents in real-world mobility scenarios. It is constructed from 100,000 anonymized real user queries from Amap, organized into 11 task scenarios across four intent families. To ensure reproducibility, the authors design a deterministic API-replay sandbox that caches responses from AMap Web Service APIs and serves them during evaluation. They propose a multi-dimensional evaluation protocol covering instruction understanding, planning, tool use, decision making, and efficiency, with a central outcome metric called Final Pass Rate (FPR). The paper evaluates a range of open- and closed-source LLMs under ReAct and Plan-and-Execute frameworks, reporting that current models perform well on basic tasks but struggle with preference-constrained route planning.","tokens_in":15220,"tokens_out":2184,"duration_ms":23773,"significance":"If the validity and reproducibility concerns are adequately addressed, MobilityBench would be a valuable contribution to the agent benchmarking literature. Its strengths include the large real-query scale, the explicit attempt to control environmental nondeterminism via a replay sandbox, the multi-dimensional evaluation protocol that goes beyond end-to-end success, and the stated public release of data, toolkit, and documentation. The empirical finding of a performance gap between basic and preference-constrained route planning is plausible and, if confirmed, useful for directing future research. However, the current manuscript omits key operational details that are load-bearing for interpreting the reported numbers, especially the definition and validation of the FPR metric and the behavior of the replay sandbox on cache misses.","major_comments":[{"comment":"Final Pass Rate (FPR) is the central outcome metric for the paper's headline claims (e.g., 'struggle considerably with Preference-Constrained Route Planning'), yet it is never defined by an equation. The text states only that a solution is successful 'if it satisfies all user-specified explicit and implicit constraints.' This is insufficient: what constitutes 'satisfies'? Is it exact equality of route features, a tolerance-based comparison, or a semantic check? Without a formal definition, FPR is not reproducible, and the reported 60-70% numbers cannot be independently verified. Please provide the precise evaluation function, including how route plans, times, transfer counts, and constraint predicates are compared against ground truth.","section":"Section 3.2.4, Table 2"},{"comment":"The replay sandbox is the keystone of the benchmark's reproducibility claim, but the manuscript gives no statistics on how often exact cache hits occur versus fallback strategies (fuzzy matching, nearest-neighbor spatial matching) and no threshold values or validation that fallback responses are contextually equivalent to the frozen ground-truth responses. If a substantial fraction of agent calls are resolved via fallback, then the measured FPR conflates route-planning skill with whether the agent's tool-call syntax matches the SOP's canonicalization (e.g., using a POI name instead of coordinates, or a nonstandard time format). This concern is especially acute for Option-Constrained and Route-Constrained planning, where parameters are more complex. Please report cache-hit rates per scenario, the maximum distance threshold used, and ideally a human-validated sample of fallback resolutions","section":"Section 3.1.4; Section 4.2.1"},{"comment":"Three evaluation indicators depend on unspecified components: the similarity threshold alpha_threshold in Eq. (1), the semantic match function f_DEC in Eq. (3), and the schema-compliance function f_SC in Eq. (5). None of these are defined or given values, and no reference is provided. The reproducibility claim of the paper is undermined because two independent implementations could choose different thresholds or matching functions and obtain different ID, DEC, and SC scores. Please specify these functions explicitly or, at minimum, provide the exact implementation used in the released toolkit.","section":"Section 3.2.1, Eq. (1); Section 3.2.2, Eq. (3); Section 3.2.3, Eq. (5)"},{"comment":"The ground-truth construction pipeline uses Qwen-4B for open-set intent classification followed by 'expert adjudication,' and scenario-specific standard operating procedures (SOPs) are defined by domain experts. Yet no inter-annotator agreement, number of experts, or quality-assurance statistics are reported. The benchmark's validity rests on the assumption that the intent taxonomy is exhaustive and mutually exclusive and that the SOPs encode the 'correct' tool sequence. Please report the number of annotators, agreement measures, and the process for resolving disagreements. Without this, the reader cannot assess whether the 11-scenario taxonomy and the expert-defined ground truth are reliable.","section":"Section 3.1.2; Section 3.1.3"}],"minor_comments":[{"comment":"The paper consistently uses 'route-planning' as a noun modifier; consider hyphenation consistency. Also, the example 'avoid tolls/highways' in Table 1 is listed under Option-Constrained, while the text uses 'Option-' and 'Route-Constrained' interchangeably in places; please unify terminology.","section":"Abstract and Section 1"},{"comment":"The Delivery Rate (DR) is described as measuring 'complete and executable final output,' but it is not formally defined either. Please provide an equation or explicit criterion, especially since DR appears prominently in Table 2.","section":"Section 3.2.4"},{"comment":"The scaling effect analysis reports improvements of 0.91% and 5.43% without confidence intervals or statistical significance tests. Given the evaluation set size of 7,098 episodes, these differences may be within noise; please report variance or significance tests.","section":"Section 4.2.3"},{"comment":"Several references are incomplete or inconsistently formatted (e.g., [7] 'EW DlJKSTRA', missing conference/journal details for some entries). Table 2 has missing spacing in '89.4896.58' and other cells. Please run a consistency pass.","section":"References and formatting"}],"recommendation":"major_revision","confidential_remarks":"The paper's authors include employees of Amap, which is both the data source and the provider of the evaluated APIs. This creates a potential conflict of interest that is not discussed in the paper. I am not recommending rejection on that basis, but the editor should ensure that the benchmark data and evaluation code are made available for independent scrutiny and that the paper discloses the affiliation clearly. The main technical concern is the missing operationalization of FPR and the unspecified sandbox fallback behavior; these are fixable within the scope of a revision, but they are central to the paper's claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWhat you should know: MobilityBench is a serious, potentially reusable benchmark for LLM route-planning agents, built from real anonymized Amap queries. The replay sandbox that freezes API responses is the right answer to the reproducibility problem. The headline gap between basic tasks and preference-constrained route planning is plausible, but the paper does not yet define one of its two headline metrics (FPR) and does not quantify how often the fallback resolver is used. Treat the exact numbers as conditional until the repo is checked.\n\nWhat is genuinely new: 100,000 episodes, a 7,098-episode evaluation set, 11 scenarios across four families, covering 350+ cities. The deterministic API-replay sandbox is a practical technical contribution; it removes live traffic and weather variance and makes agent comparisons fair. The evaluation protocol decomposes behavior into instruction understanding, planning, tool use, and decision-making, which is more useful than a single success rate. The breadth of models evaluated (open and closed, ReAct and Plan-and-Execute) is also solid.\n\nWhere it is soft: The biggest gap is the missing formal definition of Final Pass Rate. The paper says it \"satisfies all user-specified explicit and implicit constraints\" but gives no equation, so no one can reproduce it from the text. Similarly, the semantic matching functions f_DEC and f_SC and the similarity threshold alpha are unspecified. The replay-sandbox fallback is a real concern: if an agent makes a different but equally valid tool call (e.g., geocoded coordinates vs POI name) and triggers fuzzy/nearest-neighbor matching, the response may differ from the SOP's frozen response. That could conflate canonicalization strictness with planning skill. The paper needs to report cache-hit vs fallback rates and ideally show that FPR is robust to the matching threshold. No inter-annotator reliability is reported for ground-truth labels, though the expert-adjudication process is described. There are no error bars or significance tests on the model comparisons; with 7,098 episodes this is a minor omission but worth fixing. Also, Section 4.2.2 seems to say Plan-and-Execute does best on preference-constrained tasks while Table 2 shows ReAct generally higher FPR; that tension needs a sentence of explanation.\n\nNone of this kills the paper. The central design is sound, and the fallback concern is a validation question, not a demonstrated flaw. The authors should publish the artifacts and add the missing metric definitions and fallback statistics.\n\nWho it is for: anyone building or evaluating LLM agents for map and navigation tasks. It deserves a serious peer review — a competent reviewer can verify the sandbox and metrics. My recommendation: send it to review, with a request for the missing definitions and a robustness analysis of the replay sandbox.","headline":"MobilityBench is a serious, potentially reusable benchmark with a sensible replay-sandbox design, but the missing FPR definition and unquantified fallback resolution make the exact results conditional until the artifacts are published.","tokens_in":15662,"tokens_out":2789,"would_cite":true,"duration_ms":25184,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MobilityBench, a benchmark of 100,000 anonymized real mobility queries, shows that LLM route-planning agents handle basic queries well but fail far more often on preference-constrained routes.","keywords":["LLM agents","route planning","benchmark","API replay sandbox","preference-constrained routing","reproducibility","tool use","mobility"],"falsifier":"Pick the most time-sensitive scenarios (traffic-info query, arrival-time query) and replay the frozen cache months after collection; if an agent that trusts the cached 'current' traffic still passes while a freshly queried API would produce a materially different route, the sandbox is scoring state retrieval, not planning. Additionally, inspect cache-hit versus fallback rates per episode: an episode whose correct answer depends on a fuzzy match or nearest-neighbor fallback rather than an exact cached response is a direct place to test whether ground truth is truly grounded.","tokens_in":14787,"feed_emoji":"🗺️","tokens_out":7192,"duration_ms":63110,"temperature":0.7,"pith_summary":"MobilityBench is a benchmark for LLM-based route-planning agents, built from large-scale anonymized real user queries and designed to make evaluation reproducible despite live mapping services being non-deterministic. The paper's central claim is that, with a deterministic API-replay sandbox and a ground-truth protocol centered on outcome validity, agent performance can be measured fairly and decomposed into instruction understanding, planning, tool use, and decision-making. Using this setup, the paper evaluates multiple open- and closed-source LLMs under two agent frameworks and finds that models do well on basic information retrieval and basic route planning but struggle considerably with preference-constrained route planning. If this finding is right, the bottleneck for personalized mobility applications is not tool use or basic navigation but the ability to honor user constraints such as avoiding highways or minimizing transfers.","feed_headline":"LLM route agents ace basic queries, fail constrained routes","feed_subtitle":"A 100,000-query replay sandbox makes route-planning agents comparable and exposes where personalized navigation still breaks.","key_machinery":"The load-bearing mechanism is the deterministic API-replay sandbox. It intercepts every tool call an agent makes during evaluation and resolves it against a cache of responses captured when the ground truth was constructed, keyed by canonicalized arguments; exact hits are replayed, while misses fall back to fuzzy entity matching or nearest-neighbor spatial matching within a distance threshold. This freezes traffic, weather, and service state at data-collection time so that identical tool calls always yield identical outputs, removing environmental variance from live APIs. The second essential piece is the standard tool program — the minimal expert-defined sequence of tool calls that correctl","core_discovery":"On its own terms, the paper claims to provide the first large-scale, reproducible benchmark for fine-grained route-planning agents in real-world mobility settings. Each episode is a four-tuple: an anonymized natural-language query, contextual information, a fixed replayable snapshot of relevant API responses, and a structured ground-truth annotation derived from an expert-defined standard tool program. The evaluation centers on Final Pass Rate — the share of queries for which an agent produces an executable solution satisfying all explicit and implicit constraints — and supplements it with metrics for intent detection, information extraction, task decomposition, tool selection, schema compli","pith_inferences":["An implication the paper leaves implicit: because cache misses are resolved by fuzzy or nearest-neighbor fallback, some 'successful' runs may be rewarded for retrieving a plausible cached response rather than for genuine route-planning skill; auditing fallback-match rates by scenario would test this.","The no-clarification assumption makes the benchmark intentionally harder than a production voice assistant; preference-constrained failure rates might shrink if agents were allowed one clarifying question.","The same replay-sandbox architecture could transfer to other non-deterministic API domains, such as flight, hotel, or ride-hail booking, where live prices and availability make reproducibility equally hard.","A natural extension is a time-shifted variant of the benchmark that changes the cached weather or traffic snapshot between query construction and evaluation, testing whether agents notice stale data rather than blindly trusting replayed responses."],"forward_implications":["If MobilityBench is valid, route-planning agents can be compared fairly across LLMs and agent frameworks without confounding from live traffic or service variability.","The reported gap on preference-constrained route planning gives a concrete target: improving constraint honoring (avoid highways, minimize transfers, required waypoints) matters more than better retrieval or basic routing.","The metric decomposition allows researchers to see whether a failure is an intent-detection error, an information-extraction error, a planning error, a tool-call error, or a final-decision error, instead of a single pass/fail.","ReAct-style agents tended toward higher final pass rates than Plan-and-Execute, at roughly 35% higher input-token cost, implying a direct robustness-versus-cost trade-off for deployment.","Scaling model size and enabling thinking mode improved final pass rate in the paper's experiments, but both raised inference cost, so the paper points toward efficient small models as a practical need."],"fun_headline_variants":["Route-LLMs pass simple tasks, choke on preference constraints","MobilityBench shows LLM route agents stumble on user preferences","Benchmark finds LLM route planners weak on constrained routing","LLM agents good at simple routes, bad at personalized ones","New benchmark: LLM route agents struggle with preference-heavy queries"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark's validity rests on the assumption that replaying API responses cached at data-collection time — with fuzzy or nearest-neighbor fallbacks for cache misses — yields the correct, contextually consistent answer for every episode; if a cached or fallback response is wrong for the query, the Final Pass Rate measures cache retrieval rather than route-planning ability.","fun_headline_variants_meta":{"raw":{"variants":["Route-LLMs pass simple tasks, choke on preference constraints","MobilityBench shows LLM route agents stumble on user preferences","Benchmark finds LLM route planners weak on constrained routing","LLM agents good at simple routes, bad at personalized ones","New benchmark: LLM route agents struggle with preference-heavy queries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000476,"raw_usage":{"total_tokens":2204,"prompt_tokens":759,"completion_tokens":1445,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":1374}},"tokens_in":503,"tokens_out":1445,"duration_ms":10710,"temperature":1.0,"reasoning_tokens":1374,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T20:37:51.709163+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pick the most time-sensitive scenarios (traffic-info query, arrival-time query) and replay the frozen cache months after collection; if an agent that trusts the cached 'current' traffic still passes while a freshly queried API would produce a materially different route, the sandbox is scoring state retrieval, not planning. Additionally, inspect cache-hit versus fallback rates per episode: an episode whose correct answer depends on a fuzzy match or nearest-neighbor fallback rather than an exact cached response is a direct place to test whether ground truth is truly grounded.","supporting_citations":[],"review_version":1}