{"id":"24d54a39-9cb0-4fae-8c6d-b6b6a3e2350b","arxiv_id":"2608.08814","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new benchmark evaluates embodied agents in a photorealistic 360-video reconstruction of Akihabara, and state-of-the-art LMM agents score far below local human experts.","lead":"The paper introduces 360CityArena, a photorealistic virtual replica of Tokyo's Akihabara district built from 360-degree videos, with 175 tasks for testing embodied AI agents on urban navigation and spatial reasoning. A generalist reader should care because state-of-the-art vision-language agents score around 17 percent while humans score 77 percent, showing how far current agents are from usable city-scale navigation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Real-world validity rests on the inherited RVW pose graph from [41], whose street-topology and coordinate accuracy are not validated in this paper; without that validation, map-based tasks and coordinate-match ground truths may not measure real urban navigation.","rationale":"The reader's weakest assumption is that the benchmark's validity depends on the Realistic Virtual World from [41] accurately representing Akihabara's street connectivity and visual layout. My stress-test converges on that same point and sharpens it: the paper uses OSM maps for tasks, coordinate-match thresholds for success, and route statistics for difficulty calibration, all of which presuppose that the pose graph's topology and coordinates agree with the real street network. No evidence for that agreement is presented in this manuscript, and the limitation section's qualitative check of boundary discontinuities does not address topology or coordinate drift. This is the most load-bearing concern because if the pose graph is geometrically inconsistent with OSM, then the headline results—especially 0% LMM accuracy on Map Navigation versus 92% for local-expert humans—could reflect a map-to-environment alignment failure rather than a deficit in urban path reasoning. I am not claiming the environment is wrong; I am claiming the benchmark's central interpretation is unsecured without a direct topology/coordinate validation. My concrete test would settle this by comparing pose-graph shortest paths and node coordinates to OSM. If the comparison passes, the concern is resolved and the benchmark's real-world validity is substantially strengthened. I therefore keep the reader's CONDITIONAL verdict unchanged, because the paper should provide this validation (or explicit routing/coordinate error bounds) before the benchmark is adopted as a standard real-world urban navigation testbed.","tokens_in":17104,"tokens_out":9746,"duration_ms":113753,"concrete_test":"Compute the correspondence between the RVW pose graph and the OSM street network used for task maps. For each of the 25 Map Navigation start/goal pairs, compute the shortest path on the 193-node/305-edge pose graph and compare its length and turn sequence to the OSM shortest path; additionally, sample the 193 node coordinates against OSM road centerlines and report median and 95th-percentile distance. If route disagreement exceeds 10% of length or node-to-road median error exceeds about 5 m, map-based ground truths and coordinate_match are not reliable measures of real-world navigation; if the comparison passes, the inherited environment is validated for this benchmark.","verdict_should_be":"UNCHANGED","load_bearing_attack":"360CityArena's central claim—that it evaluates embodied agents on real urban path/spatial reasoning and that LMMs fail at it—depends on the fidelity of the Realistic Virtual World (RVW) built in the same group's prior paper [41]. Every task, every ground-truth coordinate, and the OpenStreetMap crops used in Map Navigation and Localization inherit the RVW pose graph's topology and geometry. The present manuscript reports graph statistics (193 nodes, 305 edges, mean branching degree 3.16, Section 3.1) and a qualitative boundary-transition check (Section 6), but it does not report any comparison of the pose graph to the actual street network or to the OSM maps used in the tasks (Section A.2). If the graph misconnects streets, omits an edge, or has GPS/SLAM coordinate drift, then (i) coordinate_match thresholds become miscalibrated, (ii) Map Navigation's 'optimal route' ground truth is not the real optimal route, and (iii) the human 92% Map Navigation result may reflect local-knowledge compensation for a broken map rather than valid benchmark measurement. This is a correctness risk, not an external-consensus disagreement: it is an unvalidated input to every number in Tables 2 and 3. The paper's own limitation section only checks discontinuity-induced failures qualitatively, not topology or coordinate accuracy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces 360CityArena, an embodied urban navigation benchmark built on a 360-degree video reconstruction of Akihabara, Tokyo, comprising 175 tasks across three categories and seven subcategories: Environment Understanding, Path Reasoning, and Spatial Reasoning. The authors evaluate six LMM-based agents (GPT-5, Claude Sonnet 4.5, Gemini 2.5 Flash, Qwen2.5-VL, and two InternVL variants) and five local-expert human participants, reporting a large performance gap (best model Gemini 2.5 Flash at 17.1% versus humans at 77.3%). The benchmark uses four evaluation metrics — exact match, fuzzy match with a validated GPT-5 judge, coordinate match with threshold sensitivity analysis, and mean relative accuracy — and includes analyses of location-information effects and failure modes. The paper also provides a detailed appendix with task prompts and experimental settings.","tokens_in":17356,"tokens_out":7404,"duration_ms":77293,"significance":"If the environment fidelity holds, 360CityArena is a valuable addition to embodied AI benchmarking: it combines photorealistic, dynamic, city-scale observations with seven task types covering perception, path reasoning, and spatial reasoning. The evaluation protocol is careful and transparent: the fuzzy-match judge is validated against human agreement (kappa 0.937), coordinate thresholds are checked for sensitivity (Kendall's tau >= 0.89), and task statistics monotonically support the Easy/Medium/Hard labels. The public benchmark infrastructure, detailed prompts, and reproducible metric definitions are strengths that make the paper useful regardless of the specific model results. However, the benchmark's real-world validity hinges on the fidelity of the underlying reconstructed pose graph, which is not independently validated in this manuscript.","major_comments":[{"comment":"The validity of the benchmark as a test of real urban navigation rests on the inherited Realistic Virtual World pose graph from reference [41], but the manuscript does not validate that pose graph's topology or coordinate accuracy. The 193-node, 305-edge graph is described, and Section 6 only qualitatively checks boundary discontinuities; there is no comparison of the pose graph to the actual Akihabara street network or to the OpenStreetMap crops used in Map Navigation, Localization, and coordinate-match ground truth. Without such validation, the optimal-route labels, grid-cell answers, and coordinate thresholds (10 m for most tasks, 20 m for Map Navigation) may be miscalibrated, and the human Map Navigation result (92%) could reflect local-knowledge compensation for a distorted map rather than valid benchmark measurement. Please add a quantitative validation, such as node/edge precision and recall against OSM street segments and coordinate error at intersections, or explicitly cite where reference [41] provides this validation and summarize its results.","section":"Section 3.1, Section 6, Section A.2"},{"comment":"The human baseline consists of only five participants, and no variance, confidence interval, or per-participant breakdown is reported. Since the headline claim \"human 77.3% vs. Gemini 2.5 Flash 17.1%\" is a central result, the current presentation overstates its precision. Please report confidence intervals (or per-participant scores) and clarify how the 25 tasks per subcategory were allocated across participants; if the five participants each attempted all tasks, report the between-subject variance, and if tasks were split, report the per-task sample size.","section":"Section 5.1, Table 2"},{"comment":"The analysis of the effect of removing location information compares cells of only 25 tasks each, with no significance testing or confidence intervals. Differences such as Landmark (Language) 24.0% vs. 16.0% and Relational Reasoning 48.0% vs. 32.0% are within plausible sampling error for n=25 (approximate standard error of 8–10 percentage points). The conclusions drawn about map-to-visual alignment difficulty are therefore not supported by the data. Please add significance tests (e.g., exact binomial or bootstrap confidence intervals) or restrict the discussion to descriptive observations.","section":"Section 5.2, Table 3"}],"minor_comments":[{"comment":"The sentence \"360CityArena would also serves as a practical guideline\" contains a grammatical error; it should be \"would also serve as.\"","section":"Section 1"},{"comment":"The failure-cause breakdown uses Gemini 3 Flash to label failures, with humans verifying each label, but the paper does not report the number of classified failures or the inter-annotator agreement between Gemini 3 Flash and the human verifiers; please add this information to assess the reliability of the failure analysis.","section":"Section 5.2, Figure 5"},{"comment":"The paper states that tasks are labeled Easy, Medium, or Hard, but it does not report the number of tasks in each difficulty level per subcategory; adding this distribution would help readers interpret Figure 4 and the difficulty-related statistics.","section":"Section 3.3"},{"comment":"The action described as \"reset the viewpoint to align with the current heading direction\" in Section 4.2 is referred to as \"S\" and labeled \"turn camera to the direction of travel\" in the system prompt; unify the terminology to avoid confusion for readers and for future implementations.","section":"Section B.1 and Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The benchmark's environment is inherited from the authors' own prior work [41], and the manuscript does not disentangle the contribution of the new task suite from the unvalidated fidelity of the inherited environment. The lack of independent validation of the pose graph's topology and coordinate accuracy is the main correctness risk, and it directly affects the interpretation of every task result. I would weigh this risk against the paper's positive aspects: the evaluation metrics are thoughtfully designed and the benchmark itself is likely to be useful. With a targeted validation study (or a clear citation to existing validation) and modest improvements to the human-baseline and significance reporting, the paper could be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: 360CityArena is a genuine benchmark contribution with a careful evaluation protocol, but it currently rests on an unvalidated inherited environment, and without that validation or artifact release, the headline 'real urban navigation' claim is premature.\n\nWhat's new: the combination of seven task types over a district-scale photorealistic 360-video pose graph, with matched language- and image-based landmark search and dynamic street content. The 175 human-crafted tasks, Easy/Medium/Hard labels backed by monotonic statistics, and the thorough metric definitions (exact, fuzzy, coordinate, MRA) show real care. The fuzzy-match judge validation (kappa 0.937) and the threshold sensitivity check are good practice. The central finding—all tested LMMs score 0-16% on map navigation while local humans hit 92%—is likely robust; even a doubled threshold wouldn't close that gap.\n\nThe soft spots are real but not fatal. The environment is the RVW from the same group's prior paper [41]. The paper reports the pose graph has 193 nodes/305 edges but never checks that topology or the coordinate frame against the OpenStreetMap crops actually used in the tasks. If an edge is missing or a node drifts, the coordinate_match thresholds and the 'optimal route' ground truths in Map Navigation inherit that error, and the human result may reflect local-knowledge compensation rather than genuine map fidelity. The limitation section only addresses discontinuity-induced failures at trajectory boundaries, not this topology/coordinate risk. The human baseline is five participants, all familiar with Akihabara, so it is a local-expert reference, not a generic human benchmark. The paper also does not release the tasks, the environment, or the agent code, which limits independent verification at the moment.\n\nThis is a useful paper for anyone building or benchmarking embodied navigation agents, but it is not yet the definitive real-world navigation testbed until those gaps are closed. The core empirical gap between LMMs and humans is credible and worth reporting.\n\nRecommendation: send it to peer review. It deserves a serious referee. The reviewers should ask for (1) a validation of the pose graph against ground-truth street geometry or at least a consistency analysis between the graph and the OSM maps, (2) a larger and less local human baseline, and (3) benchmark artifact release. If those are satisfied, this could become a standard evaluation.","headline":"A careful benchmark with a plausible strong LMM-human gap, but the unvalidated inherited pose graph and missing artifacts make the 'real urban navigation' claim premature.","tokens_in":17914,"tokens_out":2717,"would_cite":false,"duration_ms":28992,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Best AI agent scores 17.1% where humans hit 77.3% on Akihabara navigation","keywords":["embodied AI","urban navigation","benchmark","360-degree video","photorealistic virtual environment","vision-language navigation","spatial reasoning","Akihabara"],"falsifier":"Take a random sample of 360CityArena tasks, run them in the real Akihabara district with a human or instrumented recorder, and check whether every ground-truth answer — landmark identity, grid cell, route reachability, and object count — matches what is actually on the ground. Any systematic mismatch, such as a Map Navigation route that cuts through a building or a landmark appearing at the wrong corner, would invalidate the benchmark as a real-world proxy.","tokens_in":16904,"feed_emoji":"🏙️","tokens_out":4854,"duration_ms":46568,"temperature":0.7,"pith_summary":"The paper introduces 360CityArena, a benchmark for embodied agents navigating a photorealistic virtual reconstruction of Akihabara, Tokyo, built from 602 interconnected 360-degree video segments covering 85 streets. It contains 175 manually crafted tasks across three categories — environment understanding, path reasoning, and spatial reasoning — and reports that the strongest evaluated multimodal agent, Gemini 2.5 Flash, solves only 17.1% of tasks while local human participants solve 77.3%. The aim is to provide a realistic, dynamic, street-level testbed that existing outdoor simulators and Street-View-based environments lack, and to measure how far current agents are from usable urban exploration. A sympathetic reader would take the paper as showing that city-scale embodied navigation remains largely unsolved, even by large language-vision models.","feed_headline":"Best AI agent scores 17.1% where humans hit 77.3%","feed_subtitle":"A 175-task photorealistic Akihabara benchmark measures embodied navigation and spatial reasoning.","key_machinery":"The central object is the Realistic Virtual World of Akihabara: 602 360-degree video segments projected onto spheres and linked into a navigable pose graph in Unity, with 193 nodes and 305 edges over 85 streets, giving agents smooth motion along filmed trajectories with directional capture per street. This pose graph is the substrate for all 175 tasks. The evaluation also rests on four metrics: exact match for grid-coordinate answers, fuzzy match with an isolated language-model judge for relational answers, coordinate match by Euclidean distance for navigation tasks, and mean relative accuracy for counting tasks. The design enables controlled comparisons, such as identical landmark-search tasks differing only in whether the goal is given as text or as an image.","core_discovery":"On its own terms, the paper's central claim is that a pose graph of 360-degree video trajectories, grounded in the real street network of Akihabara, can serve as a valid urban-district benchmark and that current state-of-the-art LMM-based agents fall far short of human-level performance on it. The claim is supported by a task suite of 175 human-verified tasks, four evaluation protocols, and experiments with six proprietary and open models. The paper also reports that image-based landmark search generally beats language-based landmark search for the same landmarks, that performance declines with task difficulty, and that giving an agent its current location on a map does not consistently help and sometimes hurts.","pith_inferences":["If the human–agent gap holds in follow-up work, the bottleneck is less perception than exploration and grounding: the paper's failure analysis attributes many errors to action overshoot, stagnation loops, and aligning map information with first-person views rather than to object recognition alone.","The same pose-graph construction should transfer to other cities using the Movie Map paradigm, so a multi-city version would directly test whether the observed gap is specific to Akihabara's visual density or general to urban navigation.","A testable extension is to vary the step limit or the availability of the map marker to see whether the low scores reflect poor planning rather than limited observation; the current stopping conditions already suggest stagnation and wrong actions are major factors.","The 11.3% share of boundary-transition actions could be probed by comparing agent success on trajectories that cross video-seam boundaries versus trajectories that stay within a single filmed segment, isolating any artifact of the benchmark's video construction."],"forward_implications":["If the benchmark is accepted, no current LMM-based agent is close to human-level urban exploration: the best model reaches 17.1% overall versus 77.3% for local human participants.","Image-based landmark search is generally easier than language-based landmark search for LMM agents, so visual goal specification is a more reliable channel than textual landmark descriptions.","Performance degrades monotonically as tasks move from Easy to Hard, so the benchmark can rank models by difficulty and expose where specific abilities break down.","Providing self-location as a map marker does not reliably improve agents and can degrade relational spatial reasoning, implying that map-to-view alignment is a distinct, still-unsolved capability.","Because all task categories use the same photorealistic environment, the benchmark supports direct comparison of perception, planning, and counting abilities under matched visual conditions."],"supporting_citations":[{"why":"Supplies the Realistic Virtual World of Akihabara built from 602 360-degree video segments, the substrate for the benchmark.","marker":"[41]"},{"why":"Gemini 2.5 Flash is the strongest evaluated agent and defines the state-of-the-art baseline compared against human performance.","marker":"[7]"},{"why":"GPT-5 is a closed-source baseline agent and the model used for fuzzy-match evaluation.","marker":"[33]"},{"why":"Claude Sonnet 4.5 is one of the proprietary baseline agents benchmarked.","marker":"[2]"},{"why":"Qwen2.5-VL-32B-Instruct is an open-source baseline agent.","marker":"[3]"},{"why":"InternVL3.5-8B and InternVL3.5-38B are open-source baselines that cover the smaller-model end of the performance range.","marker":"[43]"},{"why":"Supplies the mean relative accuracy metric used for the Object Count task.","marker":"[50]"},{"why":"Supplies the exact-match and fuzzy-match evaluation convention used for localization and relational reasoning.","marker":"[31]"},{"why":"WebArena provides the exact-match and fuzzy-match evaluation methodology adopted here.","marker":"[55]"}],"fun_headline_variants":["Embodied AI scores 17% in Akihabara, humans 77%","Akihabara test: humans 77.3%, best AI 17.1%","360CityArena: Best AI gets 17%, humans 77%","Photorealistic Tokyo benchmark reveals AI navigation gap","City-scale navigation benchmark floors Gemini 2.5 Flash"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's value as a proxy for real urban navigation depends on the 360-degree video reconstruction faithfully representing Akihabara's street connectivity, landmark placements, and visual appearance; if the pose graph distorts the layout, the ground-truth answers and the human baseline would not transfer to real streets.","fun_headline_variants_meta":{"raw":{"variants":["Embodied AI scores 17% in Akihabara, humans 77%","Akihabara test: humans 77.3%, best AI 17.1%","360CityArena: Best AI gets 17%, humans 77%","Photorealistic Tokyo benchmark reveals AI navigation gap","City-scale navigation benchmark floors Gemini 2.5 Flash"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000619,"raw_usage":{"total_tokens":2844,"prompt_tokens":892,"completion_tokens":1952,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":1855}},"tokens_in":508,"tokens_out":1952,"duration_ms":17036,"temperature":1.0,"reasoning_tokens":1855,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:23:29.084941+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of 360CityArena tasks, run them in the real Akihabara district with a human or instrumented recorder, and check whether every ground-truth answer — landmark identity, grid cell, route reachability, and object count — matches what is actually on the ground. Any systematic mismatch, such as a Map Navigation route that cuts through a building or a landmark appearing at the wrong corner, would invalidate the benchmark as a real-world proxy.","supporting_citations":[{"cited_title":"MTAP85, 149 (2026)","cited_arxiv_id":null,"evidence_quote":"Supplies the Realistic Virtual World of Akihabara built from 602 360-degree video segments, the substrate for the benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPT-5 is a closed-source baseline agent and the model used for fuzzy-match evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Claude Sonnet 4.5 is one of the proprietary baseline agents benchmarked."},{"cited_title":"In: CVPR","cited_arxiv_id":null,"evidence_quote":"Supplies the mean relative accuracy metric used for the Object Count task."},{"cited_title":"In: ICLR (2024)","cited_arxiv_id":null,"evidence_quote":"WebArena provides the exact-match and fuzzy-match evaluation methodology adopted here."}],"review_version":1}