{"id":"5f965e26-0e7b-4c14-8580-05f88ea108ad","arxiv_id":"2505.00690","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"URBAN-SIM generates diverse interactive city scenes at high training speed and URBAN-BENCH measures four robot types on eight micromobility tasks, with scale-up training lifting navigation success from 5% to 83%.","lead":"This paper introduces a simulator for training small city robots, like delivery bots and mobility scooters, across endlessly generated urban scenes, plus a benchmark of eight tasks for wheeled and legged robots. It matters because it offers the embodied AI field a shared, high-throughput testing ground for a transportation mode that currently relies on human operation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The scalability/26.3% claim is the load-bearing soft spot: Fig. 7 lacks a controlled comparison and the headline speedup number is not tied to any reported result.","rationale":"Good-faith reading: the paper is a simulation-and-benchmark contribution, and it explicitly limits itself to simulation in Appendix G ('current version focuses solely on simulation and does not include real-world experimentation'), which is consistent with accepted sim-only platforms such as Habitat and ProcTHOR. I therefore do not treat the absence of real-robot experiments as the decisive flaw; the reader's sim-to-real concern is relevant to the long-term goal but is disclosed and not part of the paper's stated deliverable. The decisive issue is internal to the reported evaluation: the platform's signature advantage over IsaacGym/CARLA is the combination of diverse scenes and high throughput, and the only scalability evidence is a comparison whose control conditions are underspecified and whose headline 26.3% number is not tied to any reported result. This does not make the work rejectable, because the artifact is plausible and open-source release is promised; it makes acceptance conditional on a controlled reproduction and on toning down unsupported quantitative claims. I keep the reader's CONDITIONAL verdict, with the reasoning shifted from sim-to-real to the scalability evidence.","tokens_in":29638,"tokens_out":13473,"duration_ms":136440,"concrete_test":"Obtain the released code and rerun the Section 6 experiment on NavStatic with exactly 256 parallel environments and fixed per-step sample count, varying only the number of unique scenes (1, 8, 256, 1,024) and using at least 3 seeds. Record success rate and route completion at matched total environment steps, and compute the async-vs-sync relative improvement at the same step; check whether 26.3% is reproduced and whether the gap exceeds seed noise. Also run the comparison against a synchronous baseline that resets each environment to a freshly sampled unique scene, to isolate the effect of the sampling scheme from scene diversity. If the gap collapses or the 26.3% figure is not reproducible, the abstract and introduction claims should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"URBAN-SIM's claimed novelty is balancing large-scale scene diversity with high training throughput, and the quantitative evidence for this is the asynchronous-sampling comparison in Section 6 / Figure 7 and the 'more than 26.3% relative improvement' asserted in Section 1. That number is never tied to a table, curve annotation, or derivation in the paper. More importantly, the comparison is not described tightly enough to rule out a confound: Section 6 says the number of training scenes is varied from 1 to 1,024, but it does not state whether total parallel environments, per-step sample counts, or GPU resources are held fixed. If scene count and environment count are increased together, the success-rate gain (5.1% to 83.2%) could come from more on-policy samples per gradient step, not from the asynchronous sampling design. Figure 7 also shows no seeds or error bars, and the 'synchronous' baseline is described only as 'all scenes in a batch are identical'; it is not clear how a synchronous sampler that periodically resets environments to fresh unique scenes would behave. In addition, the 1,800-2,600 fps measurement in Appendix C is taken with random actions, so step-only FPS is not by itself evidence about end-to-end training speed. These gaps leave the central 'large-scale, high-performance' claim under-supported rather than disproven.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents URBAN-SIM, a GPU-accelerated urban simulation and robot-learning platform built on Nvidia Omniverse/PhysX, together with URBAN-BENCH, a suite of eight micromobility tasks (four locomotion, three navigation, one kilometer-scale traverse) evaluated on four heterogeneous robots. The platform's three claimed contributions are hierarchical procedural urban generation, GPU-based interactive dynamics via a JAX implementation of ORCA, and asynchronous scene sampling that permits training on many unique scenes in parallel. The central quantitative claims are a training throughput above 1,800 FPS on a single L40S GPU and a relative improvement of more than 26.3% from asynchronous sampling over synchronous approaches. The main empirical evidence is the scalability study in Section 6 (Figure 7), the benchmark tables (Tables 1, 2, and 7), and the performance benchmarks in Appendix C.","tokens_in":29878,"tokens_out":6345,"duration_ms":58166,"significance":"If the scalability and throughput claims hold, URBAN-SIM addresses a real gap: existing platforms tend to offer either fast training in uniform environments (IsaacGym, IsaacLab) or visually rich but slow urban scenes (CARLA, EmbodiedCity). The hierarchical generation pipeline, the GPU ORCA dynamics, and the asynchronous sampling design are plausible and useful engineering contributions. URBAN-BENCH is also a potentially valuable community asset: it covers a wider range of embodiments and longer-horizon tasks than most existing micromobility benchmarks, and the human-AI shared-autonomy study for urban traverse is a welcome addition. The paper is unusually complete in its appendix, with a datasheet, hyperparameters, asset descriptions, and a commitment to open-source the code and scene data. However, the headline numerical claims—the 26.3% improvement and the 1,800 FPS training throughput—are not yet backed by the reported experimental protocol, which is load-bearing for the paper's central thesis.","major_comments":[{"comment":"The paper's headline scalability claim—'more than 26.3% relative improvement compared to synchronous approaches with the same training steps' (Section 1)—is never tied to any reported number, curve, or derivation. In Section 6, the synchronous/asynchronous comparison varies the number of training scenes from 1 to 1,024, but the text does not state whether the total number of parallel environments, per-step sample counts, batch sizes, GPU resources, or random seeds are held fixed. If scene count and environment count are increased together, the success-rate increase from 5.1% to 83.2% could result from more on-policy samples per gradient step rather than from asynchronous sampling per se. In addition, Figure 7 (Left) reports no seeds or error bars, and the synchronous baseline is described only as 'all scenes in a batch are identical'; a baseline that periodically resets to fresh unique scenes would be needed to separate scene diversity from the asynchrony mechanism. Please report the full protocol, per-condition seeds, error bars, and the exact calculation of the 26.3% figure.","section":"Section 6, Figure 7; Section 1"},{"comment":"The abstract and Section 3 state that URBAN-SIM provides 'more than 1,800 fps high training performance' on a single L40S GPU, but Appendix C measures environment-step FPS under random actions ('We sample random actions for 1,000 steps per agent'), not end-to-end training throughput. The curves in Figure 19 include an 'Environment Step, Inference, and Train FPS' condition, but the text never reports those numerical values, so the reader cannot determine the actual training speed. Please report the end-to-end training FPS (including policy inference, PPO updates, and data transfer) for the same configurations, or explicitly re-label the 1,800–2,600 fps claim as simulator stepping speed only.","section":"Section 3.3 and Appendix C"},{"comment":"The manuscript's motivation is autonomous micromobility in real urban spaces, but Appendix G states that 'the current version focuses solely on simulation and does not include real-world experimentation.' The only reported sim-to-real evidence is the sentence 'preliminary experiments ... have already achieved good transferability to real-world environments,' with no results, metrics, or protocol. Given that URBAN-BENCH is presented as revealing each robot's strengths and limitations for autonomous micromobility, the absence of quantitative real-world validation makes those conclusions simulation-only; the transferability sentence should either be substantiated or removed, and the benchmark claims should be framed accordingly.","section":"Appendix G"}],"minor_comments":[{"comment":"The caption reads 'Comparison between synchronous and synchronous scene sampling'; the second instance should be 'asynchronous'.","section":"Figure 7 caption"},{"comment":"The tracking reward uses the notation 'ds−2−d' without a clear definition; please define this distance variable consistently.","section":"Equation (4)"},{"comment":"The locomotion section repeats the navigation training times ('It took about 8, 18, and 24 hours ... for NavClear, NavStatic, and NavDynamic'); the locomotion training times should be reported separately.","section":"Appendix E.5"},{"comment":"The sentence on GPU memory reads 'from 1.6 to 11.2' without units; Figure 17 should label units on both axes and state that the memory values are in GB.","section":"Appendix C.3"},{"comment":"The rows are labeled 'Urban-Loc-2 (Stair)' and 'Urban-Loc-3 (Slope)', which is inconsistent with the order of LocoSlope and LocoStair in Section A.1; please align the naming.","section":"Table 3"},{"comment":"The datasheet states that the dataset is 'self-contained,' while Section B.1 says assets come from external sources such as Objaverse-XL and SynBody; please clarify what 'self-contained' means in this context.","section":"Appendix F (Datasheet)"}],"recommendation":"major_revision","confidential_remarks":"This is a promising systems and benchmark paper whose main problems are experimental reporting rather than fundamental design flaws. The 26.3% number and the 1,800 FPS training-throughput claim should either be substantiated with a fully controlled protocol or removed/relabeled. The sim-to-real issue is a known limitation and can be handled by more careful framing; it should not by itself block acceptance if the platform claims are properly scoped."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper deserves a serious referee, but the most important claims need work. What is actually new is the combination: procedural urban scene generation (WFC-based), JAX-accelerated ORCA for interactive dynamics, asynchronous sampling over hundreds of unique environments, and an eight-task benchmark across four heterogeneous robot types. The pieces are borrowed, but the integrated artifact is not in the prior literature, and for a community that needs training ground for sidewalk robots, that is a real contribution.\n\nWhat the paper does well: the system appears to work. Appendix C shows credible scaling from about 100 FPS on a single environment to over 2,600 FPS with 256 parallel environments, with GPU memory staying at 11.2 GB. The benchmark design is thoughtful, especially the separation into locomotion, navigation, and traverse with a shared human-AI autonomy framework for the long-horizon task. The datasheet and the explicit statement that the current version is simulation-only (Section G) are signs of honest engagement.\n\nThe soft spots are concentrated in the scalability evidence. The headline “more than 26.3% relative improvement” appears only in the introduction and is never tied to a table or curve in the results. Figure 7 shows a striking jump in success rate from 5.1% to 83.2% as the number of training scenes increases from 1 to 1,024, but the comparison is not described tightly enough. If total parallel environments or per-step sample counts are not held fixed when the scene count changes, the gain could come from more on-policy samples per gradient step rather than from the asynchronous design. The synchronous baseline is also described only as “all scenes in a batch are identical,” which leaves room for a strawman comparison. The FPS numbers in Appendix C are measured with random actions, so they show step throughput, not end-to-end training speed. A minor issue: Table 1 compares “Smoothness” across robots with very different joint counts and actuator models, so the cross-robot ranking is not apples-to-apples. The paper also cites the authors’ own MetaUrban but does not discuss how URBAN-SIM differs from it; that should be addressed.\n\nNone of these flaws invalidate the platform. The artifact is real, the benchmark is useful, and the limitations are acknowledged. But the paper’s central claim—balancing large-scale scene diversity with high training throughput—is under-supported rather than disproven. For a reader working on sim-to-real or urban robot learning, this is worth citing and the benchmark is worth adopting. I would accept it for peer review with a required revision of the scaling analysis, including controls for environment count, error bars, and a clearly specified synchronous baseline.","headline":"URBAN-SIM/URBAN-BENCH is a genuinely useful integrated platform and benchmark for micromobility robot learning, but the headline scalability claims are under-supported by the current experimental design and need tightening before they can be taken at face value.","tokens_in":30433,"tokens_out":2473,"would_cite":true,"duration_ms":25241,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces URBAN-SIM, a simulation platform that generates infinite diverse interactive urban scenes and trains embodied agents at over 1,800 frames per second on one GPU, together with URBAN-BENCH, eight tasks revealing how…","keywords":["autonomous micromobility","urban simulation","robot learning platform","procedural urban scene generation","asynchronous scene sampling","reinforcement learning","embodied AI benchmark","sim-to-real transfer"],"falsifier":"Deploy the trained navigation and locomotion policies on a physical quadruped and wheeled robot over sidewalks with stairs, slopes, pedestrians, and static obstacles, and compare success rate, collision count, and route completion against the same metrics in URBAN-SIM; a large drop in success rate or a rise in collisions under comparable conditions would show the simulator's training signal does not carry to the target application.","tokens_in":29427,"feed_emoji":"🤖","tokens_out":6074,"duration_ms":57250,"temperature":0.7,"pith_summary":"The paper tries to establish that autonomous micromobility in urban public spaces can be advanced by a single simulation platform that reconciles two demands previously in tension: generating large numbers of diverse, interactive city scenes and training embodied agents in them fast. It introduces URBAN-SIM, which procedurally assembles urban scenes through four hierarchical stages, populates them with responsive pedestrians and cyclists, and samples scenes asynchronously on GPUs, reaching over 1,800 frames per second on one L40S GPU. It then defines URBAN-BENCH, eight tasks in locomotion, navigation, and long-horizon traverse, and benchmarks four robots with different bodies. A reader would care because it offers a way to train and compare sidewalk-scale robots at near-real-time speed in environments rich enough to expose skills like stair climbing, detouring, and sidestepping.","feed_headline":"Urban robot training hits 1,800 fps in procedurally built cities","feed_subtitle":"A new platform pairs infinite generated sidewalk scenes with GPU sampling to train and benchmark four robot designs.","key_machinery":"The load-bearing mechanism is asynchronous scene sampling, supported by a hierarchical urban generation pipeline and GPU-based interactive dynamics. In contrast to synchronous sampling used by existing GPU platforms, where every environment in a batch is identical, URBAN-SIM caches all assets and randomly samples unique configurations per environment, fully vectorizing observations, rewards, and actions on the GPU. This is what lets the platform run 256 non-uniform environments in parallel, reach 1,800–2,600 fps with RGBD sensors, and gain a 26.3% relative improvement over synchronous approaches at the same training steps. The Wave Function Collapse terrain generator, a constraint-based procedural generation algorithm, and the ORCA reciprocal collision-avoidance algorithm, ported to run on GPUs, supply the variety and interactivity that make the large-scale sampling meaningful.","core_discovery":"The central claim is that scale and performance are not mutually exclusive for urban robot learning if scene diversity is generated procedurally and scene sampling is asynchronous. URBAN-SIM's hierarchical pipeline connects street blocks, plans functional ground zones, applies Wave Function Collapse to create flat, stair, slope, and rough terrains, and places objects from a 15,000-asset repository, producing an effectively infinite range of static scenes. Its interactive dynamics strategy runs a GPU port of ORCA so pedestrians and cyclists respond to the robot in real time, and its asynchronous sampling lets each of 256 parallel environments be unique rather than identical copies. Across the eight URBAN-BENCH tasks the paper reports that the four robots show distinct strengths: the quadruped is smoothest, the wheeled-legged robot is most versatile, the humanoid handles static and dynamic obstacles best, and the wheeled robot excels on clear pathways. The paper also reports that scaling the number of unique training scenes from 1 to 1,024 raises success rate on the static-obstacle navigation task from 5.1% to 83.2%, which it presents as evidence that large-scale diversity is itself a training signal.","pith_inferences":["Editorial inference: The paper's own foundational-task results suggest a testable extension: a modular policy that chains small-scene navigation and terrain-specific locomotion models should outperform an end-to-end policy trained on the full kilometer-scale task, which is exactly the design of their shared autonomy modes.","Editorial inference: Because the paper reports no real-world experiments, the most consequential open question is whether the qualitative skills it observes, sidestepping, detouring, and stair climbing, transfer to physical sidewalks; a natural next experiment is deploying the trained depth-map policies on real robots with domain randomization and measuring the success-rate drop.","Editorial inference: The platform's open architecture invites other task definitions beyond the eight, such as curb-ramp finding or intersection crossing, which the paper itself lists as user-defined possibilities."],"forward_implications":["If URBAN-SIM works as described, a single GPU can support large-scale training across hundreds of unique urban scenes at near-real-time speed, removing the need for uniform, repetitive environments in micromobility research.","URBAN-BENCH's eight tasks provide a common yardstick: four heterogeneous robots can be trained and compared on locomotion, navigation, and kilometer-scale traverse, with metrics for completeness, efficiency, and safety.","Training on more diverse scenes measurably improves navigation performance, from 5.1% to 83.2% success rate as the scene count grows from 1 to 1,024, implying that scene diversity is a scaling axis of its own.","The human-AI shared autonomy modes for the urban traverse task demonstrate a middle path where human labor falls from 1,189 seconds to about 189 seconds while collision counts remain below the fully AI mode."],"supporting_citations":[{"why":"Supplies the high-performance GPU simulation baseline whose uniform environments the paper contrasts with its asynchronous unique-scene sampling.","marker":"[44]"},{"why":"Supplies the synchronous scene sampling baseline used in the scalability comparison.","marker":"[51]"},{"why":"Provides the reference point for rich urban scenes with low end-to-end training efficiency, motivating the scale-performance balance.","marker":"[21]"},{"why":"Supplies the reciprocal collision-avoidance algorithm that underlies the interactive dynamics generation.","marker":"[79]"},{"why":"Provides the GPU-based simulation approach that the paper follows in porting ORCA.","marker":"[26]"},{"why":"Supplies the multi-agent reinforcement learning environment library used for the GPU implementation of ORCA.","marker":"[61]"},{"why":"Provides the Wave Function Collapse algorithm used to generate terrain variety in the hierarchical pipeline.","marker":"[27]"},{"why":"Supplies the reinforcement learning algorithm used to train all navigation and locomotion policies.","marker":"[63]"},{"why":"Supplies the physics engine underlying the platform's dynamics and robot-environment interaction.","marker":"[54]"}],"fun_headline_variants":["Scaling generated cities lifts robot success from 5% to 83%","1,024 unique scenes make robot navigation 16x better","Async simulation spawns 256 distinct training worlds for robots","Procedural sidewalks train robots to dodge obstacles in 1,024 cityscapes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that skills learned and metrics measured inside this simulator transfer to real sidewalks, since the paper reports no real-world experiments and only a one-sentence note of preliminary sim-to-real results.","fun_headline_variants_meta":{"raw":{"variants":["Scaling generated cities lifts robot success from 5% to 83%","1,024 unique scenes make robot navigation 16x better","Async simulation spawns 256 distinct training worlds for robots","Procedural sidewalks train robots to dodge obstacles in 1,024 cityscapes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001449,"raw_usage":{"total_tokens":5876,"prompt_tokens":1028,"completion_tokens":4848,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":4770}},"tokens_in":644,"tokens_out":4848,"duration_ms":31670,"temperature":1.0,"reasoning_tokens":4770,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:36:23.006703+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Deploy the trained navigation and locomotion policies on a physical quadruped and wheeled robot over sidewalks with stairs, slopes, pedestrians, and static obstacles, and compare success rate, collision count, and route completion against the same metrics in URBAN-SIM; a large drop in success rate or a rise in collisions under comparable conditions would show the simulator's training signal does not carry to the target application.","supporting_citations":[{"cited_title":"Isaac gym: High performance gpu based physics simulation for robot learning","cited_arxiv_id":null,"evidence_quote":"Supplies the high-performance GPU simulation baseline whose uniform environments the paper contrasts with its asynchronous unique-scene sampling."},{"cited_title":"Orbit: A unified simulation framework for interactive robot learning environments.RAL,","cited_arxiv_id":null,"evidence_quote":"Supplies the synchronous scene sampling baseline used in the scalability comparison."},{"cited_title":"Reciprocal n-body collision avoidance","cited_arxiv_id":null,"evidence_quote":"Supplies the reciprocal collision-avoidance algorithm that underlies the interactive dynamics generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the physics engine underlying the platform's dynamics and robot-environment interaction."}],"review_version":1}