{"id":"d813bdda-729b-41de-981d-d4dc106e9f03","arxiv_id":"2511.17384","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"On IndustryNav, a dynamic Unity warehouse navigation benchmark, nine VLLMs earned only 4.9–65.3% success and high collision/warning rates, with closed-source models ahead.","lead":"IndustryNav is a new 12-scene Unity warehouse benchmark in which visual language models navigate around moving forklifts, robots, and workers using an egocentric camera plus exact coordinates. Nine tested VLLMs missed targets often and collided frequently, showing that current models are not yet safe for dynamic industrial navigation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Action-space inconsistency in the navigation prompt is the key confound: the main VLLM deficiency results may reflect an undefined action interface rather than spatial reasoning.","rationale":"The reader's verdict is CONDITIONAL and the weakest_assumption already flags the action-space inconsistency; my stress-test isolates it as the single most load-bearing flaw because it directly threatens the measurement validity of every numeric result in Table 2, not merely statistical power. The paper would remain a useful benchmark contribution, but the empirical claims require either a corrected re-evaluation or a detailed handling/validation of undefined actions. Therefore no verdict change; conditional stands.","tokens_in":19071,"tokens_out":4707,"duration_ms":41775,"concrete_test":"Instrument the evaluation loop to record every raw JSON action emitted by each model. Count occurrences of 'back', 'strafe right', and 'strafe left'. Then re-run a fixed subset (e.g., 2 scenes × 4 start-target pairs = 8 episodes per model) with the prompt corrected so the output action set and the ACTIONS & DYNAMICS table both contain only forward/turn left/turn right/stop, and compare per-model SR, CR, WR. If any model emits >5% undefined actions in the original logs, or if the corrected prompt changes SR by >5 points or CR by >3 points for any model, the headline deficiency results are substantially confounded by the action-space mismatch.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's central empirical claims rest on the validity of the evaluation protocol, but the action interface is internally inconsistent. Section 3.2 defines the agent's discrete action space as 'forward, turn left, turn right, and stop' and gives an Action–State Mapping only for these four. Yet the actual navigation prompt in Fig. 11 (also Fig. 12) instructs the model to output exactly one action from '<forward|back|strafe right|strafe left|stop>', with no dynamics table entries for back or strafe. The paper never reports how these undefined actions were parsed, mapped, or penalized. If models—especially instruction-following ones—emit back or strafe actions, those steps are either ignored (freezing the agent, inflating Average Steps and distance ratios) or mapped to something unexpected (causing collisions and warnings). Both paths directly lower Success Ratio, increase Collision Ratio, and worsen Warning Ratio for exactly the models that take the prompt literally. The claimed universal deficiency, and the closed-source vs. open-source gap in Table 2, are therefore not clean evidence about spatial reasoning; they could be artifacts of a broken interface. This is an internal inconsistency, not a matter of differing consensus, and it is checkable from existing logs.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"IndustryNav introduces a Unity-based benchmark of 12 manually constructed warehouse scenes with dynamic objects and workers, together with a zero-shot PointGoal navigation pipeline that feeds egocentric images, global odometry text, and action-state history to VLLMs. Five metrics are proposed: Success Ratio, Distance Ratio, Average Steps, Collision Ratio, and Warning Ratio. Nine VLLMs are evaluated (five closed-source, four open-source) on four start–target pairs per scene with 70-step episodes. The paper reports that no model reaches high success, closed-source models outperform open-source ones, Nemotron-nano-12B is the best open-source model, and safety remains poor. Ablations show action-state history helps and a top-down minimap does not. The supplementary material includes prompts, additional cases, further ablations, and a candid limitations section.","tokens_in":19380,"tokens_out":5450,"duration_ms":57247,"significance":"If the benchmark and evaluation are sound, IndustryNav would fill a real gap: existing spatial-reasoning benchmarks are mostly passive and household-oriented, while industrial navigation with moving obstacles and safety-oriented metrics is practically important. The proposed Collision Ratio and Warning Ratio are useful additions beyond standard success/efficiency metrics, and the paper contains concrete case analyses and ablations. The strengths are the manually built dynamic warehouse assets, the effort to combine local egocentric perception with global odometry, and the explicit attention to safety. However, the paper's central empirical conclusions are currently under-supported by the evaluation protocol and clouded by an internal action-space inconsistency.","major_comments":[{"comment":"The defined action space is internally inconsistent. Sec. 3.2 states that the agent selects from {forward, turn left, turn right, stop} and provides an Action–State Mapping only for those four actions. The actual navigation prompt in Fig. 11 (and the minimap prompt in Fig. 12) instructs the model to output exactly one action from {forward, back, strafe right, strafe left, stop}, with no dynamics or state-transition rules for back or strafe. The paper never reports how these additional actions were parsed, mapped, or penalized. This is not a cosmetic mismatch: a model that follows the literal prompt may emit back or strafe. If those outputs were ignored, the agent freezes, inflating Average Steps and damaging Distance Ratio; if they were mapped to arbitrary motions, collisions and warnings would be artificially generated. Since the load-bearing claims in Sec. 4.1 — universal deficiency an","section":"Sec. 3.2 vs. Fig. 11/Fig. 12"},{"comment":"The evaluation is statistically underpowered, with no error bars, confidence intervals, or significance tests. Each model is run once on each of 4 start–target pairs per scene, i.e., N=48 episodes per model. For binary success outcomes, the standard error at p≈0.65 and N=48 is about 6.9 percentage points. Differences such as Claude-Haiku-4.5 at 61.81% vs. Gemini-2.5-flash at 65.28% are therefore within one standard error. The conclusion that closed-source models 'consistently outperform' open-source models and the claim that Nemotron 'approaches' closed-source performance are not established without such statistical support. Additionally, no non-VLLM baseline is reported; a simple heading-to-target policy or a random policy is needed to calibrate whether the low success ratios reflect a deficiency of VLLM spatial reasoning or the inherent difficulty of the episodes under the 70-step cap.","section":"Sec. 4.1, Table 2"},{"comment":"No code, scene files, prompts, raw logs, or evaluation harness are released. For a benchmark paper whose contribution is the 12 Unity warehouse scenarios and the navigation pipeline, the absence of an artifact link is a major reproducibility gap. The community cannot audit the collision detection, warning ROI, action parsing, or trajectory computation without the simulator and scenes. The manuscript should include a release URL or explain clearly why the benchmark cannot be released; without this, the benchmark is not usable as proposed.","section":"Sec. 3.1, Sec. 4.1"},{"comment":"The construct validity of 'active spatial reasoning' is weaker than claimed because the pipeline supplies exact coordinates, target coordinates, and distance-to-target as text at every step. Under this protocol, much of the global reasoning can be performed by language-level arithmetic (current heading vs. desired heading and distance minimization) rather than visual-spatial reasoning from the egocentric image. The qualitative 'distance estimation deficiency' discussion in Sec. 4.2 is then hard to reconcile with the fact that distance is explicitly provided numerically. Either the pipeline should ablate the textual odometry/distance information, or the paper should frame the task as testing instruction-following plus local visual obstacle avoidance rather than holistic spatial reasoning.","section":"Sec. 3.2, Sec. 4.2"}],"minor_comments":[{"comment":"The abstract says 'fourteen state-of-the-art VLLMs (including models such as GPT-5.2, Claude-4.6, and Gemini-3)', while the main text and Table 2 evaluate nine models, named GPT-5-mini, Claude-4.5, and Gemini-2.5-flash. These numbers and model names must be aligned.","section":"Abstract"},{"comment":"The results analysis says 'as shown in Tab. 1', but the performance table is Table 2; Table 1 is the comparison with prior benchmarks. Please fix the cross-reference.","section":"Sec. 4.1"},{"comment":"The text refers to 'seven evaluated embodied agents' and mentions 'GPT-3.5-mini', while the main evaluation has nine models including GPT-5-mini. Also, some displayed headings are 'θ=360.0°' and 'Heading 360.0°(West)', although the coordinate system in Sec. 3.2 defines West as θ=0°; normalize angles to [0,360).","section":"Appendix B.1, Fig. 13"},{"comment":"Collision Ratio only counts collisions when a forward action is executed and position does not change. Collisions caused by dynamic vehicles or workers moving into the agent while it is stationary or turning are not counted. This should be stated explicitly, and the metric definition should be justified.","section":"Sec. 3.3, Eq. (4)"},{"comment":"The warning threshold is given as '1 meter', but the depth estimates come from Depth Pro and the ROI geometry is not defined. Please specify how the metric depth is calibrated and the exact ROI used, including its size and placement relative to the agent.","section":"Sec. 3.3, Eq. (5) and Sec. 4.1"},{"comment":"The limitation that 'the environment remains static between steps' is important and should appear in the main paper. The word 'dynamic' in the title and abstract should be qualified accordingly; the benchmark is step-based with discrete dynamics, not a continuous asynchronous environment.","section":"Appendix C.3"}],"recommendation":"major_revision","confidential_remarks":"The action-space inconsistency is the most serious issue because it directly affects the numerical results in Table 2. If the authors can provide logs showing that no model emitted back or strafe, or can re-run with a consistent action space and report parsing rules, the central comparison may survive. The missing artifact release is also a blocking issue for a benchmark paper. I do not see grounds for rejection beyond these fixable problems, but the revision needs to be substantive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: useful benchmark idea with a load-bearing evaluation flaw. The Unity warehouse scenes, the collision/warning metrics, and the nine-model sweep are genuinely new, and the limitations appendix is honest. But the paper has an internal contradiction in the action interface. Section 3.2 defines only forward, turn left, turn right, and stop; the actual prompt in Fig. 11 tells the model to output back, strafe right, or strafe left as well, with no dynamics defined for those. The paper never says how those actions were parsed or mapped. Models that take the prompt literally will emit them, and the resulting behavior — ignored, frozen, or mis-mapped — distorts Success Ratio, Average Steps, Collision Ratio, and Warning Ratio. That makes the headline claim that all VLLMs are deficient, and the closed-versus-open gap, unreliable as evidence about spatial reasoning.\n\nWhat is actually new: IndustryNav is the first dynamic industrial navigation benchmark I know of, the two safety metrics are reasonable additions, and the action-state-history ablation is a plus. The observation that a top-down map does not improve these models is counterintuitive and worth following up.\n\nSoft spots beyond the action-space issue: no code or scene release, no error bars or significance tests, 48 episodes per model with one run per start–target pair, and no non-VLLM baseline, so we cannot calibrate task difficulty. The pipeline also hands exact coordinates and distance to the model as text, which weakens the 'spatial reasoning' construct — it becomes arithmetic plus image-based obstacle avoidance. Minor: the appendix ablation mentions GPT-3.5-mini, a model not in the main table.\n\nThis is for an embodied-AI audience wanting a warehouse-domain evaluation. It deserves serious peer review — the benchmark concept is valuable — but the current experimental results should be treated as preliminary until the action interface is reconciled, assets are released, and statistical grounding is added. My recommendation: send to referees with a request for major revision; don't cite the current numbers as established fact.","headline":"Promising industrial navigation benchmark, but the action-space inconsistency in the prompt makes the headline VLLM deficiency results unreliable until fixed.","tokens_in":19898,"tokens_out":3869,"would_cite":false,"duration_ms":36114,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that nine state-of-the-art VLLMs all exhibit notable deficiencies in robust path planning, collision avoidance, and active exploration in dynamic industrial navigation, based on a new Unity-based benchmark with safety-orien","keywords":["spatial reasoning","embodied agents","navigation benchmark","vision-language models","warehouse simulation","collision avoidance","safety metrics","dynamic environments"],"falsifier":"Re-run the nine models with a single consistent action space (no back/strafe actions), give each model several independent runs per start–target pair, and add a simple greedy planner as a baseline. If the greedy planner reaches near-perfect success while the VLLMs stay below 70%, or if any VLLM jumps above 90% once the prompt is fixed, the paper's deficiency claim would be strong evidence of prompt-sensitivity rather than a fundamental spatial-reasoning gap.","tokens_in":18979,"feed_emoji":"🤖","tokens_out":4654,"duration_ms":45056,"temperature":0.7,"pith_summary":"The paper sets out to show that spatial reasoning in embodied agents is not adequately tested by existing static, household benchmarks and that it remains unsolved for dynamic industrial navigation. To that end it builds IndustryNav, a benchmark of 12 manually designed Unity warehouse scenes with moving vehicles and workers, and proposes a zero-shot PointGoal pipeline that supplies the agent an egocentric image, global coordinates, and action history each step. Nine vision-language models were run through four start–target pairs per scene, and all of them posted low success rates, double-digit collision rates, and frequent proximity warnings. The authors argue this demonstrates a real gap in local–global planning, distance estimation, and active exploration, and they introduce collision rate and warning rate as metrics for future work on safe embodied navigation.","feed_headline":"Nine VLLM agents fall short on dynamic warehouse navigation","feed_subtitle":"New benchmark tracks collisions and near-misses in 12 Unity warehouses; best model succeeds only 65% of the time.","key_machinery":"The load-bearing mechanism is the IndustryNav evaluation pipeline. At each step the agent receives an egocentric 1024×1024 image, a textual global state (position, heading, target, distance), and the last ten action-state pairs; it must output a JSON action from a discrete set (forward, turn left/right, stop) plus a reasoning trace. The benchmark's distinctive instruments are the two safety metrics: Collision Ratio, computed by checking whether a forward action fails to change position, and Warning Ratio, computed by feeding the egocentric image to a monocular depth estimator and flagging frames where any pixel in a forward region of interest is below a one-meter threshold. A fixed red-cone","core_discovery":"The central claim is a measured capability gap: under a zero-shot PointGoal protocol in dynamic warehouse scenes, none of the nine tested VLLMs reaches the target in more than 65% of runs, closed-source models consistently beat open-source ones, and safety behaviors—collisions and near-miss warnings—are poor across all models. The authors interpret this as evidence that current VLLMs lack robust spatial reasoning for active, dynamic environments, particularly in combining local obstacle avoidance with global path planning.","pith_inferences":["The reported model ranking may be underpowered: with one run per start–target pair (48 episodes per model), differences like 54% vs 55% success are within the noise; a reader should treat exact ordering cautiously.","The paper's action-space description is internally inconsistent: Section 3.2 lists forward/turn/stop, while the prompt in Figure 11 (and Figure 12) includes back and strafe actions. If any model attempted those actions, the simulator may have mis-handled them, potentially deflating success scores.","A natural next experiment is to compare against a simple classical planner (e.g., greedy heading toward the target with obstacle avoidance). If such a planner clears the success bar easily, the benchmark is mainly measuring VLLM limitations; if it also struggles, the scenes themselves are unusually hard.","The Warning Ratio could be sharpened into a time-to-collision or motion-prediction metric, which would reward agents for anticipating moving forklifts rather than merely reacting to instantaneous depth."],"forward_implications":["Embodied AI evaluation should include dynamic, domain-specific scenarios with explicit safety metrics, not only static household question-answering.","No current VLLM meets the safety bar for real warehouse deployment: collision rates and warning rates remain far too high.","Closed-source models' consistent edge over open-source models points to a concrete target for open-model development; Nemotron is the closest open baseline.","Action-state histories materially improve success and safety, while adding a top-down map does not—a practical design hint for future navigation pipelines.","The benchmark's 12 scenes and five metrics offer a reusable test suite for the next generation of embodied agents."],"fun_headline_variants":["No VLLM tops 65% success in dynamic warehouse navigation","Nine VLLMs all fail safety in warehouse navigation test","Dynamic warehouse benchmark reveals VLLM spatial reasoning gaps","VLLM agents: best success rate 65% in dynamic industrial navigation"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper's conclusion that VLLMs lack active spatial reasoning rests on the evaluation protocol genuinely measuring that competence: one 70-step run per start–target pair, exact coordinates and distances typed into the prompt, and an action space that differs between the method text and the actual prompt.","fun_headline_variants_meta":{"raw":{"variants":["No VLLM tops 65% success in dynamic warehouse navigation","Nine VLLMs all fail safety in warehouse navigation test","Dynamic warehouse benchmark reveals VLLM spatial reasoning gaps","VLLM agents: best success rate 65% in dynamic industrial navigation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000341,"raw_usage":{"total_tokens":1695,"prompt_tokens":703,"completion_tokens":992,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":447,"completion_tokens_details":{"reasoning_tokens":932}},"tokens_in":447,"tokens_out":992,"duration_ms":8566,"temperature":1.0,"reasoning_tokens":932,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T20:55:47.565260+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the nine models with a single consistent action space (no back/strafe actions), give each model several independent runs per start–target pair, and add a simple greedy planner as a baseline. If the greedy planner reaches near-perfect success while the VLLMs stay below 70%, or if any VLLM jumps above 90% once the prompt is fixed, the paper's deficiency claim would be strong evidence of prompt-sensitivity rather than a fundamental spatial-reasoning gap.","supporting_citations":[],"review_version":1}