{"id":"485954b8-867e-4d10-9f51-17f86df11171","arxiv_id":"2603.03953","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A collision-aware indoor visual navigation benchmark on Habitat 2.0/HM3D with RL environments, offline trajectory datasets, and negative collision datasets.","lead":"RVN-Bench is a new simulator benchmark that scores indoor robots on reaching goals while avoiding collisions using only a camera. It adds collision-aware metrics, a reinforcement-learning training environment, and a generator for negative trajectory datasets that end in crashes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Goal vector P_t is privileged and unspecified, contradicting the 'visual-only' claim","rationale":"The reader's weakest_assumption identifies the same issue: the benchmark silently assumes a ground-truth relative goal vector, which contradicts the 'visual-only' claim. This is the most load-bearing concern because it strikes at the central wording of the contribution: RVN-Bench is described as a reactive visual navigation benchmark where the agent uses only vision, yet the task definition includes a privileged non-visual input. The internal inconsistency between Table II and Table III is a data-quality issue but does not alter the benchmark's definition; the real-world evaluation is limited but explicitly presented as initial evidence. The privileged P_t concern affects the interpretability of all downstream results and is fixable in a revision by clarifying the input specification, renaming the claim, or adding a noisy-goal variant. Since this concern is already flagged by the reader and the verdict is CONDITIONAL, no verdict change is needed.","tokens_in":12326,"tokens_out":7549,"duration_ms":71580,"concrete_test":"On the RVN-Bench test split, re-run the DDPPO-DA V2 evaluation while replacing the exact P_t with a noisy version P_t + ε, where ε ~ N(0, σ²I) for σ = 0.1, 0.5, and 1.0 m, keeping visual observations unchanged. Record SR1 and E(G). If SR1 degrades sharply with increasing σ (e.g., >50% relative drop at σ=0.5 m), this confirms that the reported 0.928 SR1 relies on the privileged perfect goal vector and cannot be achieved from visual observations alone. Additionally, inspect the benchmark code to verify that P_t is computed from ground-truth pose rather than a vision-based estimate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-A states: 'Along with image observations, the agent is given the goal position specified relative to its current pose, P_t = (x_g, y_g).' This is a privileged, non-visual input, typically computed in Habitat from the ground-truth agent pose (GPS+compass). The abstract's claim that the agent navigates 'using only visual observations and no prior map' is therefore contradicted by the task definition. The benchmark actually measures reactive obstacle avoidance under a perfect goal-localization oracle, not end-to-end visual navigation. The paper never specifies how P_t is obtained in simulation or in the real-world Jackal experiments (Section IV-E), where NoMaD-PointGoal also requires this vector. This missing specification affects the interpretation of both the benchmark's core claim and the sim-to-real transfer results, since real deployment may not have access to such a clean goal signal.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RVN-Bench, a collision-aware benchmark for indoor reactive visual navigation built on Habitat 2.0 and the HM3D dataset. The benchmark defines a sequential PointGoal-style task in which an agent receives RGB observations and a relative goal vector, must reach up to 32 goal positions while avoiding collisions, and is evaluated with success rate, expected number of goals reached, and collisions per kilometer. The paper also provides an RL environment, a trajectory-image dataset generator including negative collision trajectories, and a new baseline NoMaD-Neg. Experiments compare IL, Safe-RL, and RL baselines, reporting that DDPPO-DA V2 performs best, and that a NoMaD-PointGoal model trained on simulated data transfers to a Jackal UGV in 20 real-world episodes.","tokens_in":12605,"tokens_out":4790,"duration_ms":46342,"significance":"If the claims hold, RVN-Bench is a useful contribution: it is a large-scale (800 train / 50 val / 50 test scenes) collision-aware indoor visual navigation benchmark with standardized metrics, RL training, and offline dataset generation, plus a public code release. The paper also provides a concrete negative-data mechanism and an initial sim-to-real study, which are valuable for the community. The benchmark is empirical and I did not find circular derivations of reported metrics from fitted parameters. However, the central 'only visual observations' framing is not supported by the task definition, and two reporting issues weaken the experimental support. These are fixable but must be addressed before the paper can be accepted.","major_comments":[{"comment":"The abstract and Section III-A state that the agent navigates 'using only visual observations', but the task gives the agent a privileged goal vector P_t = (x_g, y_g) relative to its current pose. This is typically provided by GPS+compass in Habitat PointGoal, not by vision. The manuscript never specifies how P_t is obtained in simulation or in the real-world Jackal experiments (Section IV-E), where NoMaD-PointGoal also requires this input. As written, RVN-Bench evaluates reactive obstacle avoidance under a perfect goal-localization oracle, not end-to-end visual navigation. Please either remove the 'only visual observations' claim, explicitly model P_t as an additional input modality, or specify the sensor/mechanism and discuss the implications for sim-to-real transfer.","section":"Abstract and Section III-A"},{"comment":"The same configuration is reported inconsistently. Table II lists 'DDPPO-DA V2' with Train E(G)=20.79, while Table III lists 'RGB + Predicted Depth' for DD-PPO with Train E(G)=22.70; other columns (validation/test E(G), CPK, SR1) are identical. One of these is a typo, but the discrepancy undermines the reproducibility of the main quantitative comparison. Correct the inconsistency and state which value is used in the main analysis.","section":"Tables II and III"},{"comment":"The sim-to-real conclusion is based on 20 episodes (15 office, 5 house) with no error bars, confidence intervals, or per-environment breakdown. With n=20, the differences between Sim-only (SR1=0.60) and Real+Sim (SR1=0.75) are not shown to be statistically meaningful, and the CPK values (196–223) likely have large variance. Please report the raw per-episode results or confidence intervals, and soften the '3.5x/4.3x improvement' claims accordingly.","section":"Section IV-E and Table IV"},{"comment":"The task definition says the goal is 'randomly selected under certain rules' and the geodesic distance is in [d_min, d_max], but it does not specify how the goal position is sampled relative to obstacles or whether the same goal-sampling procedure is used in training and evaluation. Since the benchmark's purpose is standardized comparison, the exact sampling protocol (e.g., rejection sampling, minimum clearance, or uniform over navigable positions) should be stated precisely.","section":"Section III-A and III-B"}],"minor_comments":[{"comment":"The real-world text uses N_g = 5 while the benchmark uses N_goal = 32; use consistent notation.","section":"Section IV-E"},{"comment":"The statement that NoMaD-Neg 'replaces 6.25% of expert trajectories with negative ones' is vague; specify the absolute counts for reproducibility.","section":"Section III-C"},{"comment":"The CoR formula is cited from [27] but the displayed equation is missing an explicit division operator in the denominator; also define the value of alpha used in the experiments.","section":"Equation (1)"},{"comment":"The figure caption and labels are difficult to parse; ensure all symbols (t_i, t_f, k_pre, k_post) are defined in the caption or the surrounding text.","section":"Figure 2"},{"comment":"The RL baselines are said to use 'the same hyperparameters as reported in [16]', but for a benchmark it would be helpful to list the key hyperparameters (learning rate, batch size, number of timesteps, seeds) in the appendix or a table.","section":"Section IV-B"}],"recommendation":"major_revision","confidential_remarks":"The benchmark itself is likely to be useful to the community, and the paper has a solid empirical core. The main risk is overselling the 'visual-only' property of the task. The goal-vector issue should be resolved or explicitly scoped, and the data inconsistencies fixed, before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: RVN-Bench is a real contribution -- a collision-aware, indoor, RGB-based evaluation harness with sequential goals and a useful negative-trajectory generator -- but the paper overstates the \"visual-only\" framing, has a numbers inconsistency between the main tables, and its real-world comparison is confounded by data volume. It deserves a serious referee, but needs a revision.\n\nWhat's actually new: the benchmark fills a genuine gap between Habitat Challenge (which ignores collisions) and outdoor simulators. The negative-trajectory dataset generator, where the occupancy map is padded with a margin smaller than the agent radius so paths end in collisions, is a neat and practical capability. The benchmark itself is well designed: 800/50/50 HM3D split, 32 sequential goals per episode, and CPK is a sensible safety metric. The baseline sweep across RL, Safe-RL, and IL is thoughtful, and the finding that DD-PPO with estimated depth reaches 0.928 SR1 on test scenes gives the field a challenging but tractable target.\n\nSoft spots, in order of size. First, the task definition gives the agent P_t = (x_g, y_g) -- a privileged relative goal coordinate -- and the paper never says how it is obtained in simulation or in the Jackal experiments. That contradicts the \"using only visual observations\" claim in the abstract; the benchmark actually measures reactive obstacle avoidance under a perfect goal-localization oracle. That is still a legitimate benchmark, but the claim needs to be scoped.\n\nSecond, the numbers don't line up: DDPPO-DA V2's training E(G) is 20.79 in Table II but 22.70 in Table III for the same configuration. One of these is wrong, and it makes the experimental section harder to trust at face value. This should be caught in revision.\n\nThird, the real-world comparison is confounded. The sim-only model used 828 hours of collected data; the real-only model used 54 hours. So \"sim beats real\" is mostly a data-volume effect, not a fair test of domain transfer. Also, 20 episodes with no error bars is thin, though acceptable as an initial sanity check.\n\nMinor: no seed-level error bars in the simulation tables, and the CoR selection for NoMaD-Neg is imported from the authors' MixGAIL paper without much justification. That's okay for a baseline.\n\nWho it's for: researchers building or using indoor visual navigation benchmarks, and anyone doing safe RL for mobile robots. The benchmark artifact is more durable than this write-up, and the issues are addressable. Send it to review.","headline":"Useful collision-aware indoor navigation benchmark, but the 'visual-only' claim is undercut by a ground-truth goal vector, and one key table number doesn't reproduce.","tokens_in":13064,"tokens_out":3568,"would_cite":true,"duration_ms":35703,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RVN-Bench makes the case that reactive visual navigation—reaching sequential goals in unseen indoor scenes from camera images alone, with no map—should be judged by collisions as much as by goal success, and its experiments show the task is","keywords":["reactive visual navigation","collision-aware benchmark","indoor mobile robots","sequential goal navigation","reinforcement learning","imitation learning","negative trajectory dataset","sim-to-real transfer"],"falsifier":"A concrete check would be to re-run the reported evaluations while replacing the exact relative goal vector with one corrupted by realistic odometry noise or estimated by a vision-based goal detector; if first-goal success and CPK degrade sharply on the same held-out scenes, the benchmark's scores are being carried by the goal oracle rather than by visual navigation.","tokens_in":12265,"feed_emoji":"🤖","tokens_out":11288,"duration_ms":103253,"temperature":0.7,"pith_summary":"RVN-Bench tries to establish that reactive visual navigation—reaching a sequence of goals in a never-seen indoor space using only camera images and no map—can be measured as a collision-aware problem, and that this formulation is learnable. The paper argues that existing indoor visual-navigation benchmarks ignore collisions and outdoor benchmarks skip indoor clutter, so a standardized simulator with collision detection is needed. Its experiments show that reinforcement-learned policies trained in the benchmark generalize to unseen scenes, with the best baseline succeeding on about 93% of first goals and incurring 3.6 collisions per kilometer on test scenes, and that a model trained on simulated data transfers to a real indoor robot better than one trained on real data alone. This gives the community a common testbed for safety-aware navigation and a way to generate collision-ending 'negative' trajectory data that is costly to collect in the real world.","feed_headline":"Collision-aware navigation benchmark: 93% first-goal success","feed_subtitle":"Robots must navigate by sight, goal after goal, without crashing; best policy transfers from simulation to reality","key_machinery":"The load-bearing object is the benchmark's collision-aware sequential-goal environment: a high-fidelity indoor simulator with discretized nonholonomic actions, a cylindrical robot model, and collision detection via a precomputed navigation mesh (a geometric walkability constraint) that restricts the endpoint of each forward step. The paired mechanisms are the relative-goal observation, given to the agent at every step, and the two dataset generators: expert trajectories come from shortest paths on an occupancy map padded beyond the robot radius, while negative trajectories come from paths padded below the radius so the agent collides; recording pre- and post-collision frames yields data that","core_discovery":"The paper's central claim is that safe indoor navigation should be evaluated as a reactive, collision-aware sequential-goal task: at each episode the agent gets a stack of RGB frames and a relative goal vector (x_g, y_g), picks from move-forward/turn-left/turn-right/stop, and must reach multiple goals in an unseen scene, with the episode terminating on collision, timeout, or N goals. To support this, RVN-Bench provides an RL training environment, a trajectory-image dataset generator, and, distinctively, a negative-trajectory generator that manufactures collision-ending trajectories by padding the occupancy map less than the robot radius. The experiments show the best model—a distributed RL a","pith_inferences":["The paper leaves implicit that the relative goal vector is a privileged signal; if a real robot has to infer the goal from vision or odometry, first-goal success and collision rates would likely change, so the benchmark's 'reactive visual navigation' claim should be read as 'vision-based obstacle avoidance with a goal oracle.'","Because collision detection is a geometric navigation-mesh constraint rather than physics-based contact, collisions with lightweight, pushable, or deformable objects are not represented; the benchmark's numbers therefore describe static-scene geometry, and real-world clutter may differ.","The benchmark's configurable robot footprint, camera parameters, and action discretization make it straightforward to extend beyond the single platform and static scenes that the paper names as future work; the same structure could support dynamic obstacles or a goal-image variant that removes the oracle goal vector.","The negative-trajectory mechanism is a transferable idea: deliberately generate failing trajectories, train an expert policy and a 'bad' policy, then use the contrast to reject unsafe actions at inference; the paper applies it only to navigation, but the scheme could be used in other imitation-learning settings where failures are expensive."],"forward_implications":["If RVN-Bench is taken as a standard testbed, navigation research can compare methods on safety—collisions per kilometer—rather than only on reaching goals, closing a gap the paper identifies in existing indoor benchmarks.","Simulation-collected data appears to substitute for expensive real-world collision data: a model trained only on RVN-Bench-generated data outperformed a model trained on 54 hours of real navigation data in real-world first-goal success, and combining datasets helped further.","Negative trajectories, which end in collisions and are hard or unsafe to collect on real robots, can be generated cheaply in simulation and improve imitation-learning agents when used as an auxiliary dataset.","Predicted monocular depth markedly improves collision avoidance, and ground-truth depth improves it further, so continued progress in depth estimation should directly translate into safer reactive navigation.","The benchmark leaves the collision-aware task open: even the strongest baseline reaches only about 93% first-goal success and 3.6 CPK on test scenes, so there is measurable headroom for future policies."],"fun_headline_variants":["Collision-aware benchmark for reactive visual navigation in unseen homes","Indoor robot nav benchmark: avoid crashes, reach sequential goals","New benchmark trains robots to navigate by sight without maps or collisions","RVN-Bench: testing collision-free sequential goal navigation in new scenes","Benchmark for visual robot navigation that punishes crashes, enables sim-to-real"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"In the task definition, the load-bearing premise is that the agent is handed the goal position relative to its current pose at every step—perfect localization of both robot and target—even though the benchmark is described as visual-only navigation.","fun_headline_variants_meta":{"raw":{"variants":["Collision-aware benchmark for reactive visual navigation in unseen homes","Indoor robot nav benchmark: avoid crashes, reach sequential goals","New benchmark trains robots to navigate by sight without maps or collisions","RVN-Bench: testing collision-free sequential goal navigation in new scenes","Benchmark for visual robot navigation that punishes crashes, enables sim-to-real"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000632,"raw_usage":{"total_tokens":2753,"prompt_tokens":743,"completion_tokens":2010,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":1934}},"tokens_in":487,"tokens_out":2010,"duration_ms":13696,"temperature":1.0,"reasoning_tokens":1934,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T18:57:39.046028+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check would be to re-run the reported evaluations while replacing the exact relative goal vector with one corrupted by realistic odometry noise or estimated by a vision-based goal detector; if first-goal success and CPK degrade sharply on the same held-out scenes, the benchmark's scores are being carried by the goal oracle rather than by visual navigation.","supporting_citations":[],"review_version":1}