REVIEW 4 major objections 5 minor 34 references
RVN-Bench makes the case that reactive visual navigation—reaching sequential goals in unseen indoor scenes from camera images alone, with no map—should be judged by collisions as much as by goal success, and its experiments show the task is
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 18:57 UTC pith:NDNBS2IH
load-bearing objection Useful collision-aware indoor navigation benchmark, but the 'visual-only' claim is undercut by a ground-truth goal vector, and one key table number doesn't reproduce. the 4 major comments →
RVN-Bench: A Benchmark for Reactive Visual Navigation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that safe indoor navigation should be evaluated as a reactive, collision-aware sequential-goal task: at each episode the agent gets a stack of RGB frames and a relative goal vector (x_g, y_g), picks from move-forward/turn-left/turn-right/stop, and must reach multiple goals in an unseen scene, with the episode terminating on collision, timeout, or N goals. To support this, RVN-Bench provides an RL training environment, a trajectory-image dataset generator, and, distinctively, a negative-trajectory generator that manufactures collision-ending trajectories by padding the occupancy map less than the robot radius. The experiments show the best model—a distributed RL a
What carries the argument
The load-bearing object is the benchmark's collision-aware sequential-goal environment: a high-fidelity indoor simulator with discretized nonholonomic actions, a cylindrical robot model, and collision detection via a precomputed navigation mesh (a geometric walkability constraint) that restricts the endpoint of each forward step. The paired mechanisms are the relative-goal observation, given to the agent at every step, and the two dataset generators: expert trajectories come from shortest paths on an occupancy map padded beyond the robot radius, while negative trajectories come from paths padded below the radius so the agent collides; recording pre- and post-collision frames yields data that
Load-bearing premise
In the task definition, the load-bearing premise is that the agent is handed the goal position relative to its current pose at every step—perfect localization of both robot and target—even though the benchmark is described as visual-only navigation.
What would settle it
A concrete check would be to re-run the reported evaluations while replacing the exact relative goal vector with one corrupted by realistic odometry noise or estimated by a vision-based goal detector; if first-goal success and CPK degrade sharply on the same held-out scenes, the benchmark's scores are being carried by the goal oracle rather than by visual navigation.
If this is right
- If RVN-Bench is taken as a standard testbed, navigation research can compare methods on safety—collisions per kilometer—rather than only on reaching goals, closing a gap the paper identifies in existing indoor benchmarks.
- Simulation-collected data appears to substitute for expensive real-world collision data: a model trained only on RVN-Bench-generated data outperformed a model trained on 54 hours of real navigation data in real-world first-goal success, and combining datasets helped further.
- Negative trajectories, which end in collisions and are hard or unsafe to collect on real robots, can be generated cheaply in simulation and improve imitation-learning agents when used as an auxiliary dataset.
- Predicted monocular depth markedly improves collision avoidance, and ground-truth depth improves it further, so continued progress in depth estimation should directly translate into safer reactive navigation.
- The benchmark leaves the collision-aware task open: even the strongest baseline reaches only about 93% first-goal success and 3.6 CPK on test scenes, so there is measurable headroom for future policies.
Where Pith is reading between the lines
- The paper leaves implicit that the relative goal vector is a privileged signal; if a real robot has to infer the goal from vision or odometry, first-goal success and collision rates would likely change, so the benchmark's 'reactive visual navigation' claim should be read as 'vision-based obstacle avoidance with a goal oracle.'
- Because collision detection is a geometric navigation-mesh constraint rather than physics-based contact, collisions with lightweight, pushable, or deformable objects are not represented; the benchmark's numbers therefore describe static-scene geometry, and real-world clutter may differ.
- The benchmark's configurable robot footprint, camera parameters, and action discretization make it straightforward to extend beyond the single platform and static scenes that the paper names as future work; the same structure could support dynamic obstacles or a goal-image variant that removes the oracle goal vector.
- The negative-trajectory mechanism is a transferable idea: deliberately generate failing trajectories, train an expert policy and a 'bad' policy, then use the contrast to reject unsafe actions at inference; the paper applies it only to navigation, but the scheme could be used in other imitation-learning settings where failures are expensive.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces RVN-Bench, a collision-aware benchmark for indoor reactive visual navigation built on Habitat 2.0 and the HM3D dataset. The benchmark defines a sequential PointGoal-style task in which an agent receives RGB observations and a relative goal vector, must reach up to 32 goal positions while avoiding collisions, and is evaluated with success rate, expected number of goals reached, and collisions per kilometer. The paper also provides an RL environment, a trajectory-image dataset generator including negative collision trajectories, and a new baseline NoMaD-Neg. Experiments compare IL, Safe-RL, and RL baselines, reporting that DDPPO-DA V2 performs best, and that a NoMaD-PointGoal model trained on simulated data transfers to a Jackal UGV in 20 real-world episodes.
Significance. If the claims hold, RVN-Bench is a useful contribution: it is a large-scale (800 train / 50 val / 50 test scenes) collision-aware indoor visual navigation benchmark with standardized metrics, RL training, and offline dataset generation, plus a public code release. The paper also provides a concrete negative-data mechanism and an initial sim-to-real study, which are valuable for the community. The benchmark is empirical and I did not find circular derivations of reported metrics from fitted parameters. However, the central 'only visual observations' framing is not supported by the task definition, and two reporting issues weaken the experimental support. These are fixable but must be addressed before the paper can be accepted.
major comments (4)
- [Abstract and Section III-A] The abstract and Section III-A state that the agent navigates 'using only visual observations', but the task gives the agent a privileged goal vector P_t = (x_g, y_g) relative to its current pose. This is typically provided by GPS+compass in Habitat PointGoal, not by vision. The manuscript never specifies how P_t is obtained in simulation or in the real-world Jackal experiments (Section IV-E), where NoMaD-PointGoal also requires this input. As written, RVN-Bench evaluates reactive obstacle avoidance under a perfect goal-localization oracle, not end-to-end visual navigation. Please either remove the 'only visual observations' claim, explicitly model P_t as an additional input modality, or specify the sensor/mechanism and discuss the implications for sim-to-real transfer.
- [Tables II and III] The same configuration is reported inconsistently. Table II lists 'DDPPO-DA V2' with Train E(G)=20.79, while Table III lists 'RGB + Predicted Depth' for DD-PPO with Train E(G)=22.70; other columns (validation/test E(G), CPK, SR1) are identical. One of these is a typo, but the discrepancy undermines the reproducibility of the main quantitative comparison. Correct the inconsistency and state which value is used in the main analysis.
- [Section IV-E and Table IV] The sim-to-real conclusion is based on 20 episodes (15 office, 5 house) with no error bars, confidence intervals, or per-environment breakdown. With n=20, the differences between Sim-only (SR1=0.60) and Real+Sim (SR1=0.75) are not shown to be statistically meaningful, and the CPK values (196–223) likely have large variance. Please report the raw per-episode results or confidence intervals, and soften the '3.5x/4.3x improvement' claims accordingly.
- [Section III-A and III-B] The task definition says the goal is 'randomly selected under certain rules' and the geodesic distance is in [d_min, d_max], but it does not specify how the goal position is sampled relative to obstacles or whether the same goal-sampling procedure is used in training and evaluation. Since the benchmark's purpose is standardized comparison, the exact sampling protocol (e.g., rejection sampling, minimum clearance, or uniform over navigable positions) should be stated precisely.
minor comments (5)
- [Section IV-E] The real-world text uses N_g = 5 while the benchmark uses N_goal = 32; use consistent notation.
- [Section III-C] The statement that NoMaD-Neg 'replaces 6.25% of expert trajectories with negative ones' is vague; specify the absolute counts for reproducibility.
- [Equation (1)] The CoR formula is cited from [27] but the displayed equation is missing an explicit division operator in the denominator; also define the value of alpha used in the experiments.
- [Figure 2] The figure caption and labels are difficult to parse; ensure all symbols (t_i, t_f, k_pre, k_post) are defined in the caption or the surrounding text.
- [Section IV-B] The RL baselines are said to use 'the same hyperparameters as reported in [16]', but for a benchmark it would be helpful to list the key hyperparameters (learning rate, batch size, number of timesteps, seeds) in the appendix or a table.
Circularity Check
Empirical benchmark paper; no predicted quantity reduces to a fitted input or to a load-bearing self-citation.
full rationale
RVN-Bench is a benchmark contribution, not a derivation. Its headline claims - that it provides a collision-aware visual navigation task and that policies trained on it generalize - are supported by held-out test-scene results (Table II) and a real-world Jackal evaluation (Table IV), i.e., measurements on data not used for fitting. There is no equation in which a predicted output equals an input by construction. The single self-citation is the CoR selection formula from the authors' MixGAIL paper [27], used in the secondary NoMaD-Neg baseline (Eq. 1-2); it is a published external formula with its own stated form, is not fitted to RVN-Bench data, and NoMaD-Neg is not the load-bearing evidence for the benchmark's validity, so this does not rise to circularity. The unspecified alpha in Eq. (1) and the privileged relative-goal vector P_t in Sec. III-A are reproducibility/scope issues for the 'visual-only' claim, not circular reductions: the task definition states what input the agent receives, and evaluation measures behavior under that input. The paper's own limitation statement in Sec. V (static environments, single platform) is a scoping note, not a concealed circular step. No quoted reduction can be exhibited, so the honest finding is no significant circularity.
Axiom & Free-Parameter Ledger
free parameters (5)
- Reward weights r_goal, r_collision, step penalty =
1.0, -0.1, -0.01
- Negative-trajectory window k_pre, k_post =
8, 6
- NoMaD-Neg sampling count k =
8
- CoR exponent alpha (α) =
unspecified
- Negative-data padding margin =
unspecified ('smaller than agent radius')
axioms (4)
- domain assumption NavMesh-based collision detection with hard kinematic constraint accurately captures real collisions with static obstacles.
- domain assumption The agent has access to the goal position relative to its current pose at every step (ground-truth localization).
- domain assumption HM3D scenes and Habitat 2.0 rendering are sufficiently realistic that policies trained in RVN-Bench transfer to real indoor environments.
- standard math A* on ground-truth occupancy with padded margin produces expert trajectories that are safe to follow with discrete actions.
read the original abstract
Safe visual navigation is critical for indoor mobile robots operating in cluttered environments. Existing benchmarks, however, often neglect collisions or are designed for outdoor scenarios, making them unsuitable for indoor visual navigation. To address this limitation, we introduce the reactive visual navigation benchmark (RVN-Bench), a collision-aware benchmark for indoor mobile robots. In RVN-Bench, an agent must reach sequential goal positions in previously unseen environments using only visual observations and no prior map, while avoiding collisions. Built on the Habitat 2.0 simulator and leveraging high-fidelity HM3D scenes, RVN-Bench provides large-scale, diverse indoor environments, defines a collision-aware navigation task and evaluation metrics, and offers tools for standardized training and benchmarking. RVN-Bench supports both online and offline learning by offering an environment for online reinforcement learning, a trajectory image dataset generator, and tools for producing negative trajectory image datasets that capture collision events. Evaluations demonstrate that policies trained on RVN-Bench generalize effectively across unseen simulated environments. Furthermore, initial physical experiments using a Jackal UGV indicate promising sim-to-real transfer. Code and additional materials are available at: https://sequor-robotics-research.github.io/projects/RVN-Bench/.
Figures
Reference graph
Works this paper leans on
-
[1]
GNM: A general navigation model to drive any robot,
D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine, “GNM: A general navigation model to drive any robot,” inProc. of the IEEE International Conference on Robotics and Automation (ICRA), 2023
2023
-
[2]
ViNT: A foundation model for visual navigation,
D. Shah, A. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine, “ViNT: A foundation model for visual navigation,” in Proc. of the Conference on robot learning (CoRL), 2023
2023
-
[3]
NoMaD: Goal masked diffusion policies for navigation and exploration,
A. Sridhar, D. Shah, C. Glossop, and S. Levine, “NoMaD: Goal masked diffusion policies for navigation and exploration,” inProc. of the IEEE International Conference on Robotics and Automation (ICRA), 2024
2024
-
[4]
Citywalker: Learning embodied urban navi- gation from web-scale videos,
X. Liu, J. Li, Y . Jiang, N. Sujay, Z. Yang, J. Zhang, J. Abanes, J. Zhang, and C. Feng, “Citywalker: Learning embodied urban navi- gation from web-scale videos,” inProc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025
2025
-
[5]
CARLA: An open urban driving simulator,
A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V . Koltun, “CARLA: An open urban driving simulator,” inProc. of the Con- ference on robot learning (CoRL), 2017
2017
-
[6]
SUMMIT: A simulator for urban driving in massive mixed traffic,
P. Cai, Y . Lee, Y . Luo, and D. Hsu, “SUMMIT: A simulator for urban driving in massive mixed traffic,” inProc. of the IEEE International Conference on Robotics and Automation (ICRA), 2020
2020
-
[7]
Habitat: A platform for embodied ai research,
M. Savva, A. Kadian, O. Maksymets, Y . Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V . Koltun, J. Maliket al., “Habitat: A platform for embodied ai research,” inProc. of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019
2019
-
[8]
Habitat 2.0: Training home assistants to rearrange their habitat,
A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y . Zhao, J. Turner, N. Maestre, M. Mukadam, D. S. Chaplot, O. Maksymetset al., “Habitat 2.0: Training home assistants to rearrange their habitat,” Advances in neural information processing systems (NeurIPS), 2021
2021
-
[9]
Design and use paradigms for gazebo, an open-source multi-robot simulator,
N. Koenig and A. Howard, “Design and use paradigms for gazebo, an open-source multi-robot simulator,” inProc. of the IEEE International Conference on Intelligent Robot Systems (IROS), 2004
2004
-
[10]
Abhishek Kadian*, Joanne Truong*, A. Gokaslan, A. Clegg, E. Wij- mans, S. Lee, M. Savva, S. Chernova, and D. Batra, “Are we making real progress in simulated environments? measuring the sim2real gap in embodied visual navigation,” inarXiv:1912.06321, 2019
Pith/arXiv arXiv 1912
-
[11]
HabiCrowd: A high performance simulator for crowd- aware visual navigation,
A. Vuong, T. Nguyen, M. N. Vu, B. Huang, H. Binh, T. V o, and A. Nguyen, “HabiCrowd: A high performance simulator for crowd- aware visual navigation,” inProc. of the IEEE International Confer- ence on Intelligent Robot Systems (IROS), 2024
2024
-
[12]
GOAT- bench: A benchmark for multi-modal lifelong navigation,
M. Khanna, R. Ramrakhya, G. Chhablani, S. Yenamandra, T. Gervet, M. Chang, Z. Kira, D. S. Chaplot, D. Batra, and R. Mottaghi, “GOAT- bench: A benchmark for multi-modal lifelong navigation,” inProc. of the IEEE/CVF International Conference on Computer Vision (ICCV), 2024
2024
-
[13]
Vision-and-language nav- igation: Interpreting visually-grounded navigation instructions in real environments,
P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. S ¨underhauf, I. Reid, S. Gould, and A. Van Den Hengel, “Vision-and-language nav- igation: Interpreting visually-grounded navigation instructions in real environments,” inProc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018
2018
-
[14]
Metaurban: A simulation platform for embodied ai in urban spaces,
W. Wu, H. He, Y . Wang, C. Duan, J. He, Z. Liu, Q. Li, and B. Zhou, “Metaurban: A simulation platform for embodied ai in urban spaces,” International Conference on Learning Representation (ICLR), 2025
2025
-
[15]
Habitat-matterport 3D dataset (hm3d): 1000 large-scale 3D environments for embodied ai,
S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg, J. Turner, E. Undersander, W. Galuba, A. Westbury, A. Chang, M. Savva, Y . Zhao, and D. Batra, “Habitat-matterport 3D dataset (hm3d): 1000 large-scale 3D environments for embodied ai,” in Proc. of the Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2021
2021
-
[16]
DD-PPO: Learning near-perfect pointgoal navigators from 2.5 billion frames,
E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra, “DD-PPO: Learning near-perfect pointgoal navigators from 2.5 billion frames,” inInternational Conference on Learning Representations (ICLR), 2020
2020
-
[17]
Proximal policy optimization algorithms,
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017
Pith/arXiv arXiv 2017
-
[18]
Benchmarking safe exploration in deep reinforcement learning,
A. Ray, J. Achiam, and D. Amodei, “Benchmarking safe exploration in deep reinforcement learning,”arXiv preprint arXiv:1910.01708, 2019
Pith/arXiv arXiv 1910
-
[19]
Habitat challenge 2023,
K. Yadav, J. Krantz, R. Ramrakhya, S. K. Ramakrishnan, J. Yang, A. Wang, J. Turner, A. Gokaslan, V .-P. Berges, R. Mootaghi, O. Maksymets, A. X. Chang, M. Savva, A. Clegg, D. S. Chaplot, and D. Batra, “Habitat challenge 2023,” https://aihabitat.org/challenge/ 2023/, 2023
2023
-
[20]
HM3D- OVON: A dataset and benchmark for open-vocabulary object goal navigation,
N. Yokoyama, R. Ramrakhya, A. Das, D. Batra, and S. Ha, “HM3D- OVON: A dataset and benchmark for open-vocabulary object goal navigation,” inProc. of the IEEE International Conference on Intelli- gent Robot Systems (IROS), 2024
2024
-
[21]
Autonomous ground navigation in highly constrained spaces: Lessons learned from the benchmark autonomous robot navigation challenge at icra 2022 competitions,
X. Xiao, Z. Xu, Z. Wang, Y . Song, G. Warnell, P. Stone, T. Zhang, S. Ravi, G. Wang, H. Karnanet al., “Autonomous ground navigation in highly constrained spaces: Lessons learned from the benchmark autonomous robot navigation challenge at icra 2022 competitions,” IEEE Robotics and Automation Magazine, 2022
2022
-
[22]
Vision and navigation for the carnegie-mellon navlab,
C. Thorpe, M. H. Hebert, T. Kanade, and S. A. Shafer, “Vision and navigation for the carnegie-mellon navlab,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 1988
1988
-
[23]
Learning by cheating,
D. Chen, B. Zhou, V . Koltun, and P. Kr ¨ahenb¨uhl, “Learning by cheating,” inProc. of the Conference on robot learning (CoRL), 2020
2020
-
[24]
Autonomous quadrotor navigation with vision based obstacle avoidance and path planning,
H.-Y . Lin and X.-Z. Peng, “Autonomous quadrotor navigation with vision based obstacle avoidance and path planning,”IEEE Access, 2021
2021
-
[25]
A formal basis for the heuristic determination of minimum cost paths,
P. E. Hart, N. J. Nilsson, and B. Raphael, “A formal basis for the heuristic determination of minimum cost paths,”IEEE transactions on Systems Science and Cybernetics, 1968
1968
-
[26]
Depth anything v2,
L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao, “Depth anything v2,”Advances in neural information processing systems (NeurIPS), 2024
2024
-
[27]
Mixgail: Autonomous driving using demonstrations with mixed qualities,
G. Lee, D. Kim, W. Oh, K. Lee, and S. Oh, “Mixgail: Autonomous driving using demonstrations with mixed qualities,” inProc. of the IEEE International Conference on Intelligent Robot Systems (IROS), 2020
2020
-
[28]
Deep visual mpc-policy learning for navigation,
N. Hirose, F. Xia, R. Mart ´ın-Mart´ın, A. Sadeghian, and S. Savarese, “Deep visual mpc-policy learning for navigation,”IEEE Robotics and Automation Letters (RA-L), 2019
2019
-
[29]
Rapid explo- ration for open-world navigation with latent goal models,
D. Shah, B. Eysenbach, N. Rhinehart, and S. Levine, “Rapid explo- ration for open-world navigation with latent goal models,” inProc. of the Conference on robot learning (CoRL), 2021
2021
-
[30]
Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation,
H. Karnan, A. Nair, X. Xiao, G. Warnell, S. Pirk, A. Toshev, J. Hart, J. Biswas, and P. Stone, “Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation,”IEEE Robotics and Automation Letters (RA-L), 2022
2022
-
[31]
Sacson: Scalable autonomous control for social navigation,
N. Hirose, D. Shah, A. Sridhar, and S. Levine, “Sacson: Scalable autonomous control for social navigation,”IEEE Robotics and Au- tomation Letters (RA-L), 2023
2023
-
[32]
Jackal UGV - Small Weatherproof Robot,
Clearpath Robotics, “Jackal UGV - Small Weatherproof Robot,” https: //clearpathrobotics.com/jackal-small-unmanned-ground-vehicle/, Dec. 2020, accessed: 2025-09-01
2020
-
[33]
ZED-X Stereo Camera – Rugged AI Stereo Camera for Robotics,
Stereolabs, “ZED-X Stereo Camera – Rugged AI Stereo Camera for Robotics,” https://www.stereolabs.com/products/zed-x, Dec. 2022, accessed: 2025-09-01
2022
-
[34]
NVIDIA Jetson Orin – AI Computing Modules and Devel- oper Kits for Robotics and Edge AI,
NVIDIA, “NVIDIA Jetson Orin – AI Computing Modules and Devel- oper Kits for Robotics and Edge AI,” https://www.nvidia.com/en-us/ autonomous-machines/embedded-systems/jetson-orin/, Sep. 2025, ac- cessed: 2025-09-01
2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.