Pith. sign in

REVIEW 4 major objections 5 minor 29 references

Robostral Navigate is an 8B vision-language model that predicts the next waypoint by pointing in the camera image, and the paper reports this single-RGB-camera system beats depth- and multi-camera methods on the R2R-CE and RxR-CE benchmarks

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 09:22 UTC pith:RA66GUP2

load-bearing objection Strong new training recipe and plausible SOTA numbers, but the benchmarked system uses a Habitat pathfinder for execution, so the 'no maps' claim is softer than the abstract suggests. the 4 major comments →

arxiv 2607.20785 v2 pith:RA66GUP2 submitted 2026-07-22 cs.RO cs.AI

Robostral Navigate

Abhijeet Somani , Aditi Kabra , Adrian Valente , Alan Jeffares , Albert Jiang , Aleksandr Timashov , Alexandre Sablayrolles , Amelie Heliou
show 207 more authors
Andre Jonasson Andrew Bai Andrew Ehrenberg Andrew Zhao Angele Lenglemetz Anmol Agarwal Antonia Calvi Arata Suzuki Aylin Guliz Akkus Aysenur Karaduman Baptiste Bout Baptiste Roziere Baudouin De Monicault Benjamin Holzschuh Benjamin Lefaudeux Bernhard Stadlbauer Blazej Osinski Camille Le Scao Chaoran Yu Chen-Yo Sun Christian Wallenwein Christophe Renaudin Clemence Lanfranchi Corentin Barreau Corentin Sautier Cristiana-Diana Diaconu Cyprien Courtot Daniel Marczak Darius Dabert Diego de Las Casas Dominik Nuss Dylan Rubini Dzmitry Soupel Emilien Fugier Erik Aas Etienne Millon Eujeong Choi Fabian Paischer Fabian Schlager Faruk Ahmed Federico Baldassarre Filip Szatkowski Gabrielle Berrada Gaetan Ecrepont Gaetan Lepage Gaspard Blanchet Gaspard Donada-Vidal Gauthier Delerce Gauthier Guinet Genevieve Hayes Georgii Novikov Giada Pistilli Gianluca Galletti Guillaume Breton Guillaume Martin Gunjan Dhanuka Gunshi Gupta Han Zhou Hasan Furkan Vural Indraneel Mukherjee Ivan Cuevas Salazar Jan Ludziejewski Jason Rute Jean-Hadrien Chabran Jean-Malo Delignon Jie Zhang Joachim Studnia Joep Barmentlo Johannes Brandstetter John Harvill Jonas Amar Jonas Schweizer Josselin Somerville Julien Denize Julien Tauran Kartik Khandelwal Kush Jain Larissa Laich Laura Calem Laurence Aitchison Laurent Callot Leo Cotteleer Leonard Blier Lingxiao Zhao Louis Martin Louis Serrano Lucile Saulnier Luis Montero Maarten Buyl Marcin Mozejko Margaret Jennings Mathieu Schmitt Mathilde Guillaumin Matthieu Dinot Matthieu Futeral Mauro Comi Max Mynter Maxim Berman Maxime Darrin Maxime Louis Maximilian Augustin Maximilian Muller Mert Unsal Mia Chiquier Michael Pilcer Michal Pietruszka Michal Zajac Mikhail Biriuchinskii Minwoo Kang Morgane Riviere Namit Katariya Nathan Grinsztajn Neeraj Aggarwal Neha Gupta Ola Mysiak Oliver Leicht Olivier Bousquet Parag Jain Patricia Wang Patrick von Platen Paul Jacob Paul Wambergue Paula Kurylowicz Pavan Kumar Reddy Philomene Chagniot Pierre Stock Pierre-Andre Savalle Piotr Milos Prateek Gupta Pravesh Agrawal Quentin Desreumaux Quentin Torroba Quercus Hernandez Ram Ramrakhya Randall Isenhour Ranjit Parva Raul Perez Pelaez Remi Delacourt Rishi Shah Rohin Arora Romain Sauvestre Roman Soletskyi Sagar Vaze Samuel Humeau Sanchit Gandhi Sandeep Subramanian Sarthak Mittal Saskia Adaime Sebastian Kaltenbach Shashwat Dalal Sherif Waly Shrimai Prabhumoye Siddharth Gandhi Simon Sorg Soham Ghosh Sophie Marbach Stanislas Lange Sumukh Aithal Szymon Antoniak Teven Le Scao Thibaut Lavril Thomas Coste Thomas Foubert Thomas Robert Thomas Wang Tianyu Zhang Tim Lawson Timothee Lacroix Tobias Kronlachner Tom Bewley Tomas Hodan Tuhin Das Tyler Wang Van Phung Vedant Nanda Victor Jouault Victor Letzelter Victor Paltz Victor Poucheret Vincent Pfister Virgile Richard Vladislav Bataev Wassim Bouaziz Wen Ding Li William Marshall Xinghui Li Xingran Guo Xinyu Yang Yann Dreze Yihan Wang Zaccharie Ramzi Zhenlin Xu Zsofia Csakany Arjun Majumdar Avinash Sooriyarachchi Benjamin Tibi Chris Bamford Elliot Chane-Sane Guillaume Lample Khyathi Raghavi Chandu Ludovic Ho Fuh Mathieu Poiree Olivier Duchenne Rosalie Millner Srijan Mishra Theo Cachet Thomas Chabal
This is my paper
classification cs.RO cs.AI
keywords embodied navigationvision-language modelmonocular RGBwaypoint pointingprefix-tree attentiondiffusion policyreinforcement learningR2R-CE
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Robostral Navigate is an 8-billion-parameter vision-language model that tries to solve instruction-following navigation from a single moving RGB camera, without depth sensors, multi-camera rigs, or pre-built maps. Its core move is to output the next waypoint by pointing at pixel coordinates in the current view, with a metric-displacement fallback when the target is out of frame; a small diffusion policy turns that waypoint into a dense low-level trajectory. The paper reports that this recipe sets a new reported high on the standard R2R-CE and RxR-CE benchmarks—77.4% and 75.1% success on validation unseen—beating the best prior single-camera method and even depth- or multi-camera systems. It also introduces a prefix-tree attention-masking training scheme that packs whole episodes into one sequence, cutting training tokens 22x, and an online RL stage that adds roughly four points of success. The sympathetic reading is that a minimal-sensor, simulation-trained pipeline can outperform privileged-sensor systems.

Core claim

The discovery the paper tries to establish is that navigation can be decomposed into a coarse pointer and a fine mover: an 8B VLM that predicts image-space waypoints and heading deltas at 0.5 Hz, and a 121M diffusion policy that converts those waypoints plus the context and current frames into dense 10 Hz action chunks. Because the high-level action is expressed in camera coordinates rather than robot coordinates, the same weights transfer across robots with different heights, radii, camera placements, and pitches, with only the low-level motion controller swapped. On R2R-CE validation unseen, the system reports 77.4% success and 74.2% SPL, and on RxR-CE it reports 75.1% success and 68.7% SP

What carries the argument

The central object is the pointing waypoint: for each step, the VLM predicts pixel coordinates (u, v) plus a yaw change when the next trajectory waypoint is visible, and falls back to metric displacements (Δx, Δy, Δθ) when it is not, along with a STOP token. Because pointing lives in pixel space, it is invariant to camera intrinsics and scene scale. The second mechanism is the prefix-tree attention mask: a full episode is encoded as one sequence (instruction followed by observations interleaved with actions), and the mask lets each action attend only to the instruction and observations in its own history branch, never to ground-truth actions from other branches. This preserves the per-time-s

Load-bearing premise

The reported numbers come from an evaluation in which the simulator's pathfinder turns the model's sparse waypoints into actual motion; if that pathfinder is already doing map-based planning and obstacle avoidance, the 'single RGB camera, no pre-built maps' framing credits the learned model with work a privileged module is doing.

What would settle it

Take the trained VLM and diffusion policy, but feed the simulator's pathfinder random waypoints sampled in the same scene (or constant forward waypoints). If success rate stays high, the pathfinder alone is carrying the navigation. Alternatively, replace the pathfinder with a no-map waypoint tracker that uses only the RGB stream and measure how much of the 77.4% R2R-CE success rate survives.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single RGB camera can be enough to reach top navigation scores: the same policy beats depth- and multi-camera systems on R2R-CE and RxR-CE while using only monocular image input.
  • Because waypoints are in image space, the trained policy can be deployed on wheeled, legged, and aerial robots without recalibration; only the low-level motion controller changes between embodiments.
  • Packing entire episodes into single training sequences with prefix-tree attention masks cuts training tokens by 22x and turns month-long runs into days without discarding action-prediction targets.
  • Online RL on a curated hard subset substantially improves over supervised fine-tuning, adding roughly four percentage points of success rate on both benchmarks by teaching exploration and recovery.
  • Training entirely in simulation on 2.4 million trajectories across 350k scenes removes the need for real-world data collection while still generalizing to unseen photorealistic environments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the simulator's pathfinder is doing load-bearing map-based planning and obstacle avoidance, the 'single RGB camera, no pre-built maps' framing applies to the learned high-level planner rather than the full deployed system; a fair comparison would disclose how much of the motion a privileged navigation mesh provides.
  • The pointing formulation suggests an economic route to fleets: one shared visual-pointer network could serve many robot bodies, with embodiment-specific controllers selected at deployment time—an implication the paper gestures at but does not quantify.
  • The prefix-tree attention-masking trick is a general training-efficiency idea for any long-horizon policy-learning task with redundant episode histories, and could be tested on manipulation or driving benchmarks beyond navigation.
  • A testable extension is to train Robostral Navigate without the metric displacement fallback, relying purely on pointing plus learned turning, to see whether the co-training branch is necessary or whether pure image-space actions suffice.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. Robostral Navigate proposes an 8B vision-language model for vision-and-language navigation in continuous environments that consumes only monocular RGB images and predicts waypoints via pointing (image coordinates plus a metric displacement fallback). A 121M diffusion policy is designed to convert waypoints into low-level trajectory chunks, and a motion-tracking controller to execute them. Training uses 2.4 million simulated trajectories across 350k scenes, a prefix-tree attention scheme that packs episodes into single sequences (claimed 22x token reduction), and CISPO reinforcement learning. The paper reports state-of-the-art results on R2R-CE validation unseen (77.4% SR, 74.2% SPL) and RxR-CE validation unseen (75.1% SR, 68.7% SPL), surpassing depth/multi-camera baselines with only a single RGB camera.

Significance. If the reported numbers characterized the full learned monocular system, this would be a significant advance for minimal-sensor, map-free VLN. The waypoint-in-image-space formulation, the large-scale synthetic data pipeline, and the 22x training-token reduction via prefix-tree attention are valuable contributions. However, the benchmark evaluation in Section 4 bypasses the learned diffusion policy and motion controller, delegating inter-waypoint navigation to Habitat's pathfinder, which consumes the scene navmesh. The headline results therefore validate waypoint-prediction accuracy under a privileged path executor, not the end-to-end RGB-only learned system claimed. With an end-to-end evaluation or a clearly qualified framing, the contribution can still be strong, but the current claims outrun the evidence.

major comments (4)
  1. [Section 4, first paragraph; Figure 1] The central SOTA claim is undermined by the evaluation protocol: 'For these evaluations, we use a pathfinder from Habitat ... for navigation between the waypoints predicted by Robostral Navigate.' This means the benchmarked system is the VLM waypoint predictor plus a privileged navmesh-based path executor, not the full monocular RGB stack (VLM + 121M diffusion policy + motion controller) described in Section 2. Habitat pathfinders typically use global agent pose and the scene navigation mesh to compute geodesic paths, providing geometry that the paper claims the learned system does not need. The 77.4% SR / 74.2% SPL numbers should be reported as waypoint-prediction performance under a privileged pathfinder, not as end-to-end minimal-sensor system performance; otherwise the central claim is overstated.
  2. [Section 4, Table 1] The comparison to prior methods is potentially apples-to-oranges. It is not stated whether the baselines (NaVid, Uni-NaVid, Qwen-RobotNav, etc.) also use a Habitat pathfinder to execute predicted waypoints, or whether they execute their own learned low-level actions. If baselines use end-to-end learned control while Robostral Navigate's benchmark numbers include a pathfinder, the comparison conflates waypoint-prediction quality with low-level execution ability. The authors should clarify the execution protocol for every compared method or report results for Robostral Navigate with its own diffusion policy + controller in the same setting.
  3. [Section 3.3, reference [Chen et al., 2025]] The RL method CISPO is cited as [Chen et al., 2025], but the cited reference is 'MiniMax-M1: Scaling test-time compute efficiently with lightning attention', which does not appear to describe CISPO or constrained/group-relative policy optimization. This makes the RL component unreproducible. Please provide a correct citation or a self-contained description of the algorithm and the group-relative advantage estimation used.
  4. [Section 3.2, 'Provably identical' claim] The paper states that the prefix-tree representation 'gives provably identical training signal' to per-time-step samples, but no proof or detailed argument is provided. This is a load-bearing claim for the training-efficiency contribution (22x token reduction). Please include a formal statement or a clear argument that the attention mask prevents conditioning on previous ground-truth actions while preserving per-step likelihoods, ideally with a small worked example.
minor comments (5)
  1. [Throughout] Several typos: Section 2.3 uses 'Robotstral Navigate' instead of 'Robostral Navigate'; Section 2.1 has 'embodiement-specific'. Please proofread.
  2. [Section 3.2] The 22x token reduction and the claim that scene-contiguous ordering outperforms random shuffling are presented without ablations or error bars. Adding a small ablation table would strengthen the training-recipe claims.
  3. [Section 2.4, Figure 2] The cross-robot generalization results are only qualitative demonstrations. No quantitative metrics (success rate, path efficiency) are reported for the Galaxea R1 or Hiwonder JetAuto deployments, so the claim of cross-embodiment transfer without retraining is not empirically supported by the benchmark data.
  4. [Table 1] The table includes Qwen-RobotNav-4B and Qwen-RobotNav-8B in both the 'Single RGB camera' and 'Depth / multi-camera' sections. Please clarify the exact sensor configurations for these variants, as the current presentation is confusing when the same model name appears twice.
  5. [Section 4] No code, model weights, or evaluation scripts are released, and no confidence intervals or multi-seed variances are reported. For a SOTA claim on a standard benchmark, releasing the eval harness and at least stating variance across seeds would substantially improve reproducibility.

Circularity Check

0 steps flagged

No circularity: results are external benchmark numbers; pathfinder concern is an evaluation-validity issue, not a derivation-level equivalence.

full rationale

The paper's central claims are external benchmark results on R2R-CE and RxR-CE, and no parameter is fitted to the reported success-rate or SPL targets. Supervised training uses a pointing/displacement prediction loss and online RL uses a geodesic-distance reward via CISPO; these are standard surrogate objectives, not the benchmark metrics themselves, and none of the reported numbers are constructed from those objectives by definition. The use of the Habitat pathfinder between predicted waypoints (Section 4) is a serious evaluation-validity concern about whether the benchmark measures the full advertised system, but it is not a circular step in the derivation sense: the VLM's waypoint outputs are still independently scored against the benchmark. The citation of Molmo2 for attention masking is presented as inspiration rather than as load-bearing evidence for the SOTA claim, and the other citations are external benchmarks, datasets, or methods. No self-citation chain, no renamed input-as-prediction, and no uniqueness argument imported from the authors are present. Therefore no significant circularity is identified.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The central claims rest on unverified experimental choices rather than a mathematical derivation. The main load-bearing assumptions are the sufficiency of monocular RGB, the availability of a privileged Habitat pathfinder at evaluation, sim-to-real transfer from undisclosed 350k scenes, and the correctness of the prefix-tree training-signal equivalence. No new physical entities are introduced.

free parameters (5)
  • Robot morphology randomization ranges = height 0.4-1.8 m; radius 0.15-0.45 m; camera height 70-100% of robot height; pitch 0-25°
    Chosen by hand in Sec. 2.4; the cross-robot generalization claim rests on these ranges covering deployment shift.
  • RL reward distance clip = 2 m
    Reward -max(2, dist_to_goal) clips the penalty to 2 m; threshold chosen by hand (Sec. 3.3) and shapes STOP behavior.
  • Hard RL subset size = 35k tasks
    Curated from SFT failures by hand-chosen count (Sec. 3.3); drives the reported +4 point RL gain.
  • Action chunk length / frequencies = 30 delta coords at 10 Hz; 0.5 Hz VLM; 100 Hz controller
    Architecture hyperparameters chosen in Sec. 2.1/2.3; performance depends on them but no sensitivity study.
  • Model sizes = 8B VLM + 121M diffusion policy
    Capacity choices; no ablation of model size is reported.
axioms (6)
  • domain assumption Monocular RGB frames contain enough information to infer the farthest visible waypoint and orientation change in unseen environments.
    Central design premise of pointing mode (Sec. 2.2); if false, the SOTA result would not transfer.
  • domain assumption The Habitat pathfinder used at evaluation is not contributing privileged map information in a way that invalidates the 'single RGB camera / no pre-built maps' claim.
    Required to reconcile Section 4's pathfinder use with the abstract's 'no pre-built maps' positioning.
  • domain assumption The 2.4M simulated trajectories are sufficiently photorealistic and diverse that policies trained only in simulation transfer to R2R-CE/RxR-CE unseen scenes.
    Data generation description (Sec. 3.1) lacks dataset details; sim-to-real transfer is assumed.
  • standard math Flattened prefix-tree representation with the stated attention mask provides training signal identical to per-time-step samples.
    Claimed 'provably identical' in Sec. 3.2 but no proof or formal check is given.
  • domain assumption Randomizing robot/camera parameters during data generation spans deployment variations of real platforms.
    Cross-robot deployment claim (Sec. 2.4) relies on this coverage; no quantitative real-robot results are reported.
  • domain assumption CISPO's group-relative advantage estimation provides a stable RL signal for this VLM policy.
    Cited as [Chen et al., 2025] (Sec. 3.3), but the cited reference is MiniMax-M1, not a CISPO paper.

pith-pipeline@v1.3.0-alltime-deepseek · 11218 in / 13605 out tokens · 126128 ms · 2026-08-01T09:22:41.152008+00:00 · methodology

0 comments
read the original abstract

Deploying navigation systems at scale requires a recipe that minimizes sensor assumptions, generalizes across robot embodiments, and trains efficiently. Yet, today's best systems depend on depth sensors, multi-camera rigs, or pre-built maps, limiting the hardware they support and increasing deployment cost. We introduce Robostral Navigate, an 8B vision-language model built around this scalability objective. The model consumes only a stream of monocular RGB images - the most ubiquitous sensor across robotic platforms and predicts waypoints by pointing to the next target location in the current camera view. Operating purely in image space, rather than robot-specific coordinates, makes the policy naturally robust to changes in camera intrinsics and scene scale, enabling deployment across wheeled, legged, and aerial robots without recalibration. We generate 2.4 million trajectories across 350k simulated scenes to reduce the reliance on real-world data collection and scale easily. We further introduce a prefix-caching training recipe that packs entire episodes into single training sequences, reducing training tokens by 22x and cutting training time from months to days. A tree-based attention mask prevents conditioning on previous ground-truth actions, encouraging visually grounded action prediction, and reinforcement learning is used to further improve exploration and recovery capabilities. On the Room-to-Room and Room-Across-Room in Continuous Environments (R2R-CE and RxR-CE) benchmarks, Robostral Navigate sets a new state of the art. On R2R-CE, it achieves a 77.4% success rate, surpassing the best monocular method by 10.5 points and the strongest depth- or multi-camera system by 5.3 points despite using only a single RGB camera. On RxR-CE, it reaches 75.1% success rate, outperforming all monocular baselines.

Figures

Figures reproduced from arXiv: 2607.20785 by Abhijeet Somani, Aditi Kabra, Adrian Valente, Alan Jeffares, Albert Jiang, Aleksandr Timashov, Alexandre Sablayrolles, Amelie Heliou, Andre Jonasson, Andrew Bai, Andrew Ehrenberg, Andrew Zhao, Angele Lenglemetz, Anmol Agarwal, Antonia Calvi, Arata Suzuki, Arjun Majumdar, Avinash Sooriyarachchi, Aylin Guliz Akkus, Aysenur Karaduman, Baptiste Bout, Baptiste Roziere, Baudouin De Monicault, Benjamin Holzschuh, Benjamin Lefaudeux, Benjamin Tibi, Bernhard Stadlbauer, Blazej Osinski, Camille Le Scao, Chaoran Yu, Chen-Yo Sun, Chris Bamford, Christian Wallenwein, Christophe Renaudin, Clemence Lanfranchi, Corentin Barreau, Corentin Sautier, Cristiana-Diana Diaconu, Cyprien Courtot, Daniel Marczak, Darius Dabert, Diego de las Casas, Dominik Nuss, Dylan Rubini, Dzmitry Soupel, Elliot Chane-Sane, Emilien Fugier, Erik Aas, Etienne Millon, Eujeong Choi, Fabian Paischer, Fabian Schlager, Faruk Ahmed, Federico Baldassarre, Filip Szatkowski, Gabrielle Berrada, Gaetan Ecrepont, Gaetan Lepage, Gaspard Blanchet, Gaspard Donada-Vidal, Gauthier Delerce, Gauthier Guinet, Genevieve Hayes, Georgii Novikov, Giada Pistilli, Gianluca Galletti, Guillaume Breton, Guillaume Lample, Guillaume Martin, Gunjan Dhanuka, Gunshi Gupta, Han Zhou, Hasan Furkan Vural, Indraneel Mukherjee, Ivan Cuevas Salazar, Jan Ludziejewski, Jason Rute, Jean-Hadrien Chabran, Jean-Malo Delignon, Jie Zhang, Joachim Studnia, Joep Barmentlo, Johannes Brandstetter, John Harvill, Jonas Amar, Jonas Schweizer, Josselin Somerville, Julien Denize, Julien Tauran, Kartik Khandelwal, Khyathi Raghavi Chandu, Kush Jain, Larissa Laich, Laura Calem, Laurence Aitchison, Laurent Callot, Leo Cotteleer, Leonard Blier, Lingxiao Zhao, Louis Martin, Louis Serrano, Lucile Saulnier, Ludovic Ho Fuh, Luis Montero, Maarten Buyl, Marcin Mozejko, Margaret Jennings, Mathieu Poiree, Mathieu Schmitt, Mathilde Guillaumin, Matthieu Dinot, Matthieu Futeral, Mauro Comi, Maxim Berman, Maxime Darrin, Maxime Louis, Maximilian Augustin, Maximilian Muller, Max Mynter, Mert Unsal, Mia Chiquier, Michael Pilcer, Michal Pietruszka, Michal Zajac, Mikhail Biriuchinskii, Minwoo Kang, Morgane Riviere, Namit Katariya, Nathan Grinsztajn, Neeraj Aggarwal, Neha Gupta, Ola Mysiak, Oliver Leicht, Olivier Bousquet, Olivier Duchenne, Parag Jain, Patricia Wang, Patrick von Platen, Paula Kurylowicz, Paul Jacob, Paul Wambergue, Pavan Kumar Reddy, Philomene Chagniot, Pierre-Andre Savalle, Pierre Stock, Piotr Milos, Prateek Gupta, Pravesh Agrawal, Quentin Desreumaux, Quentin Torroba, Quercus Hernandez, Ram Ramrakhya, Randall Isenhour, Ranjit Parva, Raul Perez Pelaez, Remi Delacourt, Rishi Shah, Rohin Arora, Romain Sauvestre, Roman Soletskyi, Rosalie Millner, Sagar Vaze, Samuel Humeau, Sanchit Gandhi, Sandeep Subramanian, Sarthak Mittal, Saskia Adaime, Sebastian Kaltenbach, Shashwat Dalal, Sherif Waly, Shrimai Prabhumoye, Siddharth Gandhi, Simon Sorg, Soham Ghosh, Sophie Marbach, Srijan Mishra, Stanislas Lange, Sumukh Aithal, Szymon Antoniak, Teven Le Scao, Theo Cachet, Thibaut Lavril, Thomas Chabal, Thomas Coste, Thomas Foubert, Thomas Robert, Thomas Wang, Tianyu Zhang, Tim Lawson, Timothee Lacroix, Tobias Kronlachner, Tomas Hodan, Tom Bewley, Tuhin Das, Tyler Wang, Van Phung, Vedant Nanda, Victor Jouault, Victor Letzelter, Victor Paltz, Victor Poucheret, Vincent Pfister, Virgile Richard, Vladislav Bataev, Wassim Bouaziz, Wen Ding Li, William Marshall, Xinghui Li, Xingran Guo, Xinyu Yang, Yann Dreze, Yihan Wang, Zaccharie Ramzi, Zhenlin Xu, Zsofia Csakany.

Figure 1
Figure 1. Figure 1: System Architecture. Given a natural language instruction and a stream of RGB observations, the VLM predicts the next waypoint at 0.5 hertz via pointing, metric displacement and orientation upon arrival. When the target lies outside the field of view, pointing is omitted. A diffusion policy takes the waypoint, the context frame (frame at which the waypoint has been inferred) and the current frame to produc… view at source ↗
Figure 2
Figure 2. Figure 2: Cross-robot deployment of Robostral Navigate on the Galaxea R1 (left) and Hiwonder JetAuto (right). The two platforms differ substantially in robot morphology, camera height, camera configuration, and base geometry. Both deployments use the same Robostral Navigate vision-language model and diffusion policy weights, with only the low-level platform controller differing between robots. 3 Training 3.1 Data Ge… view at source ↗
Figure 3
Figure 3. Figure 3: Example of training data. Left: 3D navigable area and sampled trajectories for a given scene. Right: Four frames from a cross-floor trajectory with future positions in-painted in green and waypoint in red. To enable rapid iteration and mitigate the reliance on costly real-world data collection, we built an efficient simulation-based data generation pipeline. The pipeline generates expert navigation traject… view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of (a) naive representation with our (b) prefix-caching. (c) demonstrates the tree attention mask with prefix-tree representation; I is the instruction, Oi is the i th observation/image, ai is the action at the i th time step. A key ingredient of Robostral Navigate is an efficient training algorithm based on prefix sharing, shown in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Overview of our online reinforcement learning pipeline. The trainer orchestrates the prompt sampling, physical simulator, and action generator (vLLM) in an asynchronous loop, updating the policy via CISPO using the −max(2, dist_to_goal) reward signal upon episode termination. 3.3 Online Reinforcement Learning via CISPO Although supervised fine-tuning (SFT) effectively instills a foundational visual-linguis… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references

  1. [1]

    2024 , url =

    Zhang, Jiazhao and Wang, Kunyu and Xu, Rongtao and Zhou, Gengze and Hong, Yicong and Fang, Xiaomeng and Wu, Qi and Zhang, Zhizheng and Wang, He , booktitle =. 2024 , url =

    2024

  2. [2]

    Zhang, Jiazhao and Wang, Kunyu and Wang, Shaoan and Li, Minghan and Liu, Haoran and Wei, Songlin and Wang, Zhongyuan and Zhang, Zhizheng and Wang, He , journal =

  3. [3]

    Robotics: Science and Systems , year =

    Cheng, An-Chieh and Ji, Yandong and Yang, Zhaojing and Gongye, Zaitian and Zou, Xueyan and Kautz, Jan and B. Robotics: Science and Systems , year =

  4. [4]

    Jiazhao Zhang and Anqi Li and Yunpeng Qi and Minghan Li and Jiahang Liu and Shaoan Wang and Haoran Liu and Gengze Zhou and Yuze Wu and Xingxing Li and Yuxin Fan and Wenjun Li and Zhibo Chen and Fei Gao and Qi Wu and Zhizheng Zhang and He Wang , journal =

  5. [5]

    Meng Wei and Chenyang Wan and Xiqian Yu and Tai Wang and Yuqiang Yang and Xiaohan Mao and Chenming Zhu and Wenzhe Cai and Hanqing Wang and Yilun Chen and Xihui Liu and Jiangmiao Pang , journal =

  6. [6]

    Qiuyue Wang and others , journal =

  7. [7]

    Zhang, Jiazhao and Zhou, Gengze and Yin, Hale and Huang, Yiyang and Lei, Zixing and Peng, Qihang and Yuan, Haoqi and Zhang, Jie and Guo, Xudong and Chen, Xiaoyue and others , journal =

  8. [8]

    arXiv preprint arXiv:2509.12129 , year =

    Embodied Navigation Foundation Model , author =. arXiv preprint arXiv:2509.12129 , year =

    arXiv

  9. [9]

    Chu, Zedong and Xie, Shichao and Wu, Xiaolong and Shen, Yanfen and Luo, Minghua and Wang, Zhengbo and Liu, Fei and Leng, Xiaoxu and Hu, Junjun and Yin, Mingyang and others , journal =

  10. [10]

    2026 , url =

    Xinda Xue and Junjun Hu and Minghua Luo and Shichao Xie and Jintao Chen and Zixun Xie and Kuichen Quan and Wei Guo and Mu Xu and Zedong Chu , booktitle =. 2026 , url =

    2026

  11. [11]

    Beyond the

    Jacob Krantz and Erik Wijmans and Arjun Majumdar and Dhruv Batra and Stefan Lee , booktitle =. Beyond the. 2020 , url =

    2020

  12. [12]

    Chen, Aili and Li, Aonian and Gong, Bangwei and Jiang, Binyang and Fei, Bo and Yang, Bo and Shan, Boji and Yu, Changqing and Wang, Chao and Zhu, Cheng and others , journal =

  13. [13]

    Advances in Neural Information Processing Systems , volume=

    Toward the fundamental limits of imitation learning , author=. Advances in Neural Information Processing Systems , volume=

  14. [14]

    Proceedings of the fourteenth international conference on artificial intelligence and statistics , year =

    A reduction of imitation learning and structured prediction to no-regret online learning , author =. Proceedings of the fourteenth international conference on artificial intelligence and statistics , year =

  15. [15]

    International Conference on Computer Vision , pages=

    Scalable diffusion models with transformers , author=. International Conference on Computer Vision , pages=

  16. [16]

    Conference on Computer Vision and Pattern Recognition , year =

    Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments , author =. Conference on Computer Vision and Pattern Recognition , year =

  17. [17]

    Habitat: A Platform for Embodied

    Manolis Savva and Abhishek Kadian and Oleksandr Maksymets and Yili Zhao and Erik Wijmans and Bhavana Jain and Julian Straub and Jia Liu and Vladlen Koltun and Jitendra Malik and Devi Parikh and Dhruv Batra , booktitle =. Habitat: A Platform for Embodied. 2019 , url =

    2019

  18. [18]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Habitat 2.0: Training Home Assistants to Rearrange their Habitat , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  19. [19]

    Neural Topological

    Devendra Singh Chaplot and Ruslan Salakhutdinov and Abhinav Gupta and Saurabh Gupta , booktitle =. Neural Topological. 2020 , url =

    2020

  20. [20]

    2023 , url =

    Dhruv Shah and Blazej Osinski and Brian Ichter and Sergey Levine , booktitle =. 2023 , url =

    2023

  21. [21]

    Conference on Computer Vision and Pattern Recognition , year=

    Molmo2: Open weights and data for vision-language models with video understanding and grounding , author=. Conference on Computer Vision and Pattern Recognition , year=

  22. [22]

    International Conference on Learning Representations , year =

    Sequence Level Training with Recurrent Neural Networks , author =. International Conference on Learning Representations , year =

  23. [23]

    Galaxea R1 , howpublished =

  24. [24]

    JetAuto ROS Robot , howpublished =

  25. [25]

    2023 , organization=

    Zitkovich, Brianna and Yu, Tianhe and Xu, Sichun and Xu, Peng and Xiao, Ted and Xia, Fei and Wu, Jialin and Wohlhart, Paul and Welker, Stefan and Wahid, Ayzaan and others , booktitle=. 2023 , organization=

    2023

  26. [26]

    International Conference on Computer Vision , year =

    Waypoint Models for Instruction-guided Navigation in Continuous Environments , author =. International Conference on Computer Vision , year =

  27. [27]

    2024 , publisher=

    An, Dong and Wang, Hanqing and Wang, Wenguan and Wang, Zun and Huang, Yan and He, Keji and Wang, Liang , journal =. 2024 , publisher=

    2024

  28. [28]

    Chang and Manolis Savva and Yili Zhao and Dhruv Batra , booktitle =

    Santhosh Kumar Ramakrishnan and Aaron Gokaslan and Erik Wijmans and Oleksandr Maksymets and Alexander Clegg and John Turner and Eric Undersander and Wojciech Galuba and Andrew Westbury and Angel X. Chang and Manolis Savva and Yili Zhao and Dhruv Batra , booktitle =. Habitat-Matterport

  29. [29]

    Ku, Alexander and Anderson, Peter and Patel, Roma and Ie, Eugene and Baldridge, Jason , booktitle=