Pith. sign in

REVIEW 2 major objections 5 minor 45 references

LoTIS shows that visual navigation can be reduced to localizing a recorded reference trajectory in the robot's current camera view, and that this image-space guidance alone drives different robots along a route, forward or backward, without

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 21:49 UTC pith:QQGRMAG2

load-bearing objection LoTIS's core idea—predicting the whole reference trajectory in image-space—is a real advance, and the sim evidence is strong; the real-world numbers need statistical humility. the 2 major comments →

arxiv 2602.18803 v2 pith:QQGRMAG2 submitted 2026-02-21 cs.RO

Learning to Localize Reference Trajectories in Image-Space for Visual Navigation

Finn Lukas Busch , Matti Vahs , Quantao Yang , Jes\'us Gerardo Ortega Peimbert , Yixi Cai , Jana Tumova , Olov Andersson This is my paper
classification cs.RO
keywords visual navigationreference trajectoryimage-space localizationrobot-agnostic guidancecross-trajectory trainingtransformerbackward traversalcamera mismatch
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that visual navigation should be split into perception and control: rather than learning to turn images into actions, a model learns to predict where the poses of a recorded reference route would appear in the robot's current camera view—as 2D points, visibility flags, and normalized distances. The claim is that this image-space representation is enough for an off-the-shelf local planner to drive the robot along the route, and that because the output lives in the image, the same prediction works for different robot platforms, cameras, and even backward traversal. The paper backs this with a cross-trajectory training scheme that deliberately mismatches cameras and viewpoints, and with simulation and real-world experiments showing 94–98% forward success and large gains over action-learning baselines, plus a demonstration that one phone-recorded video guides both a quadrotor and a quadruped. If the claim is right, recording a route once with any camera makes it navigable by any robot, to any point on the route, in either direction, without calibration.

Core claim

The paper's central discovery is that navigation guidance can be reduced to a single perceptual prediction: for every frame in a reference RGB trajectory, estimate where that frame's viewpoint sits in the robot's current image, whether it is visible, and how far away it is. A transformer architecture processes the full trajectory jointly (rather than choosing one subgoal image), fuses it with the current view, and iteratively refines these predictions. Because the output is defined in the robot's own image frame, any downstream planner—from a yaw-only controller to a model-predictive one—can consume it directly. Trained with a cross-trajectory strategy in which reference and query views come

What carries the argument

The load-bearing object is the image-space trajectory representation: for each reference frame i, the model predicts a 2D point p_i where that frame's camera pose would appear in the current view, a visibility logit v_i, and a normalized distance d_i. The architecture is an asymmetric transformer: a trajectory encoder processes all reference frames jointly with alternating global and per-frame attention, a lightweight query encoder handles the current view, a fusion module lets each trajectory token attend to the query view to establish correspondences, and a recurrent head progressively refines the predictions over several iterations. Training uses cross-trajectory sampling—reference and qu

Load-bearing premise

The method assumes that at every decision step some part of the reference trajectory is visible in the robot's current camera view; when that overlap disappears (sharp backward corners, extreme height mismatch, or occlusion), the model cannot produce guidance and navigation degrades—a limitation the paper explicitly concedes.

What would settle it

Start a robot in a view that shares no visible content with any frame of the reference trajectory—for instance, on the far side of a sharp corner or behind a closed door—and check whether the model reports zero visible trajectory points and the robot stalls; the paper's overlap assumption predicts exactly this failure, so a successful traversal from such a start would show the assumption is not load-bearing.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single phone-recorded route video can serve as a navigation instruction for any camera-equipped robot, without recalibration or retraining.
  • Navigation to any point along a route—including backward traversal—becomes possible from starting positions that are off the route, provided some part of the route is visible.
  • Mismatched cameras (field of view, aspect ratio, mounting height) between the recording and the robot no longer cause navigation to fail, within the ranges tested in the paper.
  • Because the output is image-space guidance, the same perception model can be paired with different local planners and action spaces, including aerial and legged platforms.
  • The full-trajectory joint processing avoids the subgoal-selection failures of prior methods, so localization accuracy and success rate degrade much more slowly with distance from the route.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The decoupling principle points toward a modular navigation stack where perception and planning improve independently; one testable extension is training the model to emit a heading or 'pointing' signal when the trajectory is out of view, which the paper lists as future work.
  • The visual-overlap dependency suggests the method is inherently a local navigation oracle; scaling to long routes will likely require memory or recovery mechanisms, which the paper acknowledges via trajectory chunking.
  • Cross-trajectory sampling could be exported to other vision-for-robot tasks that currently need action-labeled demonstrations, potentially shrinking the data burden for embodiment-agnostic perception.
  • One might test whether the model is implicitly learning the 3D shape of the trajectory: if it does, its predictions could be used to reconstruct path geometry or for localization in repetitive environments where appearance matching is unreliable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes LoTIS, a learned perception module that, given a reference RGB trajectory and a current query image, predicts for each reference frame its image-space coordinates in the query view, a visibility logit, and a normalized distance. This representation is meant to be robot-agnostic and calibration-free, so that a simple yaw controller or an MPPI controller can use it for navigation on different embodiments. Training uses a cross-trajectory sampling strategy on real and simulated datasets, with labels generated from known poses and depth. The evaluation covers Gibson and HM3D simulation, comparing against ViNT, PlaceNav, NoMaD, and FAINT under matched/cross cameras and on/off-trajectory starts, as well as real-world trials on a Crazyflie quadrotor and a Spot quadruped using phone-recorded reference trajectories. The paper reports large simulation margins and high real-world success rates, and includes an explicit limitations section. The central claim is that decoupling perception from action in this way enables zero-shot embodiment transfer, backward traversal, and robustness to camera mismatch.

Significance. If the results hold, LoTIS is a meaningful step forward: the image-space trajectory-localization representation is a clean interface between learned perception and classical planning, the cross-trajectory training strategy is an elegant way to obtain viewpoint robustness without action labels, and the real-world deployment on both an aerial and a legged robot from phone-recorded trajectories is a valuable demonstration. The simulation study is substantial—200 runs per setup, multiple held-out scene datasets, and several strong baselines with official pretrained weights—and the paper is commendably candid about the visual-overlap requirement and other limitations. The availability of code and videos is a further strength. However, the real-world quantitative claims currently rest on very small trial counts, and the main ablation table is not clearly tied to the configuration reported in Table I. These issues need to be addressed before the claims as stated can be accepted.

major comments (2)
  1. [Abstract; Sec. VI-B, Tables II and III] The abstract's quantitative claims ('94–98% success rate across diverse sim and real environments', '5× improvements on backward traversal') rest in part on real-world results reported as point estimates from 3–6 trials per condition (Table II: 36 LoTIS runs, 18 per direction; Table III: 3–6 per robustness condition). No confidence intervals, significance tests, per-trial logs, or seeds are given. With n=6, a 6/6 success rate has a 95% Clopper-Pearson lower bound near 54%; with n=3, near 29%; the aggregate 35/36 has a lower bound near 85%. The reported point estimates are therefore compatible with substantially lower true success rates. The day→night backward cell (2/3) and the crowded-env backward cell (5/6) are particularly uncertain. This does not invalidate the simulation evidence (Table I, 200 runs per cell), but it undercuts the 'sim and real' phrasing and leaves the cross-embodime
  2. [Appendix C, Table A1] The values labeled 'LoTIS-F' in Table A1 are identical to the '+ Obstcl. Avoidance' row of Table I, not the base 'LoTIS (Ours)' row. For example, Gibson To End On-Trajectory Matched is 100.0 in Table A1 but 94.7 in Table I for LoTIS without obstacle avoidance. The appendix does not state that obstacle avoidance was enabled in the ablation. Since this table is the primary evidence for the contribution that full-trajectory joint processing is responsible for the gains, the configuration must be stated explicitly, and ideally the ablation should be reported for both controller variants so the reader can separate the effect of joint processing from the effect of obstacle avoidance.
minor comments (5)
  1. [Appendix B] The video list appears to swap table references: 'Real-World Navigation' is associated with Table III (which is the robustness table) and 'Robustness Studies' with Table II (which is the main real-world results table). Please correct the cross-references.
  2. [Fig. 4 caption] The caption has missing placeholders: 'starts at and ends at , with initial experiment positions shown as .' The symbols and legend are not rendered. Please fix the caption so the figure is self-contained.
  3. [Abstract; Sec. IV-D and Appendix H-B] The abstract says the system works 'without requiring camera calibration, poses, or robot-specific training.' The MPPI controller used for the quadrotor relies on UniDepthV2 to predict camera parameters and metric depth for collision avoidance and 3D grounding. Please clarify that the calibration-free claim applies to the LoTIS perception model itself, not to every downstream controller, or state the depth/camera assumption in the system-level claim.
  4. [Sec. I] The claim 'for the first time, using an RGB reference trajectory in the general setting...' is a strong novelty statement. Please either soften it or cite prior work that attempted this general setting and explain how the present contribution differs.
  5. [Throughout] Minor typos and wording issues: 'where reference trajectory would appear' is missing an article; 'loosing visibility' in Appendix H-B should be 'losing visibility'; the phrase 'we achieve over 5x improvements' could be tightened to specify the comparison set. A careful proofread is recommended.

Circularity Check

0 steps flagged

No significant circularity: LoTIS's image-space guidance is trained on externally generated geometric labels and evaluated on independent navigation tasks.

full rationale

Walking the derivation chain: the proposed representation (p_i, v_i, d_i) is defined in the robot's current image space, but its training targets are not derived from the model's own outputs. The paper states "Ground truth labels are generated via geometric projection using camera poses and depth maps" (Sec. IV-C.1), i.e., supervision comes from poses and depth, not from the navigation success signal. Training uses roughly 25,000 reference trajectories and 850,000 query images across 500 environments, and simulation evaluation is on scenes "not seen in training" (Sec. V-A), so the reported success rates are not fitted values renamed as predictions. The downstream controllers (yaw + constant velocity, MPPI) consume the model outputs through standard costs, and success is defined as arriving within 0.5 m of the goal, independently of the predicted image-space coordinates. The normalized distance d_i is normalized per query, but its only role in the yaw controller is ordinal selection (argmin over visible points), so no metric claim reduces to the normalization by construction. Baselines are official pretrained models (ViNT, NoMaD, PlaceNav, FAINT), providing external comparison rather than a self-citation. The paper's architectural choices cite external prior work (DINOv3, VGGT, RAFT); none of these is an unverified self-citation, and there is no imported uniqueness theorem. The explicitly stated visual-overlap requirement (Sec. III and Limitations) is an honest boundary condition, not a circular redefinition. The real-world tables use only 3-6 trials per condition, which is a legitimate evidential weakness about statistical confidence, but it is not circularity: small samples do not make the evaluation equivalent to the training inputs. No load-bearing step in the paper reduces, by the paper's own equations or by self-citation, to its own inputs. Score 0.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The central claim rests on standard deep learning assumptions: feature quality, dataset coverage, and the adequacy of the normalized distance signal. No fitted constants or invented physical entities are introduced beyond the trained network's parameters.

axioms (4)
  • domain assumption Frozen DINOv3 features provide sufficient geometric correspondence across viewpoints, cameras, and appearance changes.
    LoTIS uses a frozen DINOv3 backbone for both trajectory and query images; the accuracy of the entire prediction relies on these features being viewpoint-invariant enough to learn trajectory localization. This is not proven but empirically supported.
  • domain assumption The training data distribution (HM3D, Gibson, HSSD, AI2-THOR, CODa, etc.) is representative enough to generalize to unseen test environments.
    The model is trained on a mixture of sim and real datasets and tested on held-out Gibson/HM3D scenes and four real environments. High success is attributed to this coverage, but no theoretical guarantee exists.
  • domain assumption The normalized distance representation (farthest visible point set to 1) is sufficient for navigation without metric scale.
    The controller selects the closest visible point using di and steers toward it. This assumes that relative distance ordering is enough to make progress toward the goal, which is plausible but not formally established.
  • domain assumption Ground-truth labels from depth and pose reprojection are accurate enough to train the model.
    For real data, depth is estimated by PriorDepthAnything; for simulation, depth is synthetic. Errors in these labels propagate to the learned predictions.

pith-pipeline@v1.3.0-alltime-deepseek · 19814 in / 11606 out tokens · 91932 ms · 2026-08-02T21:49:51.239724+00:00 · methodology

0 comments
read the original abstract

We present LoTIS, a model for visual navigation that provides robot-agnostic image-space guidance by localizing a reference RGB trajectory in the robot's current view, without requiring camera calibration, poses, or robot-specific training. Instead of predicting actions tied to specific robots, we predict the image-space coordinates of the reference trajectory as they would appear in the robot's current view. This creates robot-agnostic visual guidance that easily integrates with local planning. Consequently, our model's predictions provide guidance zero-shot across diverse embodiments. By decoupling perception from action and learning to localize trajectory points rather than imitate behavioral priors, we enable a cross-trajectory training strategy for robustness to viewpoint and camera changes. We outperform state-of-the-art methods by 20-50 percentage points in success rate on conventional forward navigation, achieving 94-98% success rate across diverse sim and real environments. Furthermore, we achieve over 5x improvements on challenging tasks where baselines fail, such as backward traversal. The system is straightforward to use: we show how even a video from a phone camera directly enables different robots to navigate to any point on the trajectory. Videos, demo, and code are available at https://finnbusch.com/lotis.

Figures

Figures reproduced from arXiv: 2602.18803 by Finn Lukas Busch, Jana Tumova, Jes\'us Gerardo Ortega Peimbert, Matti Vahs, Olov Andersson, Quantao Yang, Yixi Cai.

Figure 1
Figure 1. Figure 1: Given only a reference trajectory of (unposed) RGB images [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: LoTIS Architecture. Reference trajectory T and query Iq are processed by frozen DINOv3 backbones. A trajectory encoder (ET ) captures spatio￾temporal context once (offline), while a query encoder (Eq) and query-trajectory fusion (FqT ) perform online feature extraction and fusion, respectively. Finally, a recurrent transformer iteratively regresses image-space coordinates (pi), visibility (vi), and distanc… view at source ↗
Figure 3
Figure 3. Figure 3: Relative success rate (SR) for all methods on off-trajectory initial [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Four trajectories used for real-world evaluation. Each reference trajectory starts at [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Impact of environment changes on the predictions of our model with [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 9 linked inside Pith

  1. [1]

    Rethinking visual geo-localization for large-scale appli- cations

    Gabriele Berton, Carlo Masone, and Barbara Caputo. Rethinking visual geo-localization for large-scale appli- cations. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4878– 4888, 2022

    2022

  2. [2]

    Eigenplaces: Training viewpoint robust models for visual place recognition

    Gabriele Berton, Gabriele Trivigno, Barbara Caputo, and Carlo Masone. Eigenplaces: Training viewpoint robust models for visual place recognition. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11080–11090, 2023

    2023

  3. [3]

    Potential problems of stability and convergence in image-based and position-based visual servoing

    Francois Chaumette. Potential problems of stability and convergence in image-based and position-based visual servoing. InThe confluence of vision and control, pages 66–78. Springer, 2007

    2007

  4. [4]

    Visual servo control

    Francois Chaumette and Seth Hutchinson. Visual servo control. i. basic approaches.IEEE Robotics & Automa- tion Magazine, 13(4):82–90, 2006. doi: 10.1109/MRA. 2006.250573

    arXiv 2006

  5. [5]

    Visual servo control

    Francois Chaumette and Seth Hutchinson. Visual servo control. ii. advanced approaches [tutorial].IEEE Robotics & Automation Magazine, 14(1):109–118, 2007. doi: 10. 1109/MRA.2007.339609

    arXiv 2007

  6. [6]

    Cautious weight decay.arXiv preprint arXiv:2510.12402, 2025

    Lizhang Chen, Jonathan Li, Kaizhao Liang, Baiyu Su, Cong Xie, Nuo Wang Pierse, Chen Liang, Ni Lao, and Qiang Liu. Cautious weight decay.arXiv preprint arXiv:2510.12402, 2025

    arXiv 2025

  7. [7]

    A new approach to visual servoing in robotics

    Bernard Espiau, Franc ¸ois Chaumette, and Patrick Rives. A new approach to visual servoing in robotics. In Workshop on Geometric Reasoning for Perception and Action, pages 106–136. Springer, 1991

    1991

  8. [8]

    Pampc: Perception-aware model predictive control for quadrotors

    Davide Falanga, Philipp Foehn, Peng Lu, and Davide Scaramuzza. Pampc: Perception-aware model predictive control for quadrotors. In2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1–8. IEEE, 2018

    2018

  9. [9]

    Muon: An optimizer for hidden layers in neural networks,

    Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks,

  10. [10]

    Habitat synthetic scenes dataset (hssd-200): An analysis of 3d scene scale and realism tradeoffs for objectgoal navigation

    Mukul Khanna, Yongsen Mao, Hanxiao Jiang, San- jay Haresh, Brennan Shacklett, Dhruv Batra, Alexander Clegg, Eric Undersander, Angel X Chang, and Manolis Savva. Habitat synthetic scenes dataset (hssd-200): An analysis of 3d scene scale and realism tradeoffs for objectgoal navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern R...

    2024

  11. [11]

    Ai2-thor: An interactive 3d environment for visual ai.arXiv preprint arXiv:1712.05474, 2017

    Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli Vander- Bilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai.arXiv preprint arXiv:1712.05474, 2017

    Pith/arXiv arXiv 2017

  12. [12]

    Instance-specific image goal navigation: Training embodied agents to find object instances.arXiv preprint arXiv:2211.15876, 2022

    Jacob Krantz, Stefan Lee, Jitendra Malik, Dhruv Batra, and Devendra Singh Chaplot. Instance-specific image goal navigation: Training embodied agents to find object instances.arXiv preprint arXiv:2211.15876, 2022

    Pith/arXiv arXiv 2022

  13. [13]

    Navigating to objects specified by images

    Jacob Krantz, Theophile Gervet, Karmesh Yadav, Austin Wang, Chris Paxton, Roozbeh Mottaghi, Dhruv Batra, Jitendra Malik, Stefan Lee, and Devendra Singh Chap- lot. Navigating to objects specified by images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10916–10925, 2023

    2023

  14. [14]

    Grounding image matching in 3d with mast3r

    Vincent Leroy, Yohann Cabon, and J ´erˆome Revaud. Grounding image matching in 3d with mast3r. In European Conference on Computer Vision, pages 71–91. Springer, 2024

    2024

  15. [15]

    Depth anything 3: Recovering the visual space from any views.arXiv preprint arXiv:2511.10647, 2025

    Haotong Lin, Sili Chen, Junhao Liew, Donny Y Chen, Zhenyu Li, Guang Shi, Jiashi Feng, and Bingyi Kang. Depth anything 3: Recovering the visual space from any views.arXiv preprint arXiv:2511.10647, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Botanicgarden: A high- quality dataset for robot navigation in unstructured natu- ral environments.IEEE Robotics and Automation Letters, 9(3):2798–2805, 2024

    Yuanzhi Liu, Yujia Fu, Minghui Qin, Yufeng Xu, Baoxin Xu, Fengdong Chen, Bart Goossens, Poly ZH Sun, Hongwei Yu, Chun Liu, et al. Botanicgarden: A high- quality dataset for robot navigation in unstructured natu- ral environments.IEEE Robotics and Automation Letters, 9(3):2798–2805, 2024

    2024

  17. [17]

    Sampling-based mpc for constrained vision based control

    Ihab S Mohamed, Guillaume Allibert, and Philippe Mar- tinet. Sampling-based mpc for constrained vision based control. In2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3753–3758. IEEE, 2021

    2021

  18. [18]

    Riku Murai, Eric Dexheimer, and Andrew J. Davison. MASt3R-SLAM: Real-time dense SLAM with 3D re- construction priors. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025

    2025

  19. [19]

    Tartanground: A large-scale dataset for ground robot perception and navigation.arXiv preprint arXiv:2505.10696, 2025

    Manthan Patel, Fan Yang, Yuheng Qiu, Cesar Ca- dena, Sebastian Scherer, Marco Hutter, and Wen- shan Wang. Tartanground: A large-scale dataset for ground robot perception and navigation.arXiv preprint arXiv:2505.10696, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Scalable diffu- sion models with transformers

    William Peebles and Saining Xie. Scalable diffu- sion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

    2023

  21. [21]

    Unidepthv2: Universal monocular met- ric depth estimation made simpler.arXiv preprint arXiv:2502.20110, 2025

    Luigi Piccinelli, Christos Sakaridis, Yung-Hsu Yang, Mattia Segu, Siyuan Li, Wim Abbeloos, and Luc Van Gool. Unidepthv2: Universal monocular met- ric depth estimation made simpler.arXiv preprint arXiv:2502.20110, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai.arXiv preprint arXiv:2109.08238, 2021

    Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wij- mans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, et al. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai.arXiv preprint arXiv:2109.08238, 2021

    Pith/arXiv arXiv 2021

  23. [23]

    Habitat: A Platform for Embodied AI Research

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019

    2019

  24. [24]

    Gnm: A general navigation model to drive any robot

    Dhruv Shah, Ajay Sridhar, Arjun Bhorkar, Noriaki Hi- rose, and Sergey Levine. Gnm: A general navigation model to drive any robot. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7226–7233. IEEE, 2023

    2023

  25. [25]

    Vint: A foundation model for visual navigation

    Dhruv Shah, Ajay Sridhar, Nitish Dashora, Kyle Sta- chowicz, Kevin Black, Noriaki Hirose, and Sergey Levine. Vint: A foundation model for visual navigation. arXiv preprint arXiv:2306.14846, 2023

    Pith/arXiv arXiv 2023

  26. [26]

    Theia: Distilling diverse vision foundation models for robot learning

    Jinghuan Shang, Karl Schmeckpeper, Brandon B May, Maria Vittoria Minniti, Tarik Kelestemur, David Watkins, and Laura Herlant. Theia: Distilling diverse vision foundation models for robot learning. InConference on Robot Learning, pages 724–748. PMLR, 2025

    2025

  27. [27]

    Dinov3.arXiv preprint arXiv:2508.10104, 2025

    Oriane Sim ´eoni, Huy V V o, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Micha ¨el Ramamonjisoa, et al. Dinov3.arXiv preprint arXiv:2508.10104, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Nomad: Goal masked diffusion policies for navigation and exploration

    Ajay Sridhar, Dhruv Shah, Catherine Glossop, and Sergey Levine. Nomad: Goal masked diffusion policies for navigation and exploration. In2024 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pages 63–70. IEEE, 2024

    2024

  29. [29]

    Roformer: Enhanced trans- former with rotary position embedding.Neurocomputing, 568:127063, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced trans- former with rotary position embedding.Neurocomputing, 568:127063, 2024

    2024

  30. [30]

    Placenav: Topological navigation through place recognition

    Lauri Suomela, Jussi Kalliola, Harry Edelman, and Joni- Kristian K ¨am¨ar¨ainen. Placenav: Topological navigation through place recognition. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 5205–5213. IEEE, 2024

    2024

  31. [31]

    Torres, Harry Edelman, and Joni-Kristian K ¨am¨ar¨ainen

    Lauri Suomela, Sasanka Kuruppu Arachchige, German F. Torres, Harry Edelman, and Joni-Kristian K ¨am¨ar¨ainen. Synthetic vs. real training data for visual navigation. In arXiv preprint arXiv:2509.11791, 2025

    arXiv 2025

  32. [32]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. InEuropean conference on computer vision, pages 402–419. Springer, 2020

    2020

  33. [33]

    Benchmark for evaluating long-term localization in indoor environments under substantial static and dynamic scene changes

    Niklas Trekel, Tiziano Guadagnino, Thomas L ¨abe, Louis Wiesmann, Perrine Aguiar, Jens Behley, and Cyrill Stach- niss. Benchmark for evaluating long-term localization in indoor environments under substantial static and dynamic scene changes. In2025 IEEE/RSJ International Confer- ence on Intelligent Robots and Systems (IROS), pages 10770–10777. IEEE, 2025

    2025

  34. [34]

    Vggt: Visual geometry grounded transformer

    Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. InProceedings of the Computer Vision and Pattern Recognition Confer- ence, pages 5294–5306, 2025

    2025

  35. [35]

    Dust3r: Geometric 3d vision made easy

    Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20697–20709, 2024

    2024

  36. [36]

    Depth any- thing with any prior.arXiv preprint arXiv:2505.10565, 2025

    Zehan Wang, Siyu Chen, Lihe Yang, Jialei Wang, Ziang Zhang, Hengshuang Zhao, and Zhou Zhao. Depth any- thing with any prior.arXiv preprint arXiv:2505.10565, 2025

    Pith/arXiv arXiv 2025

  37. [37]

    Aggressive driving with model predictive path integral control

    Grady Williams, Paul Drews, Brian Goldfain, James M Rehg, and Evangelos A Theodorou. Aggressive driving with model predictive path integral control. In2016 IEEE international conference on robotics and automation (ICRA), pages 1433–1440. IEEE, 2016

    2016

  38. [38]

    Model predictive path integral control: From theory to parallel computation.Journal of Guidance, Control, and Dynamics, 40(2):344–357, 2017

    Grady Williams, Andrew Aldrich, and Evangelos A Theodorou. Model predictive path integral control: From theory to parallel computation.Journal of Guidance, Control, and Dynamics, 40(2):344–357, 2017

    2017

  39. [39]

    Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese

    Fei Xia, Amir R. Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson Env: real- world perception for embodied agents. InComputer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on. IEEE, 2018

    2018

  40. [40]

    Toward robust robot 3-d perception in urban environments: The ut campus object dataset.IEEE Transactions on Robotics, 40:3322–3340, 2024

    Arthur Zhang, Chaitanya Eranki, Christina Zhang, Ji- Hwan Park, Raymond Hong, Pranav Kalyani, Lochana Kalyanaraman, Arsh Gamare, Arnav Bagad, Maria Es- teva, et al. Toward robust robot 3-d perception in urban environments: The ut campus object dataset.IEEE Transactions on Robotics, 40:3322–3340, 2024. Appendix A. OVERVIEW This appendix provides additional...

    2024

  41. [42]

    At k= 0a learned empty-pose token is used instead

    The prediction from iterationk−1isdetachedfrom the computation graph and embedded via a linear layer. At k= 0a learned empty-pose token is used instead

  42. [43]

    These condition the input tokens via Adaptive Layer Normalization: x′ =x+gate⊙ h (1 +scale)⊙RMSNorm(x) +shift i

    The embedded vector is projected through SiLU→Linear to produce shift, scale, and gate vectors. These condition the input tokens via Adaptive Layer Normalization: x′ =x+gate⊙ h (1 +scale)⊙RMSNorm(x) +shift i

  43. [44]

    The conditioned tokens are processed by a trunk of 3 self-attention blocks, each followed by an MLP with expansion ratio 3

  44. [45]

    The final predictions are mapped to bounded ranges: image coordinatesp i = tanh(·)∈[−1,1]; a visibility logitv i; and a normalized distanced i = 1 2 tanh(·) + 1 ∈[0,1]

    A two-layer MLP projects the output to a 4-dimensional residual update, which is accumulated onto the running prediction. The final predictions are mapped to bounded ranges: image coordinatesp i = tanh(·)∈[−1,1]; a visibility logitv i; and a normalized distanced i = 1 2 tanh(·) + 1 ∈[0,1]. B. Training Configuration a) Optimization:We employ the MUON [9] o...

  45. [2024]

    URL https://kellerjordan.github.io/posts/muon/