Pith. sign in

REVIEW 2 major objections 6 minor 2 cited by

CityWalker: Learning Embodied Urban Navigation from Web-Scale Videos

T0 review · 2 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a navigation policy trained on 2,000+ hours of web walking and driving videos, labeled only by noisy visual odometry, reaches a 77.3% real-world success rate in urban navigation, beating fine-tuned baselines.

desk verdict CityWalker is a genuinely useful result in embodied urban navigation, with real-world quadruped trials and data-scaling evidence, but the training pipeline has a load-bearing gap: target waypoint and arrival labels for web videos are never specified. read the letter →

arxiv 2411.17820 v3 pith:4C6DEMDM submitted 2024-11-26 cs.CV cs.RO

Xinhao Liu , Jintong Li , Yicheng Jiang , Niranjan Sujay , Zhicheng Yang , Juexiao Zhang , John Abanes , Jing Zhang
show 1 more author
Chen Feng
This is my paper · ORCID
classification cs.CVcs.RO
keywords urbannavigationimitationlearningweb-scalevideovisualodometrypseudo-labelspoint-goalembodiedagentsdatascalingcross-domaintransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CityWalker claims that an embodied agent can learn to navigate dense urban environments by imitating the motion in thousands of hours of unlabeled web video of people walking and driving through cities. A data pipeline turns noisy visual-odometry estimates of the camera's movement into action labels, so no manual annotation or language-model prompting is needed. After light fine-tuning on six hours of teleoperated quadruped data, the policy navigates unseen city routes with a 77.3% real-world success rate, against 57.1% for the best fine-tuned baseline and 42.9% for a zero-shot baseline. The paper further shows that performance rises with training-data hours and that mixing driving video with walking video accelerates those gains, which matters because urban navigation has resisted both simulation-trained and teleoperation-only policies.

What carries the argument

The load-bearing mechanism is the conversion of raw web video into imitation-learning supervision: visual odometry (DPVO) gives relative poses between frames, and each action is normalized by the average step length of its trajectory, folding the scale ambiguity of monocular odometry and the different stride lengths of walking versus driving into one abstract action space. The learned policy is a transformer that ingests past frame tokens, a history of positions, and the target waypoint, and emits a five-step action horizon together with an arrival prediction. Its training combines an orientation loss (negative cosine similarity between predicted and ground-truth actions), an L1 action loss, a binary arrival loss, and a feature-hallucination loss that penalizes the gap between predicted future tokens and the actual future frame tokens, pressing the model to anticipate what it will see next. The design bet is that volume and diversity of data compensate for the noise in odometry-derived labels.

What would settle it

Reproduce the data pipeline on a subset of the sourced web videos: if target coordinates and arrival labels cannot be recovered from the visual-odometry trajectories alone (or from another described source), the training procedure as written cannot produce the reported policy. A complementary check is to rerun the real-world trials with more than the reported 8-14 runs per maneuver and with a more accurate position source, since the paper itself notes sensitivity to GPS noise; the claimed 20-point success gap over fine-tuned ViNT should survive those conditions.

Watch

Extended reading notes

Core claim

The paper's central claim is that noisy pseudo-labels from off-the-shelf visual odometry are sufficient supervision to learn urban navigation from in-the-wild video. Deep patch visual odometry (DPVO) extracts relative camera poses between nearby frames of 2,000+ hours of city walking and driving footage, and each action is normalized by the average step length of its trajectory so that scale differences between walking and driving disappear. The CityWalker policy is a transformer that consumes frozen DINOv2 features of past frames, past positions, and a target waypoint, and predicts five future actions plus an arrival flag; it is trained with an orientation loss, an L1 action loss, an arrival BCE loss, and a feature-hallucination loss that asks the transformer to predict future image tokens. Fine-tuned on six hours of teleoperated Unitree Go1 data in New York City, it reaches 77.3% real-world success versus 57.1% for fine-tuned ViNT and 42.9% for zero-shot NoMaD, with the largest gains in turns, crossings, crowds, and proximity situations; its own ablations attribute most of the gain to data scale and fine-tuning rather than the auxiliary losses. The paper's conclusion is that abundant online video can substitute for expensive expert teleoperation in building robust urban navigation policies.

Load-bearing premise

Every training sample requires a target waypoint coordinate and an arrival-status label, but the paper never explains where these come from for the unlabeled web videos, whose only extracted labels are camera-motion estimates.

Editorial extensions

If this is right

  • A navigation policy fine-tuned with just six hours of expert data after web-video pretraining reaches 77.3% real-world success, versus 57.1% for fine-tuned ViNT and 42.9% for zero-shot NoMaD.
  • The zero-shot web-trained model already matches or beats fine-tuned baselines on offline arrival accuracy (84.1% versus 70.7%), implying pretraining on diverse web video can substitute for in-domain expert data.
  • Performance scales with data: beyond roughly 1,000 training hours the zero-shot model surpasses a fine-tuned ViNT, and 250 hours of mixed walking-plus-driving video nearly matches 1,000 hours of walking video alone.
  • Because the label pipeline is parallelizable visual odometry rather than language-model prompting, scaling to larger video corpora is cheap, keeping the approach cost-effective.
  • Driving video transfers to a quadruped roughly as well as zero-shot baselines, so cross-domain and cross-embodiment video is a usable source of navigation supervision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural reading of the pipeline is that target waypoints are future positions along the VO trajectory and arrival labels come from detecting when forward motion stops; if so, the whole training procedure is annotation-free, but the paper does not state this explicitly.
  • If the scaling curve holds, training on a far larger mixed corpus (tens of thousands of hours across more cities) is a direct next experiment, and the saturation point of web-video imitation for urban navigation has yet to be found.
  • The real-world result rests on 8-14 trials per maneuver type in a limited set of neighborhoods; a larger multi-city trial is the implied test of whether the 20-point success gap over ViNT persists.
  • Since actions are stored in a step-length-normalized space, the same policy should transfer to other embodiments such as wheeled delivery robots by changing only the denormalization constant, an experiment the paper motivates but does not run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. CityWalker proposes a waypoint-goal visual navigation policy trained via imitation learning on 2000+ hours of unlabeled city walking and driving videos. Action supervision is extracted from video with a visual odometry model (DPVO) that yields relative poses, normalized per trajectory. The model uses a frozen DINOv2 encoder, a coordinate encoder, and a transformer with action and arrival prediction heads; training combines L1, orientation, arrival (BCE), and feature-hallucination losses. Evaluation on held-out NYC teleoperation data and in real-world deployment on a Unitree Go1 quadruped reports 77.3% success versus 57.1% for fine-tuned ViNT and 42.9% for zero-shot NoMaD (Table 2). The paper also presents data-scaling experiments and ablations.

Significance. If the training pipeline is fully specified, this work would be a significant demonstration of scalable imitation learning from internet video for embodied urban navigation. The paper's strengths include the large-scale data collection, real-world deployment on a quadruped, and honest ablations that acknowledge the marginal contributions of the orientation and feature-hallucination losses. The main weakness is the underspecification of the target-waypoint and arrival labels, which is essential for reproducing the central claim of 'no costly annotations'.

major comments (2)
  1. [Sec. 3.4, Eq. (4), Fig. 2] The training pipeline requires a target waypoint coordinate and a binary arrival label for every sample, but the paper never specifies how these labels are derived for the unlabeled web videos. Section 3.3 describes only how action labels are obtained from VO relative poses and step-length normalization. Without a concrete rule (e.g., the target is the VO pose at a fixed distance ahead, and arrival is a distance threshold), Eq. (4) cannot be evaluated on the purported training data. This is load-bearing because the scalability claim rests on avoiding costly annotations; an unstated heuristic or manual step would weaken that claim. A precise description or code release would resolve this gap.
  2. [Sec. 4.2, Table 2] The real-world success rates are presented without the number of trials per condition and without confidence intervals or significance testing. The text says '8-14 trails for each cases,' which is ambiguous, and the total number of trials is not reported. Since Table 2 is the main evidence for the headline claim of outperforming existing methods, the paper should report trial counts per condition and ideally confidence intervals or a statistical test; otherwise the 77.3% versus 57.1% gap may not be robust given the small sample.
minor comments (6)
  1. [Sec. 4.2, Table 1] The 'Arrival (%)' metric is reported for all methods, but GNM, ViNT, and NoMaD do not have an arrival prediction head; the paper does not specify how arrival is determined for these baselines. Please clarify the protocol used to compute arrival for methods without an arrival head.
  2. [Sec. 4.2] The heading 'Depolyment' should be 'Deployment', and '8-14 trails' should be '8-14 trials'.
  3. [Sec. 4.4, Table 3] The ablation shows that orientation loss and feature hallucination loss provide only marginal improvements (17.03 to 17.00 and 17.00 to 17.02); the text does acknowledge this, but the caption and surrounding discussion could be more explicit about the lack of a statistically meaningful effect.
  4. [Appendix A, Table II] The hyperparameter table uses 'Cord.' and 'φ' while the main text uses 'Coord.' and 'ω'; please unify the notation across the paper and appendix.
  5. [Fig. 6] The figure legend and axis labels are hard to read; the text refers to markers (+) and (x) that are not clearly visible in the printed version. Larger fonts or a separate legend would help.
  6. [Sec. 4.1] The definition of 'Turn' via φaction > 20° appears to measure the action angle relative to a global axis, not the change in direction between consecutive actions; please clarify whether this is the intended operationalization.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: training pseudo-labels are evaluated against independent LiDAR-SLAM ground truth and real-world success.

full rationale

The paper's central claim is that imitation learning on web videos with visual-odometry pseudo-labels improves urban navigation. The evaluation chain is independent of the training labels: offline metrics (Table 1) use teleoperation trajectories with ground-truth poses from LiDAR-SLAM (Section 4.1), and real-world success (Table 2) is measured by physical arrival at targets, with human interruptions treated as failures. The orientation loss in Eq. 3 is the same angular quantity as the MAOE metric, but this is a training objective, not a fitted parameter renamed as a prediction. The feature-hallucination and arrival heads are auxiliary; ablation (Table 3) shows their contribution is marginal, so no load-bearing claim reduces to them. The paper's self-citations ([23], [55]) are related-work and inspiration references, not used to justify the scaling result. The main reproducibility gap is that Section 3.4 and Eq. 4 require target-coordinate and arrival labels for web videos, but Section 3.3 only describes deriving action labels from VO; this is an omitted specification, not a circularity, because the reported evaluation does not reuse the training labels. The stated GPS-noise limitation (Section 5) is an acknowledged engineering caveat and does not affect the independence of the benchmarks. Overall, no derivation step reduces to its own inputs, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the assumption that noisy VO poses from human walking and driving videos, after scale normalization, are valid supervision for a quadruped navigation policy. The largest unstated dependency is how target waypoints and arrival status are obtained for unlabeled videos, since the paper only specifies action supervision. Scenario definitions and loss weights add several hand-chosen constants.

free parameters (4)
  • Orientation loss weight omega_ori = 5.0
    Hand-chosen in Section 3.4 to balance loss terms; affects training but is not fitted to evaluation data.
  • Feature hallucination loss weight omega_feat = 0.1
    Hand-chosen balance weight; the ablation shows marginal effect, so this parameter weakly supports the claimed benefit of feature hallucination.
  • Critical scenario thresholds (turn >20 degrees, detour angle deviation >45 degrees, proximity bbox >25 percent image… = 20, 45, 25, 5, 0.5
    Post-hoc evaluation definitions in Section 4.1; changing these thresholds would change reported scenario-mean performance.
  • Per-trajectory normalization step length = Estimated per video from VO
    Used in Section 3.3 to normalize VO actions; a data-derived scale that maps walking and driving actions into a common action space for cross-embodiment transfer.
assumptions (5)
  • domain assumption DPVO visual odometry gives reliable local relative poses within short temporal windows, sufficient for action supervision.
    Section 3.3: the entire pseudo-label pipeline rests on this assumption, though the paper acknowledges global drift and scale ambiguity.
  • domain assumption Normalizing actions by average step length transfers across embodiments and domains, so human-walking actions are valid supervision for a quadruped.
    Section 3.3: this is the core mechanism for cross-embodiment transfer, but no evidence beyond the final results is provided to validate the normalization.
  • ad hoc to paper Target waypoint and arrival labels for unlabeled web videos are available or derivable.
    Section 3.4 and Figure 2 require a target coordinate and arrival supervision, but the paper never specifies how these are produced for in-the-wild videos.
  • domain assumption Frozen DINOv2 features provide a sufficient visual representation for urban navigation.
    Section 3.4: the image backbone is frozen, so the model's perception capacity is bounded by DINOv2 features.
  • domain assumption The action space can be denormalized using the robot's step length during deployment.
    Section 3.3: predicted normalized actions are multiplied by robot step length; this assumes a linear relationship between normalized actions and actual robot motion.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CityWalker: Learning Embodied Urban Navigation from Web-Scale Videos." pith.science (2026). https://pith.science/paper/4C6DEMDM

@misc{pith2026241117820,
  author       = {Pith},
  title        = {Pith review of: CityWalker: Learning Embodied Urban Navigation from Web-Scale Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4C6DEMDM}},
  note         = {Machine review of arXiv:2411.17820}
}
read the original abstract

Navigating dynamic urban environments presents significant challenges for embodied agents, requiring advanced spatial reasoning and adherence to common-sense norms. Despite progress, existing visual navigation methods struggle in map-free or off-street settings, limiting the deployment of autonomous agents like last-mile delivery robots. To overcome these obstacles, we propose a scalable, data-driven approach for human-like urban navigation by training agents on thousands of hours of in-the-wild city walking and driving videos sourced from the web. We introduce a simple and scalable data processing pipeline that extracts action supervision from these videos, enabling large-scale imitation learning without costly annotations. Our model learns sophisticated navigation policies to handle diverse challenges and critical scenarios. Experimental results show that training on large-scale, diverse datasets significantly enhances navigation performance, surpassing current methods. This work shows the potential of using abundant online video data to develop robust navigation policies for embodied agents in dynamic urban settings. Project homepage is at https://ai4ce.github.io/CityWalker/.

Figures

Figures reproduced from arXiv: 2411.17820 by the authors.

Figure 1
Figure 1. Embodied Urban Navigation. Navigating urban spaces is challenging for (especially off-street) mobile agents. The differently colored pins ( ) along the route highlight various critical scenarios unique to complex and dynamic urban landscapes. Thumbnails on the right with corresponding colored pins demonstrate the real-world observation of these challenging cases. Our CityWalker model is trained with over 2000 hours … view at source ↗
Figure 2
Figure 2. Overall Illustration of CityWalker. Our training pipeline starts with internet-sourced videos, using visual odometry to obtain relative poses between frames. At each time step, the model receives past observations, past trajectory, and target location as input. They are encoded via a frozen image encoder and a trainable coordinate encoder. A transformer processes these inputs to generate future tokens. An action hea… view at source ↗
Figure 4
Figure 4. Data Sample and Visual Odometry (VO) Result. Our internet-source training data includes both walking and driving videos. These videos cover various scenarios in the urban envi￾ronment. The VO tool gives noisy trajectories globally, but trust￾worthy local relative pose within a short time period. We identify several key scenarios that are most critical dur￾ing navigation. AOE and MAOE are calculated separately for ea… view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Qualitative Results. Left image shows current observations of two samples. Right plots displays input trajectory, ground truth actions, and predicted actions in the current coordinate system with the agent at the origin. Ours (driving) Ours (driving + walking) [PITH_F…
Figure 6
Figure 6. Figure 6: Performance and Data Size. We show the model performance evaluated by MAOE with respect to the size of the training data measured by video length in hours. We also show the zero-shot performance of our model trained with only driving videos and mixed driving and walkin…
Figure 7
Figure 7. Figure 7: Improvement from Feature Hallucination. The plot shows the orientation loss for different amounts of training data and loss setups. For all four setups, we trian with ωl1 = 1.0, , ωarr = 1.0, and, andωori = 5.0. The unproportional train￾ing steps are due to different b…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Narrate2Nav: Real-Time Visual Navigation with Implicit Language Reasoning in Human-Centric Environments

    cs.RO 2025-06 conditional novelty 6.0 of 10

    Narrate2Nav uses Barlow Twins alignment to distill language-based reasoning from a large teacher into a small RGB-only navigation model, reporting lower trajectory error and higher goal-reaching success than four baselines.

  2. From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Pretraining a navigation foundation model on videos and then fine-tuning only a residual attention module with reinforcement learning improves success rate and collision avoidance compared to behavior cloning or super...

Reference graph

Works this paper leans on

65 extracted references · 48 canonical work pages · cited by 2 Pith papers

  1. [1]

    Navigation through urban environments by visual perception and inter- action

    Quirin Muhlbauer, Stefan Sosnowski, Tingting Xu, Tian- guang Zhang, Kolja Kuhnlenz, and Martin Buss. Navigation through urban environments by visual perception and inter- action. In ICRA, pages 3558–3564. IEEE, 2009. 1, 2

    work page 2009

  2. [2]

    A navigation system for robots operating in crowded urban environments

    Rainer K ¨ummerle, Michael Ruhnke, Bastian Steder, Cyrill Stachniss, and Wolfram Burgard. A navigation system for robots operating in crowded urban environments. In ICRA, pages 3225–3232. IEEE, 2013

    work page 2013

  3. [3]

    Autonomous robot navigation in outdoor cluttered pedestrian walkways

    Yoichi Morales, Alexander Carballo, Eijiro Takeuchi, At- sushi Aburadani, and Takashi Tsubouchi. Autonomous robot navigation in outdoor cluttered pedestrian walkways. Jour- nal of Field Robotics, 26(8):609–635, 2009. 1, 2

    work page 2009

  4. [4]

    Habitat: A plat- form for embodied ai research

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A plat- form for embodied ai research. In CVPR, pages 9339–9347,

    work page

  5. [5]

    Gibson env: Real-world per- ception for embodied agents

    Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jiten- dra Malik, and Silvio Savarese. Gibson env: Real-world per- ception for embodied agents. In CVPR, pages 9068–9079, 2018

    work page 2018

  6. [6]

    Ai2-thor: An interactive 3d environment for visual ai

    Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474, 2017

    arXiv 2017

  7. [7]

    Matterport3d: Learning from RGB- D data in indoor environments

    Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Hal- ber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from RGB- D data in indoor environments. 3DV, 2017. 2

    work page 2017

  8. [8]

    Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra

    Erik Wijmans, Abhishek Kadian, Ari S. Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. In ICLR, 2019. 2

    work page 2019

Show all 65 references
  1. [9]

    Learning to navigate in cities without a map

    Piotr Mirowski, Matt Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Andrew Zisserman, Raia Hadsell, et al. Learning to navigate in cities without a map. NeurIPS, 31,

  2. [10]

    Cross-view policy learning for street navigation

    Ang Li, Huiyi Hu, Piotr Mirowski, and Mehrdad Farajtabar. Cross-view policy learning for street navigation. In ICCV, pages 8100–8109, 2019

    2019

  3. [11]

    Learning to follow directions in street view

    Karl Moritz Hermann, Mateusz Malinowski, Piotr Mirowski, Andras Banki-Horvath, Keith Anderson, and Raia Hadsell. Learning to follow directions in street view. In AAAI, vol- ume 34, pages 11773–11781, 2020

    2020

  4. [12]

    The streetlearn en- vironment and dataset

    Piotr Mirowski, Andras Banki-Horvath, Keith Anderson, Denis Teplyashin, Karl Moritz Hermann, Mateusz Mali- nowski, Matthew Koichi Grimes, Karen Simonyan, Koray Kavukcuoglu, Andrew Zisserman, et al. The streetlearn en- vironment and dataset. arXiv preprint arXiv:1903.01292 ,

    1903 arXiv

  5. [13]

    ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hints

    Dhruv Shah and Sergey Levine. ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hints. In RSS,

  6. [14]

    Gnm: A general navigation model to drive any robot

    Dhruv Shah, Ajay Sridhar, Arjun Bhorkar, Noriaki Hirose, and Sergey Levine. Gnm: A general navigation model to drive any robot. In ICRA, pages 7226–7233. IEEE, 2023. 2, 5, 6

    2023

  7. [15]

    Metaurban: A simulation platform for embodied ai in urban spaces.arXiv preprint arXiv:2407.08725, 2024

    Wayne Wu, Honglin He, Yiran Wang, Chenda Duan, Jack He, Zhizheng Liu, Quanyi Li, and Bolei Zhou. Metaurban: A simulation platform for embodied ai in urban spaces.arXiv preprint arXiv:2407.08725, 2024. 2

    2024 arXiv

  8. [16]

    Comet: Modeling group cohesion for socially compliant robot navigation in crowded scenes

    Adarsh Jagan Sathyamoorthy, Utsav Patel, Moumita Paul, Nithish K Sanjeev Kumar, Yash Savle, and Dinesh Manocha. Comet: Modeling group cohesion for socially compliant robot navigation in crowded scenes. IEEE Robotics and Au- tomation Letters, 7(2):1008–1015, 2021. 2

    2021

  9. [17]

    Move beyond trajectories: Distribution space cou- pling for crowd navigation

    Muchen Sun, Francesca Baldini, Peter Trautman, and Todd Murphey. Move beyond trajectories: Distribution space cou- pling for crowd navigation. In RSS, 2021

    2021

  10. [18]

    Au- tonomous mobile robot navigation independent of road boundary using driving recommendation map

    Yuya Onozuka, Ryosuke Matsumi, and Motoki Shino. Au- tonomous mobile robot navigation independent of road boundary using driving recommendation map. In IROS, pages 4501–4508. IEEE, 2021. 2

    2021

  11. [19]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 2, 3

    2005 arXiv

  12. [20]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In ICCV, pages 4015–4026, 2023

    2023

  13. [21]

    Maxime Oquab, Timoth ´ee Darcet, Th´eo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael ...

    2024

  14. [22]

    Palm- e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm- e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023. 2

    2023 arXiv

  15. [23]

    Vlm see, robot do: Human demo video to robot action plan via vision language model

    Beichen Wang, Juexiao Zhang, Shuwen Dong, Irving Fang, and Chen Feng. Vlm see, robot do: Human demo video to robot action plan via vision language model. arXiv preprint arXiv:2410.08792, 2024. 2, 3, 5

    2024

  16. [24]

    Lelan: Learning a language-conditioned navigation policy from in-the-wild videos

    Noriaki Hirose, Catherine Glossop, Ajay Sridhar, Dhruv Shah, Oier Mees, and Sergey Levine. Lelan: Learning a language-conditioned navigation policy from in-the-wild videos. arXiv preprint arXiv:2410.03603, 2024. 2, 3, 5

    2024 arXiv

  17. [25]

    Deep patch vi- sual odometry

    Zachary Teed, Lahav Lipson, and Jia Deng. Deep patch vi- sual odometry. NeurIPS, 36, 2024. 2, 3, 4 9

    2024

  18. [26]

    Reinforcement learning with unsupervised auxiliary tasks

    Max Jaderberg, V olodymyr Mnih, Wojciech Marian Czar- necki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In ICLR, 2017. 2

    2017

  19. [27]

    Cognitive mapping and plan- ning for visual navigation

    Saurabh Gupta, James Davidson, Sergey Levine, Rahul Suk- thankar, and Jitendra Malik. Cognitive mapping and plan- ning for visual navigation. In CVPR, pages 2616–2625,

  20. [28]

    Semi-parametric topological memory for navigation

    Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. In ICLR, 2018. 2

    2018

  21. [29]

    No rl, no simulation: Learning to navigate without navigating

    Meera Hahn, Devendra Singh Chaplot, Shubham Tulsiani, Mustafa Mukadam, James M Rehg, and Abhinav Gupta. No rl, no simulation: Learning to navigate without navigating. NeurIPS, 34:26661–26673, 2021

    2021

  22. [30]

    Visual graph memory with unsuper- vised representation for visual navigation

    Obin Kwon, Nuri Kim, Yunho Choi, Hwiyeon Yoo, Jeongho Park, and Songhwai Oh. Visual graph memory with unsuper- vised representation for visual navigation. In CVPR, pages 15890–15899, 2021

    2021

  23. [31]

    Nav- igating to objects specified by images

    Jacob Krantz, Theophile Gervet, Karmesh Yadav, Austin Wang, Chris Paxton, Roozbeh Mottaghi, Dhruv Batra, Jiten- dra Malik, Stefan Lee, and Devendra Singh Chaplot. Nav- igating to objects specified by images. In CVPR, pages 10916–10925, 2023. 2

    2023

  24. [32]

    Navigating to objects in the real world

    Theophile Gervet, Soumith Chintala, Dhruv Batra, Jitendra Malik, and Devendra Singh Chaplot. Navigating to objects in the real world. Science Robotics, 8(79):eadf6991, 2023. 2

    2023

  25. [33]

    Object goal nav- igation using goal-oriented semantic exploration

    Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Ab- hinav Gupta, and Russ R Salakhutdinov. Object goal nav- igation using goal-oriented semantic exploration. NeurIPS, 33:4247–4258, 2020

    2020

  26. [34]

    Zson: Zero-shot object-goal navigation using multimodal goal embeddings

    Arjun Majumdar, Gunjan Aggarwal, Bhavika Devnani, Judy Hoffman, and Dhruv Batra. Zson: Zero-shot object-goal navigation using multimodal goal embeddings. NeurIPS, 35:32340–32352, 2022. 2

    2022

  27. [35]

    Embodied question answer- ing

    Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answer- ing. In CVPR, pages 1–10, 2018. 2

    2018

  28. [36]

    Learning to nav- igate unseen environments: Back translation with environ- mental dropout

    Hao Tan, Licheng Yu, and Mohit Bansal. Learning to nav- igate unseen environments: Back translation with environ- mental dropout. arXiv preprint arXiv:1904.04195, 2019

    1904 arXiv

  29. [37]

    Beyond the nav-graph: Vision-and- language navigation in continuous environments

    Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Ba- tra, and Stefan Lee. Beyond the nav-graph: Vision-and- language navigation in continuous environments. In ECCV, pages 104–120. Springer, 2020. 2

    2020

  30. [38]

    Learn- ing to navigate sidewalks in outdoor environments

    Maks Sorokin, Jie Tan, C Karen Liu, and Sehoon Ha. Learn- ing to navigate sidewalks in outdoor environments. IEEE Robotics and Automation Letters, 7(2):3906–3913, 2022. 2, 5

    2022

  31. [39]

    Rethinking sim2real: Lower fidelity simulation leads to higher sim2real transfer in navigation

    Joanne Truong, Max Rudolph, Naoki Harrison Yokoyama, Sonia Chernova, Dhruv Batra, and Akshara Rai. Rethinking sim2real: Lower fidelity simulation leads to higher sim2real transfer in navigation. In CoRL, pages 859–870. PMLR, 2023

    2023

  32. [40]

    Learning navigation skills for legged robots with learned robot em- beddings

    Joanne Truong, Denis Yarats, Tianyu Li, Franziska Meier, Sonia Chernova, Dhruv Batra, and Akshara Rai. Learning navigation skills for legged robots with learned robot em- beddings. In IROS, pages 484–491. IEEE, 2021

    2021

  33. [41]

    X- mobility: End-to-end generalizable navigation via world modeling

    Wei Liu, Huihua Zhao, Chenran Li, Joydeep Biswas, Billy Okal, Pulkit Goyal, Yan Chang, and Soha Pouya. X- mobility: End-to-end generalizable navigation via world modeling. arXiv preprint arXiv:2410.17491, 2024. 2

    2024 arXiv

  34. [42]

    Badgr: An autonomous self-supervised learning-based navigation sys- tem

    Gregory Kahn, Pieter Abbeel, and Sergey Levine. Badgr: An autonomous self-supervised learning-based navigation sys- tem. IEEE Robotics and Automation Letters , 6(2):1312– 1319, 2021. 2

    2021

  35. [43]

    Land: Learning to navigate from disengagements

    Gregory Kahn, Pieter Abbeel, and Sergey Levine. Land: Learning to navigate from disengagements. IEEE Robotics and Automation Letters, 6(2):1872–1879, 2021. 2

    2021

  36. [44]

    ViNT: A foundation model for visual navigation

    Dhruv Shah, Ajay Sridhar, Nitish Dashora, Kyle Stachowicz, Kevin Black, Noriaki Hirose, and Sergey Levine. ViNT: A foundation model for visual navigation. In CoRL, 2023. 2, 5, 6, 8, 1

    2023

  37. [45]

    Nomad: Goal masked diffusion policies for navi- gation and exploration

    Ajay Sridhar, Dhruv Shah, Catherine Glossop, and Sergey Levine. Nomad: Goal masked diffusion policies for navi- gation and exploration. In ICRA, pages 63–70, 2024. 2, 5, 6

    2024

  38. [46]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, pages 8748–8763. PMLR, 2021. 3

    2021

  39. [47]

    Learning gener- alizable robotic reward functions from” in-the-wild” human videos

    Annie S Chen, Suraj Nair, and Chelsea Finn. Learning gener- alizable robotic reward functions from” in-the-wild” human videos. arXiv preprint arXiv:2103.16817, 2021. 3

    2021 arXiv

  40. [48]

    Human- to-robot imitation in the wild

    Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. Human- to-robot imitation in the wild. In RSS, 2022. 3

    2022

  41. [49]

    Learning to imitate object interactions from internet videos

    Austin Patel, Andrew Wang, Ilija Radosavovic, and Jitendra Malik. Learning to imitate object interactions from internet videos. arXiv preprint arXiv:2211.13225, 2022. 3

    2022 arXiv

  42. [50]

    Selfd: Self-learning large-scale driving policies from the web

    Jimuyang Zhang, Ruizhao Zhu, and Eshed Ohn-Bar. Selfd: Self-learning large-scale driving policies from the web. In CVPR, pages 17316–17326, 2022. 3

    2022

  43. [51]

    Gpt-4v (ision) for robotics: Multimodal task planning from human demonstra- tion

    Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, and Katsushi Ikeuchi. Gpt-4v (ision) for robotics: Multimodal task planning from human demonstra- tion. IEEE Robotics and Automation Letters, 2024. 3

    2024

  44. [52]

    Direct sparse odometry

    Jakob Engel, Vladlen Koltun, and Daniel Cremers. Direct sparse odometry. TPAMI, 40(3):611–625, 2017. 4

    2017

  45. [53]

    Orb-slam: a versatile and accurate monocular slam system

    Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics , 31(5):1147–1163, 2015

    2015

  46. [54]

    Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras

    Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. NeurIPS, 34:16558–16569, 2021. 4

    2021

  47. [55]

    Metric-Free Exploration for Topological Mapping by Task and Motion Imitation in Feature Space

    Yuhang He, Irving Fang, Yiming Li, Rushi Bhavesh Shah, and Chen Feng. Metric-Free Exploration for Topological Mapping by Task and Motion Imitation in Feature Space. In RSS, July 2023. 5

    2023

  48. [56]

    Self-supervised learning from images with a joint-embedding predictive architecture

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bo- janowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In CVPR, pages 15619–15629, 2023. 10

    2023

  49. [57]

    Revisiting feature prediction for learn- ing visual representations from video

    Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learn- ing visual representations from video. arXiv preprint arXiv:2404.08471, 2024. 5

    2024 arXiv

  50. [58]

    Convoi: Context-aware naviga- tion using vision language models in outdoor and indoor en- vironments

    Adarsh Jagan Sathyamoorthy, Kasun Weerakoon, Mohamed Elnoor, Anuj Zore, Brian Ichter, Fei Xia, Jie Tan, Wenhao Yu, and Dinesh Manocha. Convoi: Context-aware naviga- tion using vision language models in outdoor and indoor en- vironments. arXiv preprint arXiv:2403.15637, 2024. 5

    2024 arXiv

  51. [59]

    Fast-lio: A fast, robust lidar-inertial odometry package by tightly-coupled iterated kalman fil- ter

    Wei Xu and Fu Zhang. Fast-lio: A fast, robust lidar-inertial odometry package by tightly-coupled iterated kalman fil- ter. IEEE Robotics and Automation Letters, 6(2):3317–3324,

  52. [60]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017. 5

    2017

  53. [61]

    Sacson: Scalable autonomous control for social nav- igation

    Noriaki Hirose, Dhruv Shah, Ajay Sridhar, and Sergey Levine. Sacson: Scalable autonomous control for social nav- igation. IEEE Robotics and Automation Letters, 2023. 6

    2023

  54. [62]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 1

    2010 arXiv

  55. [63]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  56. [64]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2019. 2

    2019

  57. [65]

    Learning to drive by watching youtube videos: Action-conditioned contrastive policy pretraining

    Qihang Zhang, Zhenghao Peng, and Bolei Zhou. Learning to drive by watching youtube videos: Action-conditioned contrastive policy pretraining. In ECCV, pages 111–128. Springer, 2022. 2 11 Appendix A. Details on Data, Model, and Training City Walking Videos. We source our traini...

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.