{"id":"abc6b956-7b1c-4778-9910-feb5eff9a77c","arxiv_id":"2411.17820","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Navigation policies trained on 2,000+ hours of web videos with visual odometry pseudo-labels achieve higher real-world urban navigation success than fine-tuned prior models.","lead":"CityWalker trains a navigation model on over 2,000 hours of YouTube city walking and driving videos, using visual odometry to turn those videos into training actions without manual labels. The result is a policy that, after a small amount of fine-tuning, outperforms prior navigation models on real-world sidewalk tests with a quadruped robot.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training pipeline is underspecified: Section 3.3 explains only action labels from VO, while Fig. 2/Eq. 4 also require target-coordinate and arrival labels; without specifying how these are derived from web videos, the central real-world success claim is not reproducible as written.","rationale":"The reader's weakest assumption identifies the same load-bearing gap: the paper never specifies the target-coordinate and arrival-status supervision required by Figure 2 and Eq. 4. I agree this is the most fundamental issue, because the central claim depends on training a policy with the complete loss on unlabeled web videos; if those labels have an undocumented source, the method is not reproducible as written and the scalability/no-annotation claim is undercut. I do not think this warrants rejection, because the missing information may be a simple omission (e.g., target = future VO pose at fixed lookahead, arrival = distance below threshold) and the empirical results, while not definitive, directionally support the headline numbers. The reader's conditional verdict remains appropriate: the paper should be accepted only if the authors supply the missing supervision rule or code, and ideally also report confidence intervals or more real-world trials for the Table 2 success-rate gaps. No stronger concern emerged from the full text: the feature-hallucination inconsistency is acknowledged, the ablations honestly show marginal gains, and the offline evaluation uses independent teleoperation ground truth.","tokens_in":15862,"tokens_out":4030,"duration_ms":40310,"concrete_test":"Request or inspect the released data-processing code (project homepage or revision) and verify how, for each training clip, the target coordinate is selected and the arrival label is set. Then reconstruct these labels from DPVO VO trajectories on a 100-hour subset using the stated rule, confirm that L_arr and the coordinate token are computable for every sample, and retrain a small model on that subset to check that the offline MAOE/arrival trend in Figure 6 is reproduced. If the target/arrival labels cannot be derived from the VO poses and no other source is documented, the training pipeline described in Section 3 is incomplete and the real-world results lack a reproducible mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that training on unlabeled web videos with VO pseudo-labels yields the navigation policy that achieves 77.3% real-world success in Table 2. But the described training procedure cannot be executed as written. In Figure 2, each sample includes a \"Target Coord.\" token and an arrival head supervised by L_arr in Eq. 4. Section 3.4 states that the coordinate input is \"stacked by k past positions and one single target position,\" and the loss includes BCE for \"predicted arriving status.\" Section 3.3, however, only describes extracting relative poses from DPVO and normalizing action step lengths; it never states how the target waypoint w_t or the binary arrival label is obtained for internet-sourced walking/driving videos. These labels are not free: the target could be a future VO pose at a fixed distance, and arrival could be a distance threshold, but no such rule is given. Without this, Eq. 4 cannot be evaluated on the purported training data, so the model whose performance is reported in Tables 1 and 2 is not fully specified. This is load-bearing because the scalability argument rests on \"no costly annotations,\" and any unstated label source would have to be manual, map-based, or heuristic, weakening that claim. The gap is likely fixable by a precise description or code release, so it does not by itself refute the empirical results, but it blocks full acceptance of the method as described.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"CityWalker proposes a waypoint-goal visual navigation policy trained via imitation learning on 2000+ hours of unlabeled city walking and driving videos. Action supervision is extracted from video with a visual odometry model (DPVO) that yields relative poses, normalized per trajectory. The model uses a frozen DINOv2 encoder, a coordinate encoder, and a transformer with action and arrival prediction heads; training combines L1, orientation, arrival (BCE), and feature-hallucination losses. Evaluation on held-out NYC teleoperation data and in real-world deployment on a Unitree Go1 quadruped reports 77.3% success versus 57.1% for fine-tuned ViNT and 42.9% for zero-shot NoMaD (Table 2). The paper also presents data-scaling experiments and ablations.","tokens_in":16190,"tokens_out":8320,"duration_ms":67100,"significance":"If the training pipeline is fully specified, this work would be a significant demonstration of scalable imitation learning from internet video for embodied urban navigation. The paper's strengths include the large-scale data collection, real-world deployment on a quadruped, and honest ablations that acknowledge the marginal contributions of the orientation and feature-hallucination losses. The main weakness is the underspecification of the target-waypoint and arrival labels, which is essential for reproducing the central claim of 'no costly annotations'.","major_comments":[{"comment":"The training pipeline requires a target waypoint coordinate and a binary arrival label for every sample, but the paper never specifies how these labels are derived for the unlabeled web videos. Section 3.3 describes only how action labels are obtained from VO relative poses and step-length normalization. Without a concrete rule (e.g., the target is the VO pose at a fixed distance ahead, and arrival is a distance threshold), Eq. (4) cannot be evaluated on the purported training data. This is load-bearing because the scalability claim rests on avoiding costly annotations; an unstated heuristic or manual step would weaken that claim. A precise description or code release would resolve this gap.","section":"Sec. 3.4, Eq. (4), Fig. 2"},{"comment":"The real-world success rates are presented without the number of trials per condition and without confidence intervals or significance testing. The text says '8-14 trails for each cases,' which is ambiguous, and the total number of trials is not reported. Since Table 2 is the main evidence for the headline claim of outperforming existing methods, the paper should report trial counts per condition and ideally confidence intervals or a statistical test; otherwise the 77.3% versus 57.1% gap may not be robust given the small sample.","section":"Sec. 4.2, Table 2"}],"minor_comments":[{"comment":"The 'Arrival (%)' metric is reported for all methods, but GNM, ViNT, and NoMaD do not have an arrival prediction head; the paper does not specify how arrival is determined for these baselines. Please clarify the protocol used to compute arrival for methods without an arrival head.","section":"Sec. 4.2, Table 1"},{"comment":"The heading 'Depolyment' should be 'Deployment', and '8-14 trails' should be '8-14 trials'.","section":"Sec. 4.2"},{"comment":"The ablation shows that orientation loss and feature hallucination loss provide only marginal improvements (17.03 to 17.00 and 17.00 to 17.02); the text does acknowledge this, but the caption and surrounding discussion could be more explicit about the lack of a statistically meaningful effect.","section":"Sec. 4.4, Table 3"},{"comment":"The hyperparameter table uses 'Cord.' and 'φ' while the main text uses 'Coord.' and 'ω'; please unify the notation across the paper and appendix.","section":"Appendix A, Table II"},{"comment":"The figure legend and axis labels are hard to read; the text refers to markers (+) and (x) that are not clearly visible in the printed version. Larger fonts or a separate legend would help.","section":"Fig. 6"},{"comment":"The definition of 'Turn' via φaction > 20° appears to measure the action angle relative to a global axis, not the change in direction between consecutive actions; please clarify whether this is the intended operationalization.","section":"Sec. 4.1"}],"recommendation":"major_revision","confidential_remarks":"The central methodological gap is the missing description of target-waypoint and arrival-label generation for web videos. I would encourage the editor to ask for a precise rule or the release of training code, as this is essential for reproducibility and for the 'no costly annotations' claim. Additionally, the real-world trial counts should be reported; with 8-14 trials per condition, the reported success-rate advantage may not be statistically significant. The empirical results themselves are promising and the ablation study is transparent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper does something real. It takes 2000+ hours of unlabeled YouTube walking and driving videos, uses visual odometry to pseudo-label actions, pretrains a ViNT-style transformer, then fine-tunes on 6 hours of teleoperated quadruped data. The payoff is a 77.3% real-world success rate on a Unitree Go1 versus 57.1% for fine-tuned ViNT and 42.9% for zero-shot NoMaD. The scaling curve in Figure 6, showing zero-shot performance improving with more video hours and crossing the fine-tuned ViNT baseline, is the most convincing part. Cross-domain mixing (walking + driving) helping more than either alone is also a nice, non-obvious finding. The paper is honest about the feature hallucination loss hurting in zero-shot and helps only after fine-tuning; they don't hide it, and the explanation is plausible.\n\nThe soft spot is the one the stress-test flags, and it's real. Section 3.3 explains only how action labels come from VO. But the training pipeline in Figure 2 and Eq. 4 also require a target coordinate and an arrival label for every unlabeled web video. The paper never says where those come from. It's likely the target is a future VO pose at a fixed distance and the arrival label is a distance threshold, but that is not written down. That omission blocks reproduction as-is, and it undercuts the 'no costly annotations' claim if the actual procedure needs some unstated heuristic. This is fixable with one paragraph or a code release, so it's not fatal, but it is load-bearing. The evaluation would also benefit from confidence intervals; 8–14 trials per real-world case is not a lot, and the gap between 77.3% and 57.1% is meaningful but noisy.\n\nI disagree with the reader on one point: the shared angular quantity between the orientation loss and the MAOE metric is not circular, because the offline evaluation uses LiDAR-SLAM ground truth, not the VO pseudo-labels. And the missing pipeline detail, while annoying, does not refute the empirical results—they could easily be correct.\n\nBottom line: this deserves a serious referee. The method is novel in combination, the real-world transfer is credible, and the scaling evidence is worth having. I'd send it to review and ask the authors to specify the label generation for web videos, add error bars, and ideally release code. For a navigation or robotics audience, this is a useful read.","headline":"CityWalker is a genuinely useful result in embodied urban navigation, with real-world quadruped trials and data-scaling evidence, but the training pipeline has a load-bearing gap: target waypoint and arrival labels for web videos are never specified.","tokens_in":16753,"tokens_out":2035,"would_cite":true,"duration_ms":20319,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a navigation policy trained on 2,000+ hours of web walking and driving videos, labeled only by noisy visual odometry, reaches a 77.3% real-world success rate in urban navigation, beating fine-tuned baselines.","keywords":["urban navigation","imitation learning","web-scale video","visual odometry pseudo-labels","point-goal navigation","embodied agents","data scaling","cross-domain transfer"],"falsifier":"Reproduce the data pipeline on a subset of the sourced web videos: if target coordinates and arrival labels cannot be recovered from the visual-odometry trajectories alone (or from another described source), the training procedure as written cannot produce the reported policy. A complementary check is to rerun the real-world trials with more than the reported 8-14 runs per maneuver and with a more accurate position source, since the paper itself notes sensitivity to GPS noise; the claimed 20-point success gap over fine-tuned ViNT should survive those conditions.","tokens_in":15652,"feed_emoji":"🚶","tokens_out":11173,"duration_ms":87410,"temperature":0.7,"pith_summary":"CityWalker claims that an embodied agent can learn to navigate dense urban environments by imitating the motion in thousands of hours of unlabeled web video of people walking and driving through cities. A data pipeline turns noisy visual-odometry estimates of the camera's movement into action labels, so no manual annotation or language-model prompting is needed. After light fine-tuning on six hours of teleoperated quadruped data, the policy navigates unseen city routes with a 77.3% real-world success rate, against 57.1% for the best fine-tuned baseline and 42.9% for a zero-shot baseline. The paper further shows that performance rises with training-data hours and that mixing driving video with walking video accelerates those gains, which matters because urban navigation has resisted both simulation-trained and teleoperation-only policies.","feed_headline":"2,000+ hours of web video teach urban navigation at 77.3% success","feed_subtitle":"Noisy odometry labels from walking and driving footage beat prior navigation models in real-world tests.","key_machinery":"The load-bearing mechanism is the conversion of raw web video into imitation-learning supervision: visual odometry (DPVO) gives relative poses between frames, and each action is normalized by the average step length of its trajectory, folding the scale ambiguity of monocular odometry and the different stride lengths of walking versus driving into one abstract action space. The learned policy is a transformer that ingests past frame tokens, a history of positions, and the target waypoint, and emits a five-step action horizon together with an arrival prediction. Its training combines an orientation loss (negative cosine similarity between predicted and ground-truth actions), an L1 action loss, a binary arrival loss, and a feature-hallucination loss that penalizes the gap between predicted future tokens and the actual future frame tokens, pressing the model to anticipate what it will see next. The design bet is that volume and diversity of data compensate for the noise in odometry-derived labels.","core_discovery":"The paper's central claim is that noisy pseudo-labels from off-the-shelf visual odometry are sufficient supervision to learn urban navigation from in-the-wild video. Deep patch visual odometry (DPVO) extracts relative camera poses between nearby frames of 2,000+ hours of city walking and driving footage, and each action is normalized by the average step length of its trajectory so that scale differences between walking and driving disappear. The CityWalker policy is a transformer that consumes frozen DINOv2 features of past frames, past positions, and a target waypoint, and predicts five future actions plus an arrival flag; it is trained with an orientation loss, an L1 action loss, an arrival BCE loss, and a feature-hallucination loss that asks the transformer to predict future image tokens. Fine-tuned on six hours of teleoperated Unitree Go1 data in New York City, it reaches 77.3% real-world success versus 57.1% for fine-tuned ViNT and 42.9% for zero-shot NoMaD, with the largest gains in turns, crossings, crowds, and proximity situations; its own ablations attribute most of the gain to data scale and fine-tuning rather than the auxiliary losses. The paper's conclusion is that abundant online video can substitute for expensive expert teleoperation in building robust urban navigation policies.","pith_inferences":["A natural reading of the pipeline is that target waypoints are future positions along the VO trajectory and arrival labels come from detecting when forward motion stops; if so, the whole training procedure is annotation-free, but the paper does not state this explicitly.","If the scaling curve holds, training on a far larger mixed corpus (tens of thousands of hours across more cities) is a direct next experiment, and the saturation point of web-video imitation for urban navigation has yet to be found.","The real-world result rests on 8-14 trials per maneuver type in a limited set of neighborhoods; a larger multi-city trial is the implied test of whether the 20-point success gap over ViNT persists.","Since actions are stored in a step-length-normalized space, the same policy should transfer to other embodiments such as wheeled delivery robots by changing only the denormalization constant, an experiment the paper motivates but does not run."],"forward_implications":["A navigation policy fine-tuned with just six hours of expert data after web-video pretraining reaches 77.3% real-world success, versus 57.1% for fine-tuned ViNT and 42.9% for zero-shot NoMaD.","The zero-shot web-trained model already matches or beats fine-tuned baselines on offline arrival accuracy (84.1% versus 70.7%), implying pretraining on diverse web video can substitute for in-domain expert data.","Performance scales with data: beyond roughly 1,000 training hours the zero-shot model surpasses a fine-tuned ViNT, and 250 hours of mixed walking-plus-driving video nearly matches 1,000 hours of walking video alone.","Because the label pipeline is parallelizable visual odometry rather than language-model prompting, scaling to larger video corpora is cheap, keeping the approach cost-effective.","Driving video transfers to a quadruped roughly as well as zero-shot baselines, so cross-domain and cross-embodiment video is a usable source of navigation supervision."],"supporting_citations":[{"why":"Supplies the deep patch visual odometry used to turn web videos into noisy relative-pose action labels, the core of the training pipeline.","marker":"[25]"},{"why":"GNM is a baseline that must be beaten and the source of the step-length action normalization the pipeline adopts.","marker":"[14]"},{"why":"ViNT is the main comparison baseline, the architectural basis of the transformer policy, and the source of the PD controller used in deployment.","marker":"[44]"},{"why":"NoMaD is the zero-shot diffusion-policy baseline compared in offline and real-world evaluations.","marker":"[45]"},{"why":"DINOv2 is the frozen image encoder that supplies visual features to the policy.","marker":"[21]"},{"why":"FAST-LIO provides the LiDAR-inertial ground-truth poses used for the teleoperation fine-tuning data and offline evaluation labels.","marker":"[59]"},{"why":"LeLaN is the concurrent VLM-prompting approach that the paper's visual-odometry labeling is contrasted against.","marker":"[24]"}],"fun_headline_variants":["CityWalker learns urban navigation from web videos, 77.3% success","2,000+ hours of web video train CityWalker to navigate cities","Noisy video odometry teaches robots to walk in cities, beating prior models","CityWalker: web-scale video training for robust urban navigation","77.3% real-world success via imitation from web walking videos"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every training sample requires a target waypoint coordinate and an arrival-status label, but the paper never explains where these come from for the unlabeled web videos, whose only extracted labels are camera-motion estimates.","fun_headline_variants_meta":{"raw":{"variants":["CityWalker learns urban navigation from web videos, 77.3% success","2,000+ hours of web video train CityWalker to navigate cities","Noisy video odometry teaches robots to walk in cities, beating prior models","CityWalker: web-scale video training for robust urban navigation","77.3% real-world success via imitation from web walking videos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000173,"raw_usage":{"total_tokens":1295,"prompt_tokens":981,"completion_tokens":314,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":220}},"tokens_in":597,"tokens_out":314,"duration_ms":3419,"temperature":1.0,"reasoning_tokens":220,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:48:01.130326+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce the data pipeline on a subset of the sourced web videos: if target coordinates and arrival labels cannot be recovered from the visual-odometry trajectories alone (or from another described source), the training procedure as written cannot produce the reported policy. A complementary check is to rerun the real-world trials with more than the reported 8-14 runs per maneuver and with a more accurate position source, since the paper itself notes sensitivity to GPS noise; the claimed 20-point success gap over fine-tuned ViNT should survive those conditions.","supporting_citations":[{"cited_title":"Deep patch vi- sual odometry","cited_arxiv_id":null,"evidence_quote":"Supplies the deep patch visual odometry used to turn web videos into noisy relative-pose action labels, the core of the training pipeline."},{"cited_title":"Gnm: A general navigation model to drive any robot","cited_arxiv_id":null,"evidence_quote":"GNM is a baseline that must be beaten and the source of the step-length action normalization the pipeline adopts."},{"cited_title":"ViNT: A foundation model for visual navigation","cited_arxiv_id":null,"evidence_quote":"ViNT is the main comparison baseline, the architectural basis of the transformer policy, and the source of the PD controller used in deployment."},{"cited_title":"Nomad: Goal masked diffusion policies for navi- gation and exploration","cited_arxiv_id":null,"evidence_quote":"NoMaD is the zero-shot diffusion-policy baseline compared in offline and real-world evaluations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DINOv2 is the frozen image encoder that supplies visual features to the policy."},{"cited_title":"Fast-lio: A fast, robust lidar-inertial odometry package by tightly-coupled iterated kalman fil- ter","cited_arxiv_id":null,"evidence_quote":"FAST-LIO provides the LiDAR-inertial ground-truth poses used for the teleoperation fine-tuning data and offline evaluation labels."}],"review_version":1}