{"id":"54113964-0b64-49bb-9140-868dfc99662b","arxiv_id":"2412.03353","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"One neural policy, trained with contrastive and reconstruction losses on a privileged cube map, lets a quadruped with only a front depth camera perform omnidirectional stair climbing, crawling, and jumping in 3D terrain.","lead":"MOVE is a learning framework that lets a low-cost quadruped robot with a single front camera walk, climb, jump, and crawl in any direction, even when its view is blocked or noisy. It trains one neural network that combines the robot's own motion history with whatever the camera can see, inferring what is around it beyond the camera's field of view.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim of inferring surroundings beyond the field of view is not established: standard test-time inputs cannot encode side/back terrain unless it correlates with front depth or contact history, and Table I shows non-forward performance drops only slightly without contrastive learning.","rationale":"The reader's weakest assumption focused on sim-to-real fidelity of depth noise and terrain randomization. That is a reasonable concern, but it is not the most load-bearing point. Even if the simulation perfectly matched reality, the central claim of 'inferring surroundings beyond the field of view' is not demonstrated. The learning objective in Eq. 5-7 can only transfer information that exists in the standard input; for non-forward tasks, the front camera and short proprioceptive history are informationally insufficient to know side/back terrain before contact unless the terrain distribution creates strong correlations with forward geometry. The paper's own ablation data suggests most non-forward skill comes from a relatively simple proprioceptive policy, not from representation learning. This does not invalidate the engineering contribution—the real-world demos are impressive—but it means the paper's distinctive intellectual claim is currently supported only by indirect, task-level evidence. A targeted probe of the latent representation before first contact would settle whether PS-Net actually encodes unseen terrain information. Since the paper is already CONDITIONAL for other reasons, my read does not change the verdict, but the condition should include this representation-level test in addition to code release and statistical rigor.","tokens_in":9789,"tokens_out":4337,"duration_ms":47254,"concrete_test":"In Isaac Gym, deploy the trained MOVE policy on lateral stairs and blind-crawl tasks. Record the standard latent vector at each timestep before the robot first contacts the obstacle. Train a linear probe to predict the privileged cube-map faces (ct left/right/up/down from Eq. 3) from this latent. If probe accuracy is at chance, the latent carries no recoverable information about unseen directions, and the beyond-FOV inference claim is unsupported. Additionally, ablate by removing side/back obstacles while keeping all other terrain identical; if success rates and behavior are essentially unchanged, the policy is reacting to contact rather than inferring terrain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central novelty is that PS-Net (Eq. 5-7) lets a single policy infer terrain beyond the front camera's field of view, enabling lateral/backward stair climbing and blind crawling. At deployment, the standard input is just the current front depth image plus a 10-step proprioceptive history (Section III-A.2). For non-forward motion, the side/back terrain is not in the front camera at the decision time, and the history is too short to have seen it except through prior contact. A contrastive loss cannot create information that is absent from the inputs: it can only align the standard latent with the privileged cube map on the subset of information that is actually predictable from front depth/proprioceptive history. The paper provides no direct evidence that the latent encodes anything about unseen directions; it only shows task-level success. Moreover, Table I shows that removing contrastive learning (Ours w/o C.L) still yields 88.5% lateral stairs, 98.2% backward stairs, and 75.3% blind crawl, while adding it yields only modest gains (97.1%, 99.2%, 80.7%). This is consistent with the policy mostly reacting proprioceptively after contact, not inferring the environment ahead of time. The 'inference beyond FOV' claim is therefore load-bearing but unsupported, independent of sim-to-real noise fidelity.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes MOVE, a one-stage end-to-end reinforcement learning framework for a quadruped robot equipped only with a front-facing depth camera and a short proprioceptive history. The policy is trained in Isaac Gym with a pseudo-siamese representation learner (PS-Net) that combines supervised reconstruction losses (velocity, next proprioception, front depth) with a contrastive loss against privileged cube-map observations, and is then deployed zero-shot on a DEEP Robotics Lite3. The paper claims that a single policy, without manual skill switching, can perform forward tasks such as high/long jumping, stair climbing, and crawling, as well as non-forward tasks such as lateral and backward stair climbing and blind crawling, in both simulation and the real world. Ablations in Table I compare the full method against variants without contrastive learning, without reconstruction, without cross-attention, and against a PPO baseline, and real-world experiments are reported in Figs. 7-9.","tokens_in":10073,"tokens_out":5091,"duration_ms":51530,"significance":"If the central claims are established, MOVE would be a useful contribution to low-cost legged locomotion with egocentric vision: it demonstrates a single learned policy performing a diverse set of dynamic skills, including omnidirectional motion under visual disturbance, which is more than most prior egocentric-vision locomotion works show. The paper's strengths include extensive real-world deployment in varied outdoor conditions, ablations of the main representation-learning components, and a representation-collapse diagnostic in Fig. 5. The main risk is that the load-bearing claim of 'inferring surroundings beyond the field of view' is not directly evidenced, and the quantitative support currently rests on single-run simulation numbers with no error bars and an unspecified depth-noise model. The contribution is therefore promising but needs additional evidence before the perceptual-inference claim can be accepted.","major_comments":[{"comment":"It is not stated explicitly whether the numbers in Table I come from a single policy trained once on all environments jointly or from separately trained policies per skill. The text says 'a series of simulation environments' were designed and 'each policy' was tested, which leaves the no-manual-switching claim ambiguous. If each skill is trained independently, the claim of multi-skill integration in one network is not supported by the reported experiments. Please state clearly whether one set of weights was used for all rows, and if so, describe how the shared training environment is constructed.","section":"IV-A and Table I"},{"comment":"The central claim that PS-Net enables the policy to infer surroundings beyond the camera's field of view is not directly supported by the reported results. The deployment input consists only of the front depth image and the last H=10 proprioceptive steps, while the privileged cube map in Eq. (3) explicitly excludes the rear direction. For backward stair climbing, the terrain behind the robot is therefore neither in the standard input nor in the contrastive target, so the contrastive loss cannot inject that information. Table I shows that removing contrastive learning leaves backward stairs at 98.2% vs. 99.2%, lateral stairs at 88.5% vs. 97.1%, and blind crawl at 75.3% vs. 80.7%; these small margins are consistent with reactive, contact-based proprioceptive recovery rather than anticipatory inference. Please provide direct evidence of beyond-FOV encoding, for example by probing the latent z_c_t to decode side/back terrain properties or by constructing a test in which front depth is decorrelated from side/back geometry.","section":"III-A.2, Eq. (3), and Table I"},{"comment":"The sim-to-real robustness claim relies on a depth-noise model that is not specified. Section IV-A states only that 'patterned noise' was introduced into the simulated depth images, without giving the noise model, its parameters, or any calibration against the RealSense D435i used on the Lite3. The paper also reports no quantitative comparison of simulated versus real depth artifacts for the blind, reflective-glass, leaf-pile, or tall-grass conditions described in Fig. 9. Please specify the noise model and parameters, and preferably add ablations over noise levels or real-sensor depth statistics to justify the zero-shot transfer.","section":"IV-A and IV-C"},{"comment":"All simulation results in Table I are point estimates from what appears to be a single training run, despite the use of 1,000 robots per evaluation. There are no standard errors, no seeds, and no statistical comparisons, yet several ablation margins are small (e.g., lateral stairs 97.1% vs. 96.2% for Ours vs. Ours w/o C.A., and camera-offset traversal 98.0% vs. 98.0%). Please report means and variances over at least three seeds and indicate which differences are statistically meaningful; otherwise the qualitative ranking of ablations cannot be assessed.","section":"IV-B and Table I"}],"minor_comments":[{"comment":"The symbol 'Lconstrast' in Eq. (7) appears to be a typo for 'Lcontrast' as used in Eq. (5).","section":"III-B.3, Eq. (7)"},{"comment":"The caption describes the training as a combination of 'supervised and unsupervised learning method'; since the contrastive component uses paired standard and privileged observations, 'self-supervised' would be the more accurate term.","section":"Fig. 2 caption"},{"comment":"The robot bounding box is given as '610mm × 37mm × 445mm'; the middle dimension appears to be missing a digit, since a 37 mm width is implausible for the Lite3.","section":"IV-A"},{"comment":"The real-world quantitative results in Fig. 7 would be easier to interpret if the number of trials and the success criterion per scenario were reported, since the success-rate axis alone does not indicate the statistical strength of the comparison.","section":"IV-C and Fig. 7"}],"recommendation":"major_revision","confidential_remarks":"The paper reports an impressive real-world system and a plausible training framework, but the central perceptual-inference claim is currently under-evidenced, and the experimental reporting lacks the statistical detail needed to support the finer ablation claims. I would be willing to accept after the authors clarify the single-policy training protocol, provide direct evidence for beyond-FOV encoding, and specify the depth-noise model. The issues are substantive but addressable within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the real contribution here is the integrated one-stage framework that lets a Lite3 with a single front depth camera do a wide range of skills—jumps, stair climbs, crawls, omnidirectional movement—without manual switching, and the real-world demonstrations are genuinely good. The architecture (pseudo-siamese encoders, asymmetric cross-attention, reconstruction plus contrastive loss) is a reasonable engineering solution that I don't see in the prior work I know. The ablations in Table I are useful: they show the reconstruction term matters a lot for forward tasks, and contrastive learning gives smaller but real gains on non-forward and blind tasks.\n\nThe soft spot is the claim that the robot 'infers surroundings beyond its field of view.' The deployment inputs are just the current front depth image and a 10-step proprioceptive history. The contrastive loss aligns the standard latent with a privileged cube map (sides, up, down, feet), but it can only do so on the subset of information actually predictable from front depth plus history. For lateral or backward stair climbing, there's no direct evidence the latent encodes anything about the side/back terrain; the strong performance without contrastive learning (98.2% backward stairs, 88.5% lateral stairs) suggests the policy is mostly reacting after contact, which is fine for locomotion but does not support the stronger inference narrative. That said, the modest gains from contrastive learning on the blind crawl (75.3%→80.7%) and on lateral stairs (88.5%→97.1%) are real and suggest some benefit, so this is a wording/evidence gap, not a fatal flaw.\n\nOther weaknesses are typical of this area: no error bars or multiple seeds, no comparison against recent vision-aided locomotion methods (e.g., [5] and [12]), and the paper omits the reward function details, the noise model, and code. That limits reproducibility but does not undermine the basic result. The sim-to-real transfer is demonstrated widely—obstructed cameras, leaf piles, glass floors—which is solid evidence of robustness.\n\nWho should read this: anyone working on legged locomotion with egocentric vision. It deserves a serious referee; I'd ask for the missing details and a more careful framing of the inference claim, but the experimental work warrants publication. Send to review; conditional acceptance is the right call, not a full accept as-is.","headline":"A solid end-to-end framework for omnidirectional legged locomotion with limited vision; the real-world results are impressive, but the 'inference beyond FOV' claim is softer than advertised and the evaluation lacks statistical rigor.","tokens_in":10609,"tokens_out":2944,"would_cite":true,"duration_ms":28910,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single policy gives a camera-limited quadruped all-direction parkour.","keywords":["legged locomotion","omnidirectional locomotion","egocentric vision","sim-to-real transfer","reinforcement learning","contrastive representation learning","quadruped robot","depth camera"],"falsifier":"Run the trained policy on the same real quadruped with the depth camera fully taped over, on a set of lateral and backward stair-climbing trials, and compare success to the reported blind performance; if the robot fails to clear even a few steps without vision, the claim that proprioception alone can infer unseen surroundings would be refuted. A complementary check is to measure the real camera's depth error distribution on occluded scenes and compare it to the training noise pattern, since a large mismatch would break the sim-to-real premise.","tokens_in":9582,"feed_emoji":"🦿","tokens_out":7024,"duration_ms":69813,"temperature":0.7,"pith_summary":"This paper tries to show that a low-cost quadruped robot with only a front-facing depth camera and a short history of body and joint states can learn, in one end-to-end reinforcement-learning pass, to run a portfolio of 3D terrain skills in any direction: forward jumps up to 0.7 meters high and 0.9 meters long, stair climbing up to 0.25 meters, crawling under 0.2-meter obstacles, and the same skills laterally, backward, and even when the camera is completely blocked. The central argument is that one network can decide when sighted locomotion is reliable and when it should fall back on proprioceptive inference, without explicit mapping or manual skill switching. The authors support this with a pseudo-siamese representation learner, PS-Net, that couples the robot's real camera view to a privileged five-direction cube map available only in simulation, trained with reconstruction and contrastive losses. If correct, this would expand what cheap, egocentric-vision robots can do outdoors without a runtime voxel or elevation map.","feed_headline":"One policy gives a camera-limited quadruped all-direction parkour","feed_subtitle":"Reconstruction plus contrastive learning lets the robot infer terrain beyond its field of view, and it works in the real world.","key_machinery":"The central object is PS-Net, the pseudo-siamese representation learner: two encoders with similar but not shared weights. The standard input encoder processes proprioceptive history through an MLP and the front depth image through a CNN, fuses them with self-attention, and passes the result through a GRU to produce a latent vector. The surroundings encoder processes privileged information, including a cube map made of five egocentric depth images (forward, up, down, left, right) plus sparse foot-depth data, with a cross-attention module whose query comes from the privileged proprioceptive state, forcing the encoder to focus on visual structure. The two encoders are coupled by a mixed loss of supervised reconstruction (velocity, next proprioceptive state, front depth) and a stop-gradient contrastive similarity term; this is what lets a robot with a narrow front camera infer the shape of terrain beside, below, and around it. The cube map itself is the privileged representation that makes the inference possible, chosen to sample depth more uniformly than LiDAR-like projections and cheap to generate in simulation.","core_discovery":"The paper's central claim is that MOVE, a one-stage end-to-end learning framework, makes multi-skill omnidirectional legged locomotion possible with limited and noisy egocentric vision. The policy inputs are proprioceptive history and a single depth image, and a single neural network outputs the motor commands for all skills with no mode switching. The key idea is to train a pseudo-siamese representation learner (PS-Net) with asymmetric attention: one encoder sees only real-robot observations, while a structurally similar but non-weight-sharing encoder sees privileged simulation data including a five-face cube map and sparse foot-depth samples. A mixed loss of reconstruction terms (velocity, next proprioceptive state, front depth) plus a stop-gradient contrastive term makes the real-observation encoder learn to infer terrain beyond its field of view, so the robot can climb stairs laterally and backward, crawl with vision fully obstructed, and recover from sudden visual noise, while still using vision aggressively for forward jumps and high steps. The authors report zero-shot sim-to-real transfer on a low-cost quadruped with onboard inference, with quantitative success rates in simulation and qualitative demonstrations in real indoor and outdoor settings.","pith_inferences":["The method is not tied to a specific simulator or robot: the cube-map supervisor could be rendered in any reasonable physics simulator, so the pseudo-siamese recipe is a candidate template for other vision-limited mobile robots.","Because the contrastive term is what carries beyond-field-of-view inference, a direct stress test is to shrink the camera's field of view in simulation and measure how much performance degrades; the paper does not report such a sweep.","The paper's own limitation, that depth images lack semantic content, suggests the natural next increment is adding RGB supervision, which the asymmetric attention mechanism would likely carry over without redesigning the policy."],"forward_implications":["One neural network can span sighted parkour (0.7-meter-high jumps, 0.9-meter-long jumps, 0.25-meter stairs, 0.2-meter crawls) and blind or omnidirectional versions of similar skills, removing the need for explicit skill scheduling.","A simulation-only cube-map supervisor lets the standard encoder learn to infer terrain beyond its field of view without building a runtime map.","The mixed reconstruction and stop-gradient contrastive objective improves omnidirectional speed tracking on complex 3D terrain over either loss alone, and the asymmetric cross-attention mechanism prevents representation collapse during contrastive learning.","The same policy transfers zero-shot to a real quadruped with onboard inference, maintaining success in environments with leaf piles, tall grass, reflective glass, and complete camera obstruction.","Ablation results show degradation is graded: removing contrastive learning or reconstruction individually lowers success rates, and removing both drops forward long-jump success from about 99.6 percent to about 7 percent."],"supporting_citations":[{"why":"Supplies the PPO algorithm that optimizes the asymmetric actor-critic architecture.","marker":"[38]"},{"why":"Supplies the stop-gradient operation used in the contrastive loss so the representation does not collapse.","marker":"[39]"},{"why":"Provides the massively parallel reinforcement learning setup and terrain curriculum used to train the single-stage policy.","marker":"[40]"},{"why":"Provides the reward function and prior parkour formulation the paper adopts with minimal changes.","marker":"[7]"},{"why":"Defines the Lite3 quadruped hardware, including its onboard processor and front-facing depth camera, used for real-world deployment.","marker":"[41]"},{"why":"Exemplifies the hierarchical mapping-based perception approach that MOVE is designed to avoid, serving as a key baseline contrast.","marker":"[2]"}],"fun_headline_variants":["Legged robot with one camera navigates 3D terrain omnidirectionally","Single policy lets a robot see beyond its camera's view","Camera-limited quadruped climbs, crawls, and leaps in any direction","MOVE: one neural net for all-direction legged mobility with limited view","Inferring unseen terrain: robot parkour with minimal vision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole robustness story depends on the simulated depth images and terrain randomization matching the real camera and real outdoor environments closely enough, because the paper adds a hand-crafted patterned noise to training depth images without specifying or validating a quantitative noise model.","fun_headline_variants_meta":{"raw":{"variants":["Legged robot with one camera navigates 3D terrain omnidirectionally","Single policy lets a robot see beyond its camera's view","Camera-limited quadruped climbs, crawls, and leaps in any direction","MOVE: one neural net for all-direction legged mobility with limited view","Inferring unseen terrain: robot parkour with minimal vision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000604,"raw_usage":{"total_tokens":2848,"prompt_tokens":1003,"completion_tokens":1845,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":1751}},"tokens_in":619,"tokens_out":1845,"duration_ms":12340,"temperature":1.0,"reasoning_tokens":1751,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:29:16.351977+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained policy on the same real quadruped with the depth camera fully taped over, on a set of lateral and backward stair-climbing trials, and compare success to the reported blind performance; if the robot fails to clear even a few steps without vision, the claim that proprioception alone can infer unseen surroundings would be refuted. A complementary check is to measure the real camera's depth error distribution on occluded scenes and compare it to the training noise pattern, since a large mismatch would break the sim-to-real premise.","supporting_citations":[{"cited_title":"Exploring simple siamese representation learn- ing,","cited_arxiv_id":null,"evidence_quote":"Supplies the stop-gradient operation used in the contrastive loss so the representation does not collapse."},{"cited_title":"Deeprobotics lite3,","cited_arxiv_id":null,"evidence_quote":"Defines the Lite3 quadruped hardware, including its onboard processor and front-facing depth camera, used for real-world deployment."}],"review_version":1}