Pith. sign in

REVIEW 4 major objections 6 minor 23 references

YOPO-Rally: A Sim-to-Real Single-Stage Planner for Off-Road Terrain

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A depth-image-only network trained by imitation in a randomized forest simulator plans safe off-road trajectories at 10 Hz and transfers zero-shot to a real robot.

desk verdict A working zero-shot sim-to-real off-road planner with a useful new simulator, but the cost-label supervision in the BC loss is undefined—fixable and worth review. read the letter →

arxiv 2505.18714 v1 pith:XEN37NPK submitted 2025-05-24 cs.RO

Hongyu Cao , Junjie Lu , Xuewei Zhang , Yulin Hui , Zhiyu Li , Bailing Tian This is my paper
classification cs.RO
keywords off-roadnavigationsim-to-realtransfersingle-stageplanningbehaviorcloningterraintraversabilityanalysisdepth-imagetrajectoryoptimizationforest
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that off-road navigation for a small ground robot can be reduced to a single forward pass of a neural network. The network consumes a depth image, the robot's velocity, and the goal direction, and produces several candidate trajectories with costs, fusing traversability analysis and pathfinding into one learned module. Training uses only simulated expert trajectories obtained by optimizing a hand-crafted cost map in a randomized forest simulator; the resulting policy transfers directly to real forest terrain without fine-tuning. If true, this makes fast, low-compute off-road planning practical with just a depth camera, avoiding the latency and error propagation of modular TTA-plus-search pipelines.

What carries the argument

The load-bearing object is the set of $M_\theta$ motion-primitive anchors arranged across the camera's horizontal field of view. For each anchor $p_i = r[\cos\theta_i, \sin\theta_i]^T$ the network regresses an end-state offset $(p_n, p_\theta)$, an end velocity, and a cost; the lowest-cost anchor is selected and converted into a non-uniform cubic Hermite curve $p(t)$ whose boundary conditions match the vehicle's start and end states. Expert demonstrations are produced by a trajectory optimizer (Eq. 8) inside a cone constraint around each anchor, so behavior cloning can learn the implicit traversability analysis without ever computing a cost map at run time.

What would settle it

Find a real or simulated patch of terrain whose slope, roughness, and distance-to-obstacle values are all low but which is nonetheless untraversable (for example, deep mud or a hidden ditch), and check whether the planner assigns it a low cost and drives into it; if it does, the cost map's blind spot propagates through the behavior-cloned policy.

Watch

Extended reading notes

Core claim

The central claim is that Terrain Traversability Analysis and pathfinding can be collapsed into a single regression problem. YOPO-Rally inputs a depth image, current velocity, and goal vector, and outputs $M_\theta$ trajectory candidates with associated costs in one forward pass; the planner then picks the cheapest candidate and hands the resulting Hermite curve to an MPC controller. Behavior cloning on simulator-generated expert trajectories suffices: in a real forest with a stereo depth camera, the planner ran at 10 Hz with roughly 25 ms inference on an embedded computer and kept an average safety distance of 1.35 m (minimum 0.4 m) while reaching 1.6 m/s, with no fine-tuning after simulation training.

Load-bearing premise

The expert trajectories the planner imitates are optimal with respect to a hand-written cost map of slope, roughness, and obstacle distance; if that map does not capture what actually makes off-road terrain unsafe, the learned planner inherits the blind spot.

Editorial extensions

If this is right

  • Planning latency drops from about 2.66 s for the compared modular Gaussian-process planner to 7.26 ms in simulation, and real-world inference is about 25 ms on an embedded processor.
  • Depth-only input sidesteps the color-image sim-to-real gap, so a policy trained on simulated depth can be deployed on a real stereo camera without adaptation.
  • Because the network outputs a cost for every motion-primitive anchor, the planner keeps multiple distinct trajectory hypotheses and can select among them, preserving multi-modality in cluttered forests.
  • Behavior cloning from cost-map-optimal expert trajectories is sufficient to learn traversability-aware planning, so no reinforcement-learning reward shaping or real-world data collection is required.
  • The MPC controller at 20 Hz can track the 10 Hz planned Hermite trajectories within the differential-drive model, giving smooth velocity profiles.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the hand-crafted cost map mislabels terrain, the planner will confidently imitate the mislabel; a natural next step is to test whether replacing the cost map with learned traversability labels or human demonstrations improves safety on untrained surface types.
  • The single-stage architecture is sensor-agnostic in principle: the same primitive-anchor regression could accept LiDAR or fused depth and would likely inherit the same zero-shot transfer, because only the input distribution changes.
  • The cost outputs per anchor could serve as a learned proposal distribution for a sampling-based global planner, potentially resolving long-horizon local minima that the fixed planning horizon $r$ cannot see.
  • A quantitative ablation of the weights $\lambda_r, \lambda_s, \lambda_c$ in Eq. (7) would directly measure how much planner safety depends on the expert cost model; the paper does not report such an ablation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents YOPO-Rally, a single-stage neural planner for off-road forest navigation that takes a depth image, current velocity, and goal vector as input and outputs a set of Hermite-curve trajectory candidates with associated costs, selecting the lowest-cost primitive at inference. The planner is trained by behavior cloning on expert trajectories generated in a custom Unity-based simulator, YOPO-Sim, by minimizing a hand-crafted traversability cost map (slope, roughness, and obstacle safety terms). The authors claim zero-shot transfer from simulation to real forest terrain without fine-tuning, and support this with simulated comparisons against GP-Nav plus two real-world runs on a DIABLO robot. The manuscript also evaluates simulator sensor rates and voxelization speed against AirSim, Flightmare, and CARLA.

Significance. If the central claims stand, the paper offers a useful engineering contribution: it collapses traversability analysis and path selection into a single forward pass at 10 Hz, and the reported simulated latency advantage over GP-Nav (7.26 ms versus 2661 ms total planning time) is substantial. The use of cone constraints in the trajectory optimization to generate per-primitive expert labels is an elegant mechanism for avoiding label assignment. The zero-shot real-world deployment on a Jetson Orin NX with depth-only input is also a nontrivial demonstration. However, the validation is currently weaker than the claims: the cost head in the behavior-cloning loss has no defined supervision target, the real-world experiments have no baseline and only two runs, and the simulator's fidelity is not quantified against real depth data. These gaps are load-bearing for the zero-shot transfer claim, so I recommend a major revision.

major comments (4)
  1. [Section II-B, Eq. (11); Section II-A2, Eq. (4)] The expert cost label c* is never defined. The loss in Eq. (11) supervises all components of y from Eq. (4), including the per-primitive cost c, and the inference rule in Section II-A2 selects the final trajectory as the one with minimum predicted cost. The paper states only that trajectory optimization generates expert trajectories via the objective J_t in Eq. (8), but it does not explicitly state that the expert cost c* is the optimized J_t value, nor how that continuous value is normalized to the [0, cmax] range used in Eq. (4). Without a definition of c*, the cost head has no well-defined training target and the trajectory-selection mechanism is unsupported. This must be stated explicitly, including the exact normalization procedure.
  2. [Section III-C, Table IV] The real-world validation consists of exactly two runs (Environments A and B) with no baseline, no repeated trials, and no error bars. Since the paper's central claim is zero-shot sim-to-real transfer for safe off-road navigation, this evidence is too thin. I recommend adding a real-world comparison against a modular baseline (for example, the TTA-plus-pathfinding approach represented by GP-Nav, or a cost-map-based planner) and reporting statistics over multiple repeated runs, including minimum and average safety distances with variance.
  3. [Section II-B, Eqs. (5)-(8)] The expert demonstrations are generated by minimizing a hand-crafted cost map whose parameters λ_r, λ_s, λ_c, Gsmax, Grmax, rdilate, d0, and k are introduced without calibration or sensitivity analysis. Because these expert trajectories are the only training signal for the planner, every one of these parameters directly shapes the learned policy. The paper should justify the chosen values and, ideally, include a sensitivity study showing that the planner's simulated and real-world behavior is robust to reasonable variations in these parameters.
  4. [Section II-B and Section III-C] The zero-shot transfer claim would be much better grounded by a direct quantitative analysis of the sim-to-real depth-image gap. The text asserts in Section II-B that 'there is a little discrepancy' between simulated and real-world depth images, but no supporting comparison is provided, such as depth-error statistics, intensity histograms, or feature-space distances between YOPO-Sim and OAK-D-Pro images. Given that the planner receives only a depth image as perception input, this gap analysis is central to the transfer claim and should be made explicit.
minor comments (6)
  1. [Eq. (1a)] There is a typo in the definition of p_s: it is written as p_s = [p_x^s, p_x^s]^T, but the second component should be p_y^s.
  2. [Eq. (2)] The index range 'i ∈ [0, Mθ]' is ambiguous and likely should be 'i = 0, 1, ..., Mθ-1' to denote Mθ primitives; please clarify.
  3. [Section II-A2] The number of primitives Mθ and the corresponding depth-image resolution 32Mθ × 32 are not specified in the experiments; please report Mθ and the actual input resolution used for training and deployment.
  4. [Section II-C, Eq. (13d)] The admissible control set U in Eq. (13d) is never defined; please specify the bounds on linear and angular velocity used in the MPC formulation.
  5. [Table III] For the proposed method, only the total latency is reported in the latency columns; please clarify whether the 7.26 ms figure is purely the neural-network inference time or includes all planning-related computation, to make the comparison with GP-Nav's component breakdown meaningful.
  6. [Section III-A and III-B] The simulator is described as 'high-fidelity,' but the evaluation measures only sensor update rates and voxelization speed; either temper the fidelity claim or add quantitative fidelity checks, such as comparing simulated depth images against real OAK-D-Pro depth images in terms of noise statistics and range accuracy.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the planner is trained by behavior cloning on externally generated expert trajectories and tested zero-shot on a real robot; the unspecified cost-label source is a completeness gap, not a circular step.

full rationale

The derivation chain is not circular. TTA builds a cost map from terrain geometry (Eqs. 5-7); trajectory optimization minimizes J_t with respect to that map (Eqs. 8-10); the resulting expert trajectories supervise the network through the BC loss (Eq. 11); at inference the network selects the minimum predicted cost and is evaluated on held-out simulated environments and on a real DIABLO robot. Each stage is independently computable: the cost map is derived from point-cloud geometry, the expert trajectories are genuine optima of J_t, and the zero-shot claim is tested against external metrics (safety distance, trajectory length, latency) in physical forest experiments. The only ambiguity is that c*, the cost component of the expert label y* used in Eq. (11), is never explicitly defined in the paper. If c* is taken from the optimized J_t, then the cost head is supervised by the same objective that generated the expert trajectories, which is standard behavior cloning rather than circular reasoning; if c* is not taken from J_t, then the target is undefined, which is a completeness/correctness gap, not a circularity. The citation to the authors' prior YOPO work [15] is used only for the primitive-anchor representation and is consistent with the equations developed in this paper; it is not load-bearing, does not import a uniqueness theorem, and does not smuggle in an ansatz on which the central result depends. The hand-tuned weights in Eq. (7) are a modeling assumption and a correctness risk, but they do not make the derivation circular. Overall, no claimed prediction reduces by construction to its own input, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entities. Its load-bearing commitments are the hand-tuned cost model, the BC generalization assumption, and the unspecified sim-to-real image similarity, all of which are domain assumptions rather than derivations.

free parameters (5)
  • Cost map weights (lambda_r, lambda_s, lambda_c)
    Hand-selected in Eq. (7); no sensitivity analysis or learning is reported, and they directly shape the expert trajectories and thus the learned policy.
  • TTA thresholds (Gsmax, Grmax)
    Hand-set in Eq. (5) to classify obstacles; no justification or ablation.
  • Safety distance parameters (rdilate, d0, k)
    Hand-chosen in Eq. (6) for dilation and exponential safety cost.
  • Output bounds (pnmax, pthetamax, vmax, cmax)
    Normalization constants in Eq. (4) that define the network output range and the cone constraint; no values are given.
  • Primitive count M_theta and horizon r
    Design choices in Section II-A1 that determine resolution and coverage; no analysis of their effect is provided.
assumptions (5)
  • domain assumption Slope and roughness thresholds define traversability in the DEM (Eq. 5).
    The paper assumes these geometric features are sufficient to identify obstacles in off-road terrain.
  • domain assumption The exponential safety cost with hand-set d0 and k is a valid risk model (Eq. 6).
    No empirical validation of the safety-cost shape is provided.
  • domain assumption Behavior cloning from a finite set of optimal trajectories yields a policy that generalizes to novel real environments.
    Standard BC assumption, not validated with distribution shift measures.
  • domain assumption Simulator depth images are close enough to OAK-D depth images for zero-shot transfer.
    Asserted in Sections I and II-B, supported only by a successful real-world run, not by quantitative domain-gap metrics.
  • standard math The Hermite curve representation and differential drive model are valid for this planner (Eqs. 1, 12).
    Standard robotics and geometry results; no further proof is needed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of YOPO-Rally: A Sim-to-Real Single-Stage Planner for Off-Road Terrain." pith.science (2026). https://pith.science/paper/XEN37NPK

@misc{pith2026250518714,
  author       = {Pith},
  title        = {Pith review of: YOPO-Rally: A Sim-to-Real Single-Stage Planner for Off-Road Terrain},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XEN37NPK}},
  note         = {Machine review of arXiv:2505.18714}
}
read the original abstract

Off-road navigation remains challenging for autonomous robots due to the harsh terrain and clustered obstacles. In this letter, we extend the YOPO (You Only Plan Once) end-to-end navigation framework to off-road environments, explicitly focusing on forest terrains, consisting of a high-performance, multi-sensor supported off-road simulator YOPO-Sim, a zero-shot transfer sim-to-real planner YOPO-Rally, and an MPC controller. Built on the Unity engine, the simulator can generate randomized forest environments and export depth images and point cloud maps for expert demonstrations, providing competitive performance with mainstream simulators. Terrain Traversability Analysis (TTA) processes cost maps, generating expert trajectories represented as non-uniform cubic Hermite curves. The planner integrates TTA and the pathfinding into a single neural network that inputs the depth image, current velocity, and the goal vector, and outputs multiple trajectory candidates with costs. The planner is trained by behavior cloning in the simulator and deployed directly into the real-world without fine-tuning. Finally, a series of simulated and real-world experiments is conducted to validate the performance of the proposed framework.

Figures

Figures reproduced from arXiv: 2505.18714 by the authors.

Figure 1
Figure 1. Visualization of the primitives and the predicted trajectories. The [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. System overview. The planner inputs the depth image and the state [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the cone constraint, cost map, and expert trajectories. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Snapshots of the YOPO-Sim. The insets show some of the sensors provided by the simulator: (a) depth camera and RGB camera, (b) LiDAR, and [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The DIABLO vehicle platform for the real-world experiment is [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: The simulated and real-world experimental views of the robot in different environments. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Trajectory comparisons in the simulated off-road forest environment [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Trajectories visualization of real-world experiments in forest terrain [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 12 canonical work pages

  1. [1]

    A survey on path planning for autonomous ground vehicles in unstructured environments,

    N. Wang, X. Li, K. Zhang, J. Wang, and D. Xie, “A survey on path planning for autonomous ground vehicles in unstructured environments,” Machines, vol. 12, no. 1, p. 31, 2024

    work page 2024

  2. [2]

    Challenges and solutions for autonomous ground robot scene understanding and navigation in unstructured outdoor environments: A review,

    L. Wijayathunga, A. Rassau, and D. Chai, “Challenges and solutions for autonomous ground robot scene understanding and navigation in unstructured outdoor environments: A review,”Applied Sciences, vol. 13, no. 17, p. 9877, 2023

    work page 2023

  3. [3]

    Hybrid map- based path planning for robot navigation in unstructured environments,

    J. Liu, X. Chen, J. Xiao, S. Lin, Z. Zheng, and H. Lu, “Hybrid map- based path planning for robot navigation in unstructured environments,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023, pp. 2216–2223

    work page 2023

  4. [4]

    Putn: A plane-fitting based uneven terrain navigation framework,

    Z. Jian, Z. Lu, X. Zhou, B. Lan, A. Xiao, X. Wang, and B. Liang, “Putn: A plane-fitting based uneven terrain navigation framework,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, pp. 7160–7166

    2022

  5. [5]

    History- aware planning for risk-free autonomous navigation on unknown uneven terrain,

    Y . Wang, N. Du, Y . Qin, X. Zhang, R. Song, and C. Wang, “History- aware planning for risk-free autonomous navigation on unknown uneven terrain,” in 2024 IEEE International Conference on Robotics and Au- tomation (ICRA) . IEEE, 2024, pp. 7583–7589

    work page 2024

  6. [6]

    Gaussian process-based traversability analysis for terrain mapless navigation,

    A. Leininger, M. Ali, H. Jardali, and L. Liu, “Gaussian process-based traversability analysis for terrain mapless navigation,” in 2024 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2024, pp. 10 925–10 931

    work page 2024

  7. [7]

    Neupan: Direct point robot navigation with end- to-end model-based learning,

    R. Han, S. Wang, S. Wang, Z. Zhang, J. Chen, S. Lin, C. Li, C. Xu, Y . C. Eldar, Q. Hao et al. , “Neupan: Direct point robot navigation with end- to-end model-based learning,” IEEE Transactions on Robotics , 2025

    work page 2025

  8. [8]

    Badgr: An autonomous self- supervised learning-based navigation system,

    G. Kahn, P. Abbeel, and S. Levine, “Badgr: An autonomous self- supervised learning-based navigation system,” IEEE Robotics and Au- tomation Letters , vol. 6, no. 2, pp. 1312–1319, 2021

    work page 2021

Show all 23 references
  1. [9]

    CARLA: An open urban driving simulator,

    A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V . Koltun, “CARLA: An open urban driving simulator,” in Proceedings of the 1st Annual Conference on Robot Learning , 2017, pp. 1–16

    2017

  2. [10]

    Airsim: High-fidelity visual and physical simulation for autonomous vehicles,

    S. Shah, D. Dey, C. Lovett, and A. Kapoor, “Airsim: High-fidelity visual and physical simulation for autonomous vehicles,” in Field and Service Robotics, 2017. [Online]. Available: https://arxiv.org/abs/1705.05065

    2017 arXiv

  3. [11]

    Sim-to-real transfer in deep reinforcement learning for robotics: a survey,

    W. Zhao, J. P. Queralta, and T. Westerlund, “Sim-to-real transfer in deep reinforcement learning for robotics: a survey,” in 2020 IEEE symposium series on computational intelligence (SSCI) . IEEE, 2020, pp. 737–744

    2020

  4. [12]

    Wroom: An autonomous driving approach for off-road navigation,

    D. Kalaria, S. Sharma, S. Bhagat, H. Xue, and J. M. Dolan, “Wroom: An autonomous driving approach for off-road navigation,”arXiv preprint arXiv:2404.08855, 2024

    2024 arXiv

  5. [13]

    Anycar to anywhere: Learning universal dynamics model for agile and adaptive mobility,

    W. Xiao, H. Xue, T. Tao, D. Kalaria, J. M. Dolan, and G. Shi, “Anycar to anywhere: Learning universal dynamics model for agile and adaptive mobility,” arXiv preprint arXiv:2409.15783 , 2024

    2024 arXiv

  6. [14]

    Anynav: Visual neuro-symbolic friction learning for off-road navigation,

    T. Fu, Z. Zhan, Z. Zhao, S. Su, X. Lin, E. T. Esfahani, K. Dantu, S. Chowdhury, and C. Wang, “Anynav: Visual neuro-symbolic friction learning for off-road navigation,” arXiv preprint arXiv:2501.12654 , 2025

    2025

  7. [15]

    You only plan once: A learning-based one-stage planner with guidance learning,

    J. Lu, X. Zhang, H. Shen, L. Xu, and B. Tian, “You only plan once: A learning-based one-stage planner with guidance learning,” IEEE Robotics and Automation Letters , vol. 9, no. 7, pp. 6083–6090, 2024

    2024

  8. [16]

    Ultralytics YOLO,

    G. Jocher, J. Qiu, and A. Chaurasia, “Ultralytics YOLO,” Jan. 2023. [Online]. Available: https://github.com/ultralytics/ultralytics

    2023

  9. [17]

    Gradient-based online safe trajectory generation for quadrotor flight in complex environments,

    F. Gao, Y . Lin, and S. Shen, “Gradient-based online safe trajectory generation for quadrotor flight in complex environments,” in 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, 2017, pp. 3681–3688

    2017

  10. [18]

    Cloudcompare,

    “Cloudcompare,” July 2024. [Online]. Available: https://www. cloudcompare.org/

    2024

  11. [19]

    Drake: Model-based design and verification for robotics,

    R. Tedrake and the Drake Development Team, “Drake: Model-based design and verification for robotics,” 2019. [Online]. Available: https://drake.mit.edu

    2019

  12. [20]

    CasADi – A software framework for nonlinear optimization and opti- mal control,

    J. A. E. Andersson, J. Gillis, G. Horn, J. B. Rawlings, and M. Diehl, “CasADi – A software framework for nonlinear optimization and opti- mal control,” Mathematical Programming Computation , vol. 11, no. 1, pp. 1–36, 2019

    2019

  13. [21]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101 , 2017

    2017 arXiv

  14. [22]

    Unitysensors,

    R. Tanaka and A. Harada, “Unitysensors,” 2025. [Online]. Available: https://github.com/Field-Robotics-Japan/UnitySensors

    2025

  15. [23]

    Flightmare: A flexible quadrotor simulator,

    Y . Song, S. Naji, E. Kaufmann, A. Loquercio, and D. Scaramuzza, “Flightmare: A flexible quadrotor simulator,” in Conference on Robot Learning. PMLR, 2021, pp. 1147–1157

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.