pith. sign in

arxiv: 2606.20458 · v1 · pith:POPJ7VWFnew · submitted 2026-06-18 · 💻 cs.RO

Slow Brain, Fast Planner: Latency-Resilient VLM-Augmented Urban Navigation

Zhenghao "Mark'' Peng , Honglin He , Quanyi Li , Yukai Ma , Bolei Zhou This is my paper

Pith reviewed 2026-06-26 17:14 UTC · model grok-4.3

classification 💻 cs.RO
keywords vision-language modeltrajectory planningurban navigationscore fusionlatency resiliencemobile robotsidewalk navigationADE reduction
0
0 comments X

The pith

A slow VLM picks the best trajectory from a fast planner's candidates, then a geometric fusion layer converts the choice into real-time scores despite 1-3 second delays.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that learning-based planners often pick poor trajectories in hard sidewalk scenes even when better options sit in the same candidate set. Rather than swap the entire planner for a slow vision-language model, the authors add a VLM that simply names the best index and then fuse that signal with the planner's output. A training-free layer uses geometric similarity plus exponential decay to turn a stale VLM answer into usable real-time scores. On roughly two thousand real-world challenging cases the VLM choice cuts average displacement error by thirty percent while the original planner stays competitive on easy routes; the fused system keeps over eighty percent success even when the VLM lags up to five seconds. The full pipeline runs on a physical robot across campus sidewalks with varying network delays.

Core claim

The authors establish that a VLM-Planner interface closes the trajectory scoring gap by letting the VLM select an index from the planner's real-time proposal set and then applying a latency-resilient fusion layer; this combination yields thirty percent lower ADE than the planner's own top choice across two thousand difficult real scenes while preserving the planner's speed and routine performance, and the fused scores sustain over eighty percent success under simulated delays up to five seconds.

What carries the argument

The training-free latency-resilient trajectory-level fusion layer that converts a stale VLM index selection into real-time planner scores using geometric similarity with exponential decay.

If this is right

  • VLM selection reduces ADE by thirty percent versus the planner's best choice on two thousand challenging real-world sidewalk scenarios.
  • The original planner stays competitive on routine navigation tasks.
  • Score fusion keeps success above eighty percent in simulation for delays reaching five seconds.
  • The complete system runs on a physical mobile robot across varied campus sidewalks with real network latency.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same index-selection-plus-fusion pattern could let future faster or cheaper VLMs plug into existing planners without retraining either component.
  • The geometric-decay fusion might apply to any setting where a slow high-level selector must correct a fast low-level generator that already produces diverse candidates.
  • If VLM accuracy on index selection improves, the thirty-percent gain could grow without changes to the fusion layer itself.

Load-bearing premise

The VLM can correctly name the single best trajectory index from the planner's set in difficult real scenes and that geometric similarity with exponential decay is enough to make a delayed selection useful for live scoring.

What would settle it

A direct measurement of how often the VLM actually selects the trajectory that produces the lowest collision or deviation cost when the robot later executes it, or a trial showing success rate drop below eighty percent at delays shorter than five seconds.

Figures

Figures reproduced from arXiv: 2606.20458 by Bolei Zhou, Honglin He, Quanyi Li, Yukai Ma, Zhenghao "Mark'' Peng.

Figure 1
Figure 1. Figure 1: Slow Brain, Fast Planner. A fast planner generates dynamically feasible candidates and a slow VLM selects among them. Score Fusion blends the stale VLM choice into the planner’s real-time scoring via geometric similarity with exponential decay, en￾abling continuous control. Closing this scoring gap requires a source of high-level scene understanding. Vision￾Language Models (VLMs) are a good candidate: they… view at source ↗
Figure 2
Figure 2. Figure 2: VLM selection performance. The planner is better on normal and the VLM outperforms on hard. Slow Loop (VLM). A VLM asynchronously receives a visual query: the current camera image with K candidate trajectories overlaid as colored polylines. The VLM outputs a selected index k ⋆ ∈ {1, . . . , K}, representing its judgment of which trajectory best satisfies the navigation goal and social norms. This call take… view at source ↗
Figure 3
Figure 3. Figure 3: Horizon-aware similarity for delayed VLM guidance. The stale VLM trajectory is motion-compensated into the current frame; arclength progress s0 defines the remaining horizon over which similarity is computed (Sec. 3.3). horizon-aware similarity ( [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablations on VLM trajectory selection (1,412 hard scenarios). (A) Candidate gen￾eration: score-based Top-K vs. geometric diversification. (B) Candidate set size K. (C) Prompt content: goal, scores, and history frames [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: (A) At a junction, the VLM picks tra￾jectory 14 along the sidewalk curve. (B) On a routine straight, the VLM applies near-identical reasoning but again picks 14, while the planner’s argmax 7 was better. 1                                  *&       ( &&%' ( &&%' )& * ! ' # & $ "" %… view at source ↗
Figure 7
Figure 7. Figure 7: Real-world deployment. (A) Robot platform. We use a four-wheeled delivery robot on a campus sidewalk shared with pedestrians. (B) Real-world VLM input. The actual overlay the VLM sees: the front-camera frame with the trajectory overlay and a pedestrian is directly ahead. (C) VLM reasoning trace. The natural-language reason field by VLM. variants and Qwen2.5-VL-72B. GPT-5 models show weaker performance, and… view at source ↗
Figure 8
Figure 8. Figure 8: VLM request–response latency during real-world deployment (Probability Fusion + Streaming, Gemini 2.5 Flash Lite over 4G cellular). Each dot is one VLM response; the y-axis shows the wall-clock delay between request submission and response receipt. The median latency is ≈2.7 s (min 1.7 s, max 4.5 s). The variation is driven by network jitter rather than model inference time. All responses arrive well withi… view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative examples: when VLM selection helps most. Four scenarios where the VLM outperforms the planner’s argmax, with JSON outputs showing VLM reasoning. Top-left: sidewalk centering (VLM selects trajectory 38 over planner’s 8; 2.5 m ADE improvement). Top-right: night navigation near curb and wall (trajectory 32 vs. planner’s 10; 2.3 m improvement). Bottom-left: crosswalk approach with a car visible (tr… view at source ↗
Figure 10
Figure 10. Figure 10: Trajectory-selection prompt (system/user text) and example outputs. The actual request includes the overlay image in addition to this text. SYSTEM PROMPT You are a navigation assistant controlling a ground robot . Robot footprint : assume the robot is 0.80 m wide ( use this as the clearance envelope ) . At each step , a local planner proposes multiple candidate trajectories ( shown in an overlay image ). … view at source ↗
read the original abstract

Learning-based planners for sidewalk navigation can generate diverse candidate trajectories in real time, yet their scoring functions often fail to select the best trajectory in challenging situations, outputting trajectories that make the mobile robot drive onto grass, toward pedestrians, or in the wrong direction, even when better candidates exist in the same set. We call this the trajectory scoring gap: in real-world sidewalk navigation, the gap between an anchor-based planner's top choice and the best possible candidate is substantial, likely due to limited high-level scene understanding capability of the planner. Rather than replacing the planner with an end-to-end Vision-Language-Action model, we propose a VLM-Planner interface that uses a VLM to select a candidate index from the planner's proposal set and then fuse it with the planner's initial output. However, VLMs take 1--3s per query and so cannot directly drive a 5--20Hz control loop. We contribute a training-free, latency-resilient trajectory-level fusion layer that turns a stale VLM selection into real-time planner scoring via geometric similarity with exponential decay. On $\sim$2,000 challenging real-world scenarios (e.g., junctions, pedestrian encounters), VLM selection achieves 30% ADE reduction versus the planner's best selection, while the planner remains competitive in routine situations. In simulation, Score Fusion maintains >80% success rate with delays up to 5s. We demonstrate the full system on a mobile robot navigating challenging campus sidewalks with varied network latency.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper proposes a hybrid VLM-Planner interface for sidewalk navigation in which a slow VLM selects an index from a fast planner's candidate trajectories and a training-free Score Fusion layer converts the stale selection into real-time scoring via geometric similarity combined with exponential decay. It claims that VLM selection yields a 30% ADE reduction versus the planner's best choice on ~2000 challenging real-world scenarios while the planner remains competitive in routine cases, that Score Fusion sustains >80% success rate in simulation under delays up to 5 s, and that the full system runs on a physical mobile robot.

Significance. If the quantitative claims hold after clarification, the work offers a practical, training-free bridge between high-level VLM reasoning and low-latency control that avoids end-to-end replacement of existing planners. The real-robot demonstration and explicit handling of network latency are concrete strengths that could influence hybrid perception-planning architectures.

major comments (3)
  1. [Abstract and §4] Abstract and §4 (Evaluation): the central quantitative claim of a 30% ADE reduction is presented without any description of the baseline selection methods, the precise ADE formula, error bars, or how the ~2000 scenarios were constructed and labeled, which directly undermines assessment of whether the data support the trajectory-scoring-gap hypothesis.
  2. [§3.3] §3.3 (Score Fusion): the exponential-decay fusion assumes that geometric similarity to the originally selected index remains a reliable proxy for safety after 1–5 s of delay; no analysis or counter-example experiments are provided for cases in which moving agents invalidate the stale index, leaving the latency-resilience claim dependent on untested scene-dynamics assumptions.
  3. [Simulation results] Simulation results paragraph: the >80% success rate under delay is reported without stating the number of trials, the dynamic-agent models used, or the failure-mode distribution, making it impossible to judge whether the fusion layer generalizes beyond the paper's specific simulation conditions.
minor comments (2)
  1. [Notation] Define ADE explicitly on first use and state whether it is computed only on the selected trajectory or averaged over the full set.
  2. [Ablation] Add a short table or paragraph contrasting the proposed fusion against a simple “use last VLM index” baseline to isolate the contribution of the geometric-decay term.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. We address each major comment below with clarifications drawn from the evaluation setup and will revise the paper to improve transparency on quantitative claims, fusion assumptions, and simulation details.

read point-by-point responses

  1. Referee: [Abstract and §4] Abstract and §4 (Evaluation): the central quantitative claim of a 30% ADE reduction is presented without any description of the baseline selection methods, the precise ADE formula, error bars, or how the ~2000 scenarios were constructed and labeled, which directly undermines assessment of whether the data support the trajectory-scoring-gap hypothesis.

    Authors: The baseline is the planner's highest-scoring trajectory from its internal scoring function (without VLM input). ADE is the average L2 displacement error between the selected trajectory and the ground-truth human-driven path over the 3-second horizon. The ~2000 scenarios were extracted from real-world sidewalk logs collected on a mobile robot, filtered to challenging cases (junctions, pedestrian encounters) where the planner's top choice deviated from ground truth by more than a threshold. We will add the exact ADE formula, error bars from 5-fold cross-validation on the scenario set, and a description of scenario construction and labeling criteria to the revised §4 and abstract. revision: yes

  2. Referee: [§3.3] §3.3 (Score Fusion): the exponential-decay fusion assumes that geometric similarity to the originally selected index remains a reliable proxy for safety after 1–5 s of delay; no analysis or counter-example experiments are provided for cases in which moving agents invalidate the stale index, leaving the latency-resilience claim dependent on untested scene-dynamics assumptions.

    Authors: Score Fusion weights the stale VLM index by a combination of geometric similarity (trajectory overlap) and exponential decay on time elapsed. The simulation experiments already incorporate dynamic agents (pedestrians with constant-velocity motion) and report sustained success rates under 1–5 s delays, providing indirect support for the proxy. We acknowledge the lack of dedicated counter-example analysis for rapidly changing scenes and will add a limitations paragraph in §3.3 discussing the assumption and its dependence on moderate scene dynamics. revision: partial

  3. Referee: [Simulation results] Simulation results paragraph: the >80% success rate under delay is reported without stating the number of trials, the dynamic-agent models used, or the failure-mode distribution, making it impossible to judge whether the fusion layer generalizes beyond the paper's specific simulation conditions.

    Authors: The reported >80% success rate is averaged over 500 independent simulation trials per delay setting. Dynamic agents are modeled as constant-velocity pedestrians with randomized initial positions and speeds drawn from real-world sidewalk statistics. Failure modes are collisions with agents or deviation beyond 1 m from the reference path. We will expand the simulation paragraph with these details, a table of success rates by delay, and the failure-mode breakdown in the revised manuscript. revision: yes

Circularity Check

0 steps flagged

No significant circularity; method is a proposed heuristic evaluated empirically

full rationale

The paper proposes a training-free fusion layer using geometric similarity and exponential decay to handle VLM latency, without any derivation that reduces a claimed prediction or result back to its inputs by construction. No self-citations are load-bearing for the central claim, no parameters are fitted and then renamed as predictions, and the success rates are presented as simulation and real-world measurements rather than forced by the method definition itself. The approach relies on explicit geometric operations and empirical validation on ~2000 scenarios, making the derivation self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters, axioms, or invented entities can be identified from the abstract alone.

pith-pipeline@v0.9.1-grok · 5814 in / 1154 out tokens · 22525 ms · 2026-06-26T17:14:23.414034+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 7 linked inside Pith

  1. [1]

    Sridhar, D

    A. Sridhar, D. Shah, C. Glossop, and S. Levine. Nomad: Goal masked diffusion policies for navigation and exploration. In2024 IEEE International Conference on Robotics and Automa- tion (ICRA), pages 63–70. IEEE, 2024

    2024

  2. [2]

    D. Shah, A. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine. Vint: A foundation model for visual navigation.arXiv preprint arXiv:2306.14846, 2023

    arXiv 2023

  3. [3]

    H. He, Y . Ma, W. Wu, and B. Zhou. From seeing to experiencing: Scaling navigation founda- tion models with reinforcement learning.arXiv preprint arXiv:2507.22028, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Z. Li, W. Yao, Z. Wang, X. Sun, J. Chen, N. Chang, M. Shen, J. Song, Z. Wu, S. Lan, and J. M. Alvarez. Ztrs: Zero-imitation end-to-end autonomous driving with trajectory scoring.arXiv preprint arXiv:2510.24108, 2025

    arXiv 2025

  5. [5]

    J. Song, Z. Li, S. Lan, X. Sun, N. Chang, M. Shen, J. Chen, K. A. Skinner, and J. M. Alvarez. Drivecritic: Towards context-aware, human-aligned evaluation for autonomous driving with vision-language models.arXiv preprint arXiv:2510.13108, 2025

    arXiv 2025

  6. [6]

    S. Shi, L. Jiang, D. Dai, and B. Schiele. Motion transformer with global intention localiza- tion and local movement refinement. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  7. [7]

    Kirby, A

    E. Kirby, A. Boulch, Y . Xu, Y . Yin, G. Puy, ´E. Zablocki, A. Bursuc, S. Gidaris, R. Marlet, F. Bartoccioni, A.-Q. Cao, N. Samet, T.-H. Vu, and M. Cord. Driving on registers.arXiv preprint arXiv:2601.05083, 2026

    arXiv 2026

  8. [8]

    A. Li, Z. Wang, J. Zhang, M. Li, Y . Qi, Z. Chen, Z. Zhang, and H. Wang. Urbanvla: A vision- language-action model for urban micromobility.arXiv preprint arXiv:2510.23576, 2025

    arXiv 2025

  9. [9]

    Zitkovich, T

    B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR, 2023

    2023

  10. [10]

    Driess, F

    D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Nasiriany, F

    S. Nasiriany, F. Xia, W. Yu, T. Xiao, J. Liang, I. Dasgupta, A. Xie, D. Driess, A. Wahid, Z. Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. InInternational Conference on Machine Learning, pages 37321–37341. PMLR, 2024

    2024

  13. [13]

    D. Song, J. Liang, X. Xiao, and D. Manocha. Vl-tgs: Trajectory generation and selection using vision language models in mapless outdoor environments.IEEE Robotics and Automation Letters, 10(6), 2025

    2025

  14. [14]

    Cheng, Y

    A.-C. Cheng, Y . Ji, Z. Yang, Z. Gongye, X. Zou, J. Kautz, E. Bıyık, H. Yin, S. Liu, and X. Wang. Navila: Legged robot vision-language-action model for navigation.arXiv preprint arXiv:2412.04453, 2024

    arXiv 2024

  15. [15]

    M. Wei, C. Wan, J. Peng, X. Yu, Y . Yang, D. Feng, W. Cai, C. Zhu, T. Wang, J. Pang, and X. Liu. Ground slow, move fast: A dual-system foundation model for generalizable vision- and-language navigation.arXiv preprint arXiv:2512.08186, 2025. 10

    arXiv 2025

  16. [16]

    Internvla-n1: An open dual-system vision-language navigation foun- dation model with learned latent plans

    InternVLA-N1 Team. Internvla-n1: An open dual-system vision-language navigation foun- dation model with learned latent plans. Technical Report, Shanghai AI Laboratory, 2025. https://internrobotics.github.io/internvla-n1.github.io/

    2025

  17. [17]

    T. Zou, H. Zeng, Y . Nong, Y . Li, K. Liu, H. Yang, X. Ling, X. Li, and L. Ma. Asynchronous fast-slow vision-language-action policies for whole-body robotic manipulation.arXiv preprint arXiv:2512.20188, 2025

    arXiv 2025

  18. [18]

    H. Chen, J. Liu, C. Gu, Z. Liu, R. Zhang, X. Li, X. He, Y . Guo, C.-W. Fu, S. Zhang, et al. Fast- in-slow: A dual-system foundation model unifying fast manipulation within slow reasoning. arXiv preprint arXiv:2506.01953, 2025

    arXiv 2025

  19. [19]

    D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine. Gnm: A general navigation model to drive any robot. In2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023

    2023

  20. [20]

    W. Cai, J. Peng, Y . Yang, Y . Zhang, M. Wei, H. Wang, Y . Chen, T. Wang, and J. Pang. Navdp: Learning sim-to-real navigation diffusion policy with privileged information guidance.arXiv preprint arXiv:2505.08712, 2025

    arXiv 2025

  21. [21]

    Hwang, J

    M. Hwang, J. Hejna, D. Sadigh, and Y . Bisk. Motif: Motion instruction fine-tuning.IEEE Robotics and Automation Letters, 2025. arXiv:2409.10683

    arXiv 2025

  22. [22]

    D. Song, J. Liang, A. Payandeh, A. H. Raj, X. Xiao, and D. Manocha. Vlm-social-nav: Socially aware robot navigation through scoring using vision-language models.IEEE Robotics and Automation Letters, 2024

    2024

  23. [23]

    Y . Wu, R. Tian, G. Swamy, and A. Bajcsy. From foresight to forethought: Vlm-in-the-loop policy steering via latent alignment.arXiv preprint arXiv:2502.01828, 2025

    arXiv 2025

  24. [24]

    X. Liu, J. Li, Y . Jiang, N. Sujay, Z. Yang, J. Zhang, J. Abanes, J. Zhang, and C. Feng. City- walker: Learning embodied urban navigation from web-scale videos. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 6875–6885, 2025

    2025

  25. [25]

    B. Han, J. Kim, and J. Jang. A dual process vla: Efficient robotic manipulation leveraging vlm. arXiv preprint arXiv:2410.15549, 2024

    arXiv 2024

  26. [26]

    Helix: A vision-language-action model for generalist humanoid control.https: //www.figure.ai/news/helix, 2025

    Figure AI. Helix: A vision-language-action model for generalist humanoid control.https: //www.figure.ai/news/helix, 2025

    2025

  27. [27]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al.π 0.5: A vision-language-action model with open-world generalization.arXiv preprint arXiv:2504.16054, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Gr00t n1: An open foundation model for generalist humanoid robots

    NVIDIA GEAR Team. Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025

    Pith/arXiv arXiv 2025

  29. [29]

    Y . Zhao, L. Zhao, B. Cheng, G. Yao, X. Wen, and H. Gao. Vla-rail: A real-time asynchronous inference linker for vla models and robots.arXiv preprint arXiv:2512.24673, 2025

    arXiv 2025

  30. [30]

    J. Tang, Y . Sun, Y . Zhao, S. Yang, Y . Lin, Z. Zhang, J. Hou, Y . Lu, Z. Liu, and S. Han. Vlash: Real-time vlas via future-state-aware asynchronous inference.arXiv preprint arXiv:2512.01031, 2025

    arXiv 2025

  31. [31]

    M. Wei, C. Wan, X. Yu, T. Wang, Y . Yang, X. Mao, C. Zhu, W. Cai, H. Wang, Y . Chen, X. Liu, and J. Pang. Streamvln: Streaming vision-and-language navigation via slowfast context modeling.arXiv preprint arXiv:2507.05240, 2025. 11

    arXiv 2025

  32. [32]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al.π 0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    X. Tian, J. Gu, B. Li, Y . Liu, Y . Wang, Z. Zhao, K. Zhan, P. Jia, X. Lang, and H. Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models.arXiv preprint arXiv:2402.12289, 2024. 12 A VLM Interface for Trajectory Selection A.1 Interface Overview Our VLM doestrajectory selectionrather than low-level control: at each step, a f...

    Pith/arXiv arXiv 2024

  34. [34]

    ascoreq m ∈[0,1](softmax-normalized), representing the model’s confidence that anchormis the best behavioral mode for the current situation

  35. [35]

    aregression trajectoryτ m: a sequence of(x, y)waypoints (normalized offsets from the anchor), forming a 4 s, 20-waypoint polyline in the robot frame

  36. [36]

    reach the goal

    avelocity scalev m, converting the normalized trajectory into metric coordinates. The result is 64 candidate trajectories, each with an associated planner score. In our pipeline, we select the top-Kcandidates by score (defaultK=18) for presentation to the VLM (see ablation in the main paper, Fig. 4(B)). Why anchor-based design matters for our method.The a...

    1920

  37. [37]

    Scene u n d e r s t a n d i n g first : i de nti fy si dew al k / path vs grass / dirt / planters , and any p e d e s t r i a n s / o b s t a c l e s

  38. [38]

    HARD REJECT any t r a j e c t o r y that goes onto grass / off - limits surface or too close to a p e d e s t r i a n / ob sta cl e

  39. [39]

    Only among the r e m a i n i n g safe options , use goal geo me try ( pr og re ss / angle ) as a tie - breaker

  40. [40]

    action

    If you are unsure whether a t r a j e c t o r y stays on si dew al k ( a m b i g u o u s ) , choose a safer option or stop . If none look safe , choose stop . If the image is missing / unclear or you cannot decide , choose stop . S o m e t i m e s a very short t r a j e c t o r y i n d i c a t e s a ’ stop - like ’ action . In that case , prefer r e t u r...