REVIEW 3 major objections 4 minor 26 references
The paper shows that a robot using only onboard computing can match the natural-language instruction parsing of a top cloud API and turn that parse into a metric goal with about a tenth of the error, completing 31 of 32 outdoor navigation t
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 05:23 UTC pith:QMGHOF4P
load-bearing objection Solid, honest systems paper showing fully onboard VLN is feasible; the SLM-vs-cloud result holds up, but the 0.20 m MAE headline needs qualification because it drops the two traffic-cone failures. the 3 major comments →
Offline Vision-Language Navigation with Geometric Goal Localization for Outdoor Environments
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Edge-BehAV establishes that behavior-guided vision-language navigation can run entirely onboard: the Qwen2.5-7B model decomposes navigation instructions into landmarks, navigation actions, behavioral actions, targets, and costs as accurately as the strongest cloud API (Macro-F1 0.775 vs 0.765, overlapping confidence intervals) at roughly a ninth of the latency. The paper's second claim is that a semantic-goal estimate becomes a metric-goal estimate by gating LiDAR points with a VLM-produced object mask and taking an IQR-filtered median centroid, reducing mean absolute distance error from 2.05 m (visual heuristic) to 0.20 m. The third claim is that these pieces compose: the complete onboard s
What carries the argument
The central mechanism is the LiDAR-visual goal-latching pipeline: Florence-2 detects a bounding box for the target, Mobile-SAM segments it, LiDAR points are projected through extrinsics into the mask, and an IQR-filtered median of planar distances yields range and bearing, latched as a world-frame goal (Eq. 3-4). A sigma-switch decides between Point-Goal mode (valid median centroid with at least 7 points) and Bearing-Only mode (visual bearing from pixel center), and a Stop-and-Measure state machine stops the robot, refreshes image and scan at rest, and latches the goal. This hybrid cost and mode-switching structure is what converts semantic detection into a reliable metric goal for the downs
Load-bearing premise
Goal localization accuracy rests on the assumption that the median LiDAR range inside the segmentation mask is the true target position, which requires the mask to cover only the target, at least seven LiDAR returns on it, and no occlusion of its centroid; the paper's own distance table shows this fails for a traffic cone at 10 and 20 m.
What would settle it
Run the distance benchmark counting every attempted measurement (including the '—' cone cases) as an error; if mean and worst-case errors then exceed 0.20 m substantially, the latching assumption is not general. Alternatively, a paraphrased-instruction test set that drops Qwen2.5-7B's Macro-F1 outside the confidence interval versus GPT-5.5 would show the language match is benchmark-specific.
If this is right
- Navigation instruction decomposition can be served by a locally quantized 7B model, removing multi-second cloud round-trips and enabling operation in connectivity-denied areas.
- Semantic goals like 'the orange cone' become metric waypoints with roughly 0.20 m mean error, making language-guided goals directly usable by traditional planners.
- The bearing-only fallback keeps the robot advancing toward a target even when LiDAR cannot produce a centroid, so localization failures degrade gracefully rather than halting the mission.
- Because quality is nearly platform-independent under INT4 quantization, model selection done on a laptop transfers to embedded hardware like the Jetson Orin NX.
- The 31/32 success rate in disambiguation scenarios indicates the main bottleneck in prior systems was goal estimation, not language understanding or obstacle avoidance.
Where Pith is reading between the lines
- The 0.20 m mean error is reported over valid measurements only; counting the traffic-cone failures at 10 and 20 m as misses gives a more conservative worst-case picture for small, low-reflectance targets.
- The stop-and-measure latching likely explains why closed-loop goal-distance error (0.01-0.04 m) beats the component benchmark (0.20 m); a testable extension is to relax the stop requirement by compensating for motion-induced blur and sensor desynchronization.
- The benchmark's human-annotated ground truth and fixed prompt may favor models that happen to match the annotation style; a natural next test is measuring robustness to paraphrased instructions and prompt variants.
- If this approach generalizes, cloud-free VLN could extend to long-duration environmental monitoring and search-and-rescue in intermittent-connectivity settings; a straightforward extension is running the same stack on a legged or aerial platform.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents Edge-BehAV, a fully onboard vision-language navigation system for outdoor robots. It makes three contributions: (i) a navigation-specific benchmark comparing 17 edge-deployable SLMs with 4 cloud APIs on 350 self-annotated instruction decompositions; (ii) a hybrid semantic-geometric goal localization pipeline that combines Florence-2 detection, Mobile-SAM segmentation, and LiDAR projection with a robust median centroid, plus a bearing-only fallback when metric lock fails; and (iii) integration into a complete system validated in 32 closed-loop trials on a Husky A200. The headline results are that Qwen2.5-7B matches GPT-5.5 on Macro-F1 (0.775 vs 0.765) at ~9x lower latency, LiDAR-visual goal localization reduces mean distance error from 2.05 m to 0.20 m, and the integrated system succeeds in 31/32 trials. The architecture and experimental setup are described in detail, and several limitations are disclosed in the text, notably the exclusion of invalid LiDAR measurements from the distance MAE.
Significance. If the results hold, this is a useful step toward cloud-independent VLN: the benchmark is a potentially reusable resource, the system-level comparison is meaningful, and 31/32 closed-loop successes with fully onboard inference is a strong system result. The paper deserves credit for releasing the benchmark, using tape-measured ground truth and nuImages for perception evaluation, reporting bootstrap confidence intervals for the main language result, and being transparent about several limitations. However, the headline localization claim is an asymmetric comparison that excludes exactly the failure regime the fallback is designed for, and the language ground truth is single-annotator without agreement statistics. These two issues are load-bearing for the benchmark and localization contributions, but both are addressable with additional analysis and rewording.
major comments (3)
- [Abstract; §IV-D; §V-D; Table IV] The headline 'reduces mean goal-distance error from 2.05 m to 0.20 m' is an asymmetric comparison. Table IV reports one trial per cell; the traffic-cone rows at 10 and 20 m are marked '—' and excluded from Edge-BehAV's MAE, while the BehAV baseline contributes values in all 16 cells. Thus 0.20 m is a 14-trial MAE over valid measurements, and 2.05 m is a 16-trial MAE. The text in §V-D that 'worst-case error below 0.45 m over the full 2–20 m range' is not supported because the invalid cells are assigned no error. The paper does disclose 'over all valid measurements' in §IV-D, but the abstract and conclusion state the reduction unqualifiedly. Please either quantify the bearing-only fallback's error when metric lock fails and include it in the reported MAE, or qualify all claims to valid measurements and explicitly report the rate and range of measurement failures. This is load-bearing for c
- [§IV-B; Table I] The language decomposition ground truth is self-annotated by the authors, and no inter-annotator agreement is reported. The central claim that Qwen2.5-7B matches GPT-5.5 within confidence intervals depends on the stability of that ground truth. Given that per-field F1 values differ materially (e.g., Lmk-F1 0.750 vs 0.644), label noise on 350 instructions could shift the comparison. An inter-annotator agreement measure on a subset, or a second independent annotation, is needed to support the claim of a 'human-annotated' benchmark rather than one team's parsing preference. This is a missing support for the benchmark contribution, though not fatal to the system result.
- [§III.C–III.D; Table V] The regime where Table IV shows failures — small, low-reflectance targets at range — is exactly the case the bearing-only fallback is designed to handle, yet the only closed-loop failure is in Scn.2 (87%, 7/8), the bearing-to-metric handover scenario. With N=8 per scenario, the fallback is validated by very few trials and the single failure is in the relevant transition. Please provide additional trials or an analysis of that failure, and report closed-loop success conditional on fallback usage. The 31/32 claim is factually correct but does not yet establish that the fallback is reliable in the regime where Table IV has gaps.
minor comments (4)
- [Eq. (8)] The notation 'T max t=1' appears to be a typo for a summation over trajectory steps; please correct to a standard sum or max expression.
- [Fig. 1] The radar-chart axes are normalized but not defined in the caption or text, making it difficult for a reader to verify the 'at a glance' claims. Please define the normalization and the per-axis metrics.
- [§III.C] The transforms T_lidar→cam and the camera intrinsic matrix K are used in Eq. (1) but not defined in terms of calibration procedure or frame conventions; a sentence on the extrinsic calibration would help reproducibility.
- [§IV.D] Table IV reports one trial per cell with no variance. Given that the invalid-cell decision and MAE are central to the localization claim, at least a few repeated trials per distance/object would strengthen the result.
Circularity Check
No significant circularity: the main results are external-measurement benchmarks, not derivations from fitted inputs.
full rationale
The paper's three headline claims rest on external references: (i) the SLM-vs-cloud comparison is scored against a fixed, human-annotated instruction set (Sec. IV-B), which is a dataset contribution rather than a fitted parameter renamed as a prediction; (ii) the 2.05 m-to-0.20 m goal-localization result is computed against tape-measured ground truth on a Husky A200 (Sec. IV-D, Table IV) using a median LiDAR centroid defined in Eq. (3), with no parameter fitted to those same measurements; and (iii) closed-loop success (31/32) is judged against goal tolerance, teleoperation references, and pre-surveyed forbidden-region maps (Sec. IV-F, Table V). The paper explicitly discloses the '—' invalid trials for the traffic cone at 10/20 m and states the MAE is over valid measurements, which is a robustness/fairness limitation of the reported comparison rather than a circular reduction: the goal estimate is not defined in terms of the evaluation outcome. There are no self-citation chains, uniqueness theorems, or ansatz-by-citation steps; BehAV [3] is prior work by different authors and is used as a baseline, not as justification for the proposed method's correctness.
Axiom & Free-Parameter Ledger
free parameters (8)
- N_pts =
7
- IQR multiplier for LiDAR outlier rejection =
0.5
- Bearing-mode MPC weights w_alpha, w_v =
1.5, 0.5
- MPC weights w_goal, w_behav =
2.0, 8.0
- Behavioral cost decay lambda_b =
0.3
- Obstacle ramp parameters (factor, distance) =
3.0, 2.0 m
- Obstacle exclusion radius around goal =
1.2 m
- Behavioral cost levels =
0.0/0.5/0.7/1.0
axioms (8)
- standard math LiDAR points project onto the image plane via a calibrated pinhole model (Eq. 1), and the extrinsic transformation T_{lidar->cam} and intrinsics K are accurate.
- domain assumption Florence-2 open-vocabulary detection returns a bounding box covering the goal object for the SLM-provided landmark phrase.
- domain assumption Mobile-SAM's prompt-conditioned mask accurately delineates the goal object within the detected box.
- domain assumption The median of valid LiDAR points inside the mask is an appropriate estimate of the metric goal point (Eq. 3).
- domain assumption Odometry pose (x_r, y_r, theta_r) is accurate enough over the short latching range (Eq. 4).
- domain assumption The CLIPSeg behavioral cost map inherited from BehAV reliably encodes per-pixel behavioral rule costs.
- domain assumption The set-level Macro-F1 against the authors' human-annotated ground truth measures navigation-instruction decomposition quality.
- domain assumption The behavioral cost mapping (stay-on 0.0, avoidance 0.5, yielding 0.7, stopping 1.0) is a valid encoding of user intent.
read the original abstract
Foundation-model-based vision-language navigation (VLN) has advanced autonomous robot navigation by enabling robots to interpret natural-language instructions, identify semantic goals, and follow user-specified behavioral rules. However, existing VLN systems rely heavily on cloud-hosted foundation models for language understanding and semantic grounding, limiting their applicability where network connectivity is unavailable and reliable metric goal localization is required. Although recent small language models (SLMs) enable fully onboard inference, their suitability for navigation instruction decomposition has not been systematically evaluated. This paper makes three contributions toward fully onboard VLN for outdoor environments. First, we present the first systematic benchmark of 17 edge-deployable SLMs against 4 online APIs for robotic navigation instruction decomposition, evaluating accuracy and latency on human-annotated instructions across three computing platforms and providing practical guidance for selecting onboard language models. Second, we propose a lightweight hybrid semantic-geometric goal localization framework that combines open-vocabulary object detection, prompted segmentation, and LiDAR geometry to estimate metric goals, while maintaining visual bearing guidance when reliable geometric observations are unavailable. Third, we integrate these advances into Edge-BehAV, a fully onboard extension of the BehAV architecture that enables cloud-independent behavior-guided navigation. Experimental results show that the best offline SLM matches the instruction decomposition performance of the strongest cloud API while running approximately 9x faster and without network connectivity. The proposed goal localization framework reduces mean goal-distance error from 2.05 m to 0.20 m at lower computational cost, and the complete system succeeds in 31 of 32 closed-loop outdoor trials.
Figures
Reference graph
Works this paper leans on
-
[1]
LM-Nav: Robotic navigation with large pre-trained models of language, vision, and action,
D. Shah, B. Osi ´nski, b. ichter, and S. Levine, “LM-Nav: Robotic navigation with large pre-trained models of language, vision, and action,” inConference on Robot Learning. PMLR, 2023, pp. 492–504
2023
-
[2]
Convoi: Context-aware naviga- tion using vision language models in outdoor and indoor environments,
A. J. Sathyamoorthy, K. Weerakoon, M. Elnoor, A. Zore, B. Ichter, F. Xia, J. Tan, W. Yu, and D. Manocha, “Convoi: Context-aware naviga- tion using vision language models in outdoor and indoor environments,” in2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024, pp. 13 837–13 844
2024
-
[3]
Behav: Behavioral rule guided autonomy using vlms for robot navigation in outdoor scenes,
K. Weerakoon, M. Elnoor, G. Seneviratne, V . Rajagopal, S. H. Arul, J. Liang, M. K. M. Jaffar, and D. Manocha, “Behav: Behavioral rule guided autonomy using vlms for robot navigation in outdoor scenes,” in2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 7044–7051
2025
-
[4]
Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,
P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. van den Hengel, “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” in2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 3674–3683
2018
-
[5]
Vision-and- language navigation: A survey of tasks, methods, and future directions,
J. Gu, E. Stefani, Q. Wu, J. Thomason, and X. Wang, “Vision-and- language navigation: A survey of tasks, methods, and future directions,” inProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 7606–7623
2022
-
[6]
Do as i can, not as i say: Grounding language in robotic affordances,
A. Brohan, Y . Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julianet al., “Do as i can, not as i say: Grounding language in robotic affordances,” inConference on robot learning. PMLR, 2023, pp. 287–318
2023
-
[7]
Small language models: Survey, measurements, and insights,
Z. Lu, X. Li, D. Cai, R. Yi, F. Liu, X. Zhang, N. D. Lane, and M. Xu, “Small language models: Survey, measurements, and insights,”arXiv preprint arXiv:2409.15790, 2024
Pith/arXiv arXiv 2024
-
[8]
Phi-3 technical report: A highly capable language model locally on your phone,
M. Abdin, S. A. Jacobs, A. A. Awan, J. Aneja, A. Awadallah, H. Awadalla, N. Bach, A. Bahree, A. Bakhtiariet al., “Phi-3 technical report: A highly capable language model locally on your phone,”arXiv preprint arXiv:2404.14219, 2024
Pith/arXiv arXiv 2024
-
[9]
Qwen Team, “Qwen2.5 technical report,”arXiv preprint arXiv:2412.15115, 2025
Pith/arXiv arXiv 2025
-
[10]
MobileVLM: A fast, strong and open vision language assistant for mobile devices,
X. Chu, L. Qiao, X. Lin, S. Xu, Y . Yang, Y . Hu, F. Wei, X. Zhang, B. Zhang, X. Wei, and C. Shen, “MobileVLM: A fast, strong and open vision language assistant for mobile devices,”arXiv preprint arXiv:2312.16886, 2023
Pith/arXiv arXiv 2023
-
[11]
Florence-2: Advancing a unified representation for a variety of vision tasks,
B. Xiao, H. Wu, W. Xu, X. Dai, H. Hu, Y . Lu, M. Zeng, C. Liu, and L. Yuan, “Florence-2: Advancing a unified representation for a variety of vision tasks,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024
2024
-
[12]
Vint: A foundation model for visual navigation,
D. Shah, A. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine, “Vint: A foundation model for visual navigation,” in Conference on Robot Learning. PMLR, 2023, pp. 711–733
2023
-
[13]
Nomad: Goal masked diffusion policies for navigation and exploration,
A. Sridhar, D. Shah, C. Glossop, and S. Levine, “Nomad: Goal masked diffusion policies for navigation and exploration,” in2024 IEEE In- ternational Conference on Robotics and Automation (ICRA), 2024, pp. 63–70
2024
-
[14]
Learning transferable visual models from natural language supervi- sion,
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervi- sion,” inProceedings of the 38th International Conference on Machine Learning. PMLR, 2021, pp. 8748–8763
2021
-
[15]
Image segmentation using text and image prompts,
T. Lüddecke and A. Ecker, “Image segmentation using text and image prompts,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 7086–7096
2022
-
[16]
Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection,
S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, and L. Zhang, “Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection,” inProceedings of the European Conference on Computer Vision, 2024
2024
-
[17]
nuscenes: A multimodal dataset for autonomous driving,
H. Caesar, V . Bankiti, A. H. Lang, S. V ora, V . E. Liong, Q. Xu, A. Kr- ishnan, Y . Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” inCVPR, 2020
2020
-
[18]
Faster segment anything: Towards lightweight SAM for mobile applications,
C. Zhang, D. Han, Y . Qiao, J. U. Kim, S.-H. Bae, S. Lee, and C. S. Hong, “Faster segment anything: Towards lightweight SAM for mobile applications,”arXiv preprint arXiv:2306.14289, 2023
Pith/arXiv arXiv 2023
-
[19]
Segment anything,
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, P. Dollár, and R. Girshick, “Segment anything,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 4015–4026
2023
-
[20]
GPTQ: Accurate post-training quantization for generative pre-trained transformers,
E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accurate post-training quantization for generative pre-trained transformers,” in Proceedings of the 11th International Conference on Learning Repre- sentations, 2023
2023
-
[21]
Google DeepMind, “Gemma 3 technical report,”arXiv preprint arXiv:2503.19786, 2025
Pith/arXiv arXiv 2025
-
[22]
PointPainting: Sequen- tial fusion for 3D object detection,
S. V ora, A. H. Lang, B. Helou, and O. Beijbom, “PointPainting: Sequen- tial fusion for 3D object detection,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 4604–4612
2020
-
[23]
Frustum PointNets for 3D object detection from RGB-D data,
C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas, “Frustum PointNets for 3D object detection from RGB-D data,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 918– 927
2018
-
[24]
OpenScene: 3D scene understanding with open vocabu- laries,
S. Peng, K. Genova, C. Jiang, A. Tagliasacchi, M. Pollefeys, and T. Funkhouser, “OpenScene: 3D scene understanding with open vocabu- laries,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023
2023
-
[25]
Information-theoretic model predictive control: Theory and applications to autonomous driving,
G. Williams, P. Drews, B. Goldfain, J. M. Rehg, and E. A. Theodorou, “Information-theoretic model predictive control: Theory and applications to autonomous driving,”IEEE Transactions on Robotics, vol. 34, no. 6, pp. 1603–1622, 2018
2018
-
[26]
Fast segment anything,
X. Zhao, W. Ding, Y . An, Y . Du, T. Yu, M. Li, M. Tang, and J. Wang, “Fast segment anything,” 2023
2023
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.