REVIEW 3 major objections 5 minor 34 references
SkyVLN: Vision-and-Language Navigation and NMPC Control for UAVs in Urban Environments
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read SkyVLN is a training-free framework that gives a drone an LLM navigation brain, spatial memory, and NMPC flight control, and its authors report more than doubling the unseen-route success rate over the strongest baseline.
desk verdict Plausible LLM+NMPC integration for aerial VLN, but the headline AVDN numbers are not reproducible as reported—environment mapping and LLM choice are both unspecified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing components are the High-resolution Spatial Descriptor (HSD), the TrackBack Memory Array (TBMA), and an NMPC layer. HSD divides each frontal camera image into nine named sectors and lets the LLM rank candidate landmark matches by their spatial relationship, so the model knows not just that a KFC exists but that it sits upper-left. TBMA builds a graph whose nodes are observed landmarks and whose edges store the navigation instructions between them, enabling backtracking when the agent realizes it has taken a wrong route. The NMPC layer, solved with the PANOC algorithm through the OpEn software, tracks the LLM's waypoints while respecting actuator limits and spherical dynamic-obstacle constraints.
What would settle it
Run the full SkyVLN agent on the official AVDN benchmark simulator and compare its SPL and success rate against the reported NavGPT numbers. If the improvement largely disappears outside the paper's custom AirSim city, then the gains are tied to the new simulator rather than to the HSD, TBMA, and NMPC design.
Extended reading notes
Core claim
The central claim is that an LLM-based aerial navigation agent can outperform specialist trained baselines without any task-specific fine-tuning, provided it is given three things: a fine-grained spatial verbalizer that labels the nine sectors of each camera view, a TrackBack Memory Array that stores visited landmarks as a graph with instruction edges, and a nonlinear model predictive controller that converts the LLM's textual motion commands into collision-free trajectories. In the paper's experiments on the AVDN benchmark, this combination reaches 42.37% success rate and 28.11% SPL on unseen test data, compared with 16.6% and 18.9% for NavGPT, the strongest baseline.
Load-bearing premise
The reported navigation gains rest on the assumption that the custom AirSim/UE4 simulator used for evaluation matches the platform on which the AVDN benchmark data were collected; the paper does not explicitly establish this link.
Editorial extensions
If this is right
- A no-training pipeline can handle ambiguous instructions by decomposing them into sub-goals, asking clarification questions when needed, and backtracking via the memory graph.
- The spatial verbalizer can distinguish landmark identity from landmark placement, letting the model reject observations that match the right objects in the wrong spatial arrangement.
- The NMPC backend turns coarse linguistic waypoints into smooth, dynamically feasible trajectories, which is the missing link between LLM planning and low-level drone control.
- The reported gains on unseen test routes suggest the framework generalizes to new layouts without retraining, since the LLM and detector are not fine-tuned.
Reading between the lines
- If the custom AirSim/UE4 simulator is not the same platform on which the AVDN benchmark data were collected, the two contributions should be evaluated separately: HSD and TBMA against other VLN agents in a shared simulator, and NMPC against PID in real flight.
- The clarify-question mechanism could be tested independently as a way to reduce ambiguous-instruction failures in ground-based VLN benchmarks, not just aerial ones.
- A natural extension is to replace the fixed nine-sector grid with learned open-vocabulary spatial relations, which might reduce prompt sensitivity and improve generalization across different urban styles.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SkyVLN, a framework for vision-and-language navigation (VLN) of UAVs in urban environments. The system combines an off-the-shelf vision foundation model (GroundingDINO) for landmark extraction, a spatial verbalizer that divides the view into nine sectors, a TrackBack Memory Array for handling ambiguous instructions, and an LLM-based motion command generator whose outputs are tracked by a nonlinear model predictive controller (NMPC). The authors also describe a custom AirSim/UE4 urban simulator with buildings, vehicles, pedestrians, and semantic/depth/RGB observations. Experiments are reported on the AVDN dataset, with a headline result of 28.11% SPL and 42.37% SR on the unseen test split, compared with NavGPT's 18.9% SPL and 16.6% SR, plus ablations over LLM backends and control modules.
Significance. If the reported results are reproducible, SkyVLN would be a useful contribution: it integrates LLM-based spatial reasoning with model predictive control for aerial VLN, uses a public dataset for evaluation, and provides a high-fidelity simulator that could support further work. The paper also has the strength of relying on off-the-shelf components and external baselines rather than training a new policy, which makes the approach relatively easy to inspect. However, the evaluation has load-bearing gaps: the relationship between the custom simulator and the AVDN evaluation environment is never stated explicitly, the LLM ablation does not reproduce the headline configuration, and all headline numbers are single runs without variance estimates. These issues currently prevent the central claim of improved navigation performance from being accepted at face value.
major comments (3)
- [Section IV and Section V-A] The paper never explicitly states that the custom AirSim/UE4 simulator described in Section IV is the same environment used to replay the AVDN episodes for the results in Table I. Section V-A says only that experiments use the AVDN dataset and that the algorithm was 'evaluated on AirSim and UE4'. If the AVDN episodes were rendered in the newly built city environment rather than AVDN's official evaluation scenes, then the baseline numbers in Table I are not directly comparable and the reported gains are not an AVDN benchmark result. Please state the environment mapping explicitly, and if the custom simulator is not AVDN's official platform, provide a direct comparison in the official environment or clearly reframe the claims.
- [Table II and Section V-D] The ablation over LLM backends does not reproduce the headline 'Ours Full' result from Table I. Table I reports 28.11% SPL and 42.37% SR for 'Ours Full', but Table II reports GPT-4V with 16.62/34.9, GPT-4o with 34.25/20.44, and GPT-4 Turbo with 25.12/15.62; no row equals the headline configuration. The text never states which LLM produced the Table I results. Please specify the LLM used for each row, explain the discrepancy, and state whether Table II's unseen-test numbers were obtained under the same conditions as Table I.
- [Section V-B, Table I] All quantitative comparisons in Table I appear to be single runs: there are no error bars, no number of episodes, no seeds, and no statistical significance tests. The claimed improvements over NavGPT are large (28.11 vs 18.9 SPL; 42.37 vs 16.6 SR), but without variance estimates or repeated trials it is impossible to assess whether the differences are reliable or due to randomness in the LLM sampling and the environment. Please report means and standard deviations over multiple runs or episode subsets.
minor comments (5)
- [General] There are frequent typographical inconsistencies, such as 'UA Vs' instead of 'UAVs' and 'Figure.' with a period before the number; these should be cleaned up throughout.
- [Section III-B] The acronym 'TBMA' (TrackBack Memory Array) is introduced, but later the text uses 'TMA' (e.g., in the paragraph after Figure 3). Please use one acronym consistently.
- [Section III-A] The state vector is introduced as P = [x, y, z, u, v, w, phi, theta, psi, p, q, r], where [u,v,w] and [p,q,r] are called 'corresponding derivatives'; however, [p,q,r] are body angular rates, not time derivatives of the Euler angles. Please clarify the notation.
- [Section V-A] The dataset name appears as 'A VDN' and as 'AVDN' in different places; please make the name consistent and verify that the correct acronym is used for reference [34].
- [Section III-B] The name 'Wayfinding Prompt Optimization' suggests the prompts are optimized automatically, but the paper does not describe such an optimization procedure; please clarify what is optimized and how.
Circularity Check
No circular derivation: the headline AVDN comparison is against external baselines, and the only self-citation is non-load-bearing.
full rationale
SkyVLN's central claim is the unseen-test SPL/SR improvement over NavGPT and other baselines in Table I. These baseline numbers are external published results (Random, CMA, Seq2Seq [35], NavGPT [6]) on the public AVDN dataset [34], and the proposed framework is explicitly "without supervised training," so no fitted parameter is later renamed as a prediction. The NMPC controller is derived from a standard quadrotor model credited to [27] and solved with the external OpEn/PANOC library [28]; no self-referential uniqueness theorem or answer-forcing ansatz is employed. The only self-citation is [25] by co-author Y. Gao, used for a motivational claim about repeated execution under ambiguous instructions and for a sim-to-real caveat; neither citation carries the headline result, so it is not load-bearing. The apparent mismatch between the custom AirSim/UE4 simulator (Section IV) and the AVDN evaluation setup (Section V-A), and the absence of a Table II row matching "Ours Full," are reproducibility and external-validity concerns about whether the reported comparison is an apples-to-apples AVDN result; under the circularity criteria these are not derivation loops, so they do not raise the circularity score beyond the minor self-citation band.
Assumptions & free parameters
free parameters (5)
- NMPC weight matrices Qx, Qu, QΔu
- Safety radius rs and obstacle radius robs
- Spatial sector count (9 sectors) =
9
- Top-K landmark candidates
- LLM sampling parameters (temperature, top-p)
assumptions (4)
- domain assumption The quadrotor dynamics in Eq. (2) (first-order attitude response, linear damping, SO(3) rotation) accurately represent the simulated UAV.
- domain assumption The AVDN dataset provides valid instruction-trajectory pairs for evaluating urban aerial VLN.
- domain assumption GPT-4 retains landmark-relevant features from front-view RGB images without fine-tuning, and hallucination is controlled by prompting.
- ad hoc to paper The custom AirSim environment (Section IV) is the same platform used for the AVDN evaluation in Section V.
Cite this review
Pith. "Pith review of SkyVLN: Vision-and-Language Navigation and NMPC Control for UAVs in Urban Environments." pith.science (2026). https://pith.science/paper/KFJEWE5R
@misc{pith2026250706564,
author = {Pith},
title = {Pith review of: SkyVLN: Vision-and-Language Navigation and NMPC Control for UAVs in Urban Environments},
year = {2026},
howpublished = {\url{https://pith.science/paper/KFJEWE5R}},
note = {Machine review of arXiv:2507.06564}
}
read the original abstract
Unmanned Aerial Vehicles (UAVs) have emerged as versatile tools across various sectors, driven by their mobility and adaptability. This paper introduces SkyVLN, a novel framework integrating vision-and-language navigation (VLN) with Nonlinear Model Predictive Control (NMPC) to enhance UAV autonomy in complex urban environments. Unlike traditional navigation methods, SkyVLN leverages Large Language Models (LLMs) to interpret natural language instructions and visual observations, enabling UAVs to navigate through dynamic 3D spaces with improved accuracy and robustness. We present a multimodal navigation agent equipped with a fine-grained spatial verbalizer and a history path memory mechanism. These components allow the UAV to disambiguate spatial contexts, handle ambiguous instructions, and backtrack when necessary. The framework also incorporates an NMPC module for dynamic obstacle avoidance, ensuring precise trajectory tracking and collision prevention. To validate our approach, we developed a high-fidelity 3D urban simulation environment using AirSim, featuring realistic imagery and dynamic urban elements. Extensive experiments demonstrate that SkyVLN significantly improves navigation success rates and efficiency, particularly in new and unseen environments.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
Networked unmanned aerial vehicles for surveillance and monitoring: A survey,
X. Li and A. V . Savkin, “Networked unmanned aerial vehicles for surveillance and monitoring: A survey,” Future Internet, vol. 13, no. 7, p. 174, 2021
work page 2021
-
[2]
UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobility
Y . Tian, F. Lin, Y . Li, T. Zhang, Q. Zhang, X. Fu, J. Huang, X. Dai, Y . Wang, C. Tian, B. Li, Y . Lv, L. Kov ´acs, and F.-Y . Wang, “Uavs meet llms: Overviews and perspectives toward agentic low-altitude mobility,” 2025. [Online]. Available: https://arxiv.org/abs/2501.02341
work page Pith review arXiv 2025
-
[3]
F. Al-Turjman and H. Zahmatkesh, “A comprehensive review on the use of ai in uav communications: Enabling technologies, applications, and challenges,” Unmanned Aerial Vehicles in Smart Cities , pp. 1–26, 2020
work page 2020
-
[4]
J. Rasley, S. Rajbhandari, O. Ruwase, and Y . He, “Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2020, pp. 3505–3506
work page 2020
-
[5]
H. Kurunathan, H. Huang, K. Li, W. Ni, and E. Hossain, “Machine learning-aided operations and communications of unmanned aerial vehicles: A contemporary survey,” IEEE Communications Surveys and Tutorials, 2023
work page 2023
-
[6]
Navgpt: Explicit reasoning in vision-and-language navigation with large language models,
G. Zhou, Y . Hong, and Q. Wu, “Navgpt: Explicit reasoning in vision-and-language navigation with large language models,” in AAAI Conference on Artificial Intelligence , 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:258947250
work page 2023
-
[7]
Navgpt-2: Unleashing navigational reasoning capability for large vision-language models,
G. Zhou, Y . Hong, Z. Wang, X. E. Wang, and Q. Wu, “Navgpt-2: Unleashing navigational reasoning capability for large vision-language models,” in European Conference on Computer Vision, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:271244657
work page 2024
-
[8]
Visual language navigation: A survey and open challenges,
S.-M. Park and Y .-G. Kim, “Visual language navigation: A survey and open challenges,” Artificial Intelligence Review , vol. 56, no. 1, pp. 365–427, 2023
work page 2023
Show all 34 references
-
[9]
Tell me where you are: Multimodal llms meet place recognition,
Z. Lyu, J. Zhang, M. Lu, Y . Li, and C. Feng, “Tell me where you are: Multimodal llms meet place recognition,” arXiv preprint arXiv:2406.17520 , no. arXiv:2406.17520, June 2024, arXiv:2406.17520 [cs]. [Online]. Available: http://arxiv.org/abs/2406. 17520
2024 arXiv
-
[10]
Touchdown: Natural language navigation and spatial reasoning in visual street envi- ronments,
H. Chen, A. Suhr, D. Misra, N. Snavely, and Y . Artzi, “Touchdown: Natural language navigation and spatial reasoning in visual street envi- ronments,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 12 538–12 547
2019
-
[11]
Beyond the nav-graph: Vision-and-language navigation in continuous environ- ments,
J. Krantz, E. Wijmans, A. Majumdar, D. Batra, and S. Lee, “Beyond the nav-graph: Vision-and-language navigation in continuous environ- ments,” in Computer Vision – ECCV 2020 , A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, Eds. Cham: Springer International Publishing, 2020,...
2020
-
[13]
Advances in uav avionics systems architecture, classification and integration: A comprehensive review and future perspectives,
H. A. Hashim, “Advances in uav avionics systems architecture, classification and integration: A comprehensive review and future perspectives,” Results in Engineering , vol. 25, p. 103786, Mar. 2025. [Online]. Available: http://dx.doi.org/10.1016/j.rineng.2024.103786
2025
-
[14]
Coordinated navigation control of cross-domain unmanned systems via guiding vector fields,
B.-B. Hu, H.-T. Zhang, B. Liu, J. Ding, Y . Xu, C. Luo, and H. Cao, “Coordinated navigation control of cross-domain unmanned systems via guiding vector fields,” IEEE Transactions on Control Systems Technology, vol. 32, no. 2, pp. 550–563, 2024
2024
-
[15]
Vision- based deep reinforcement learning of uav autonomous navigation using privileged information,
J. Wang, Z. Yu, D. Zhou, J. Shi, and R. Deng, “Vision- based deep reinforcement learning of uav autonomous navigation using privileged information,” 2024. [Online]. Available: https: //arxiv.org/abs/2412.06313
2024 arXiv
-
[16]
A multi-lingual speech recognition-based framework to human-drone interaction,
K. Choutri, M. Lagha, S. Meshoul, M. Batouche, Y . Kacel, and N. Mebarkia, “A multi-lingual speech recognition-based framework to human-drone interaction,” Electronics, vol. 11, no. 12, p. 1829, 2022
2022
-
[17]
Real: Resilience and adaptation using large language models on autonomous aerial robots,
A. Tagliabue, K. Kondo, T. Zhao, M. Peterson, C. T. Tewari, and J. P. How, “Real: Resilience and adaptation using large language models on autonomous aerial robots,” arXiv preprint arXiv:2311.01403, 2023
2023 arXiv
-
[18]
Towards realistic uav vision-language navigation: Platform, benchmark, and methodology,
X. Wang, D. Yang, Z. Wang, H. Kwan, J. Chen, W. Wu, H. Li, Y . Liao, and S. Liu, “Towards realistic uav vision-language navigation: Platform, benchmark, and methodology,” 2024. [Online]. Available: https://arxiv.org/abs/2410.07087
2024 arXiv
-
[19]
Bert: Pre-training of deep bidirectional transformers for language understanding,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018
2018 arXiv
-
[20]
A language model-based fine- grained address resolution framework in uav delivery system,
S. Luo, Y . Yao, H. Zhao, and L. Song, “A language model-based fine- grained address resolution framework in uav delivery system,” IEEE Journal of Selected Topics in Signal Processing , 2024
2024
-
[21]
Optimal navigation of an automatic guided vehicle with obstacle constraints: A broad learning-based approach,
J. Lai, Z. Wu, Z. Ren, Q. Tan, and H. Xiao, “Optimal navigation of an automatic guided vehicle with obstacle constraints: A broad learning-based approach,” IEEE Transactions on Emerging Topics in Computational Intelligence, pp. 1–15, 2024
2024
-
[22]
A review of gnss-independent uav navigation techniques,
N. Gyagenda, J. V . Hatilima, H. Roth, and V . Zhmud, “A review of gnss-independent uav navigation techniques,” Robotics and Autonomous Systems , vol. 152, p. 104069, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0921889022000343
2022
-
[23]
Beyond the nav-graph: Vision-and-language navigation in continuous environments,
J. Krantz, E. Wijmans, A. Majumdar, D. Batra, and S. Lee, “Beyond the nav-graph: Vision-and-language navigation in continuous environments,” CoRR, vol. abs/2004.02857, 2020. [Online]. Available: https://arxiv.org/abs/2004.02857
2004 arXiv
-
[24]
Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection,
S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, and L. Zhang, “Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection,” 2024. [Online]. Available: https://openreview.net/forum?id=DS5qRs0tQz
2024
-
[25]
Aerial vision-and-language navigation via semantic-topo-metric representation guided llm reasoning,
Y . Gao, Z. Wang, L. Jing, D. Wang, X. Li, and B. Zhao, “Aerial vision-and-language navigation via semantic-topo-metric representation guided llm reasoning,” Oct. 2024, arXiv:2410.08500 [cs]. [Online]. Available: http://arxiv.org/abs/2410.08500
2024 arXiv
-
[26]
Embodiedcity: A bench- mark platform for embodied agent in real-world city environment,
C. Gao, B. Zhao, W. Zhang, J. Zhang, J. Mao, Z. Zheng, F. Man, J. Fang, Z. Zhou, J. Cui, X. Chen, and Y . Li, “Embodiedcity: A bench- mark platform for embodied agent in real-world city environment,” arXiv preprint, 2024
2024
-
[27]
Model predictive control for trajectory tracking of unmanned aerial vehicles using robot operating system,
M. Kamel, T. Stastny, K. Alexis, and R. Siegwart, “Model predictive control for trajectory tracking of unmanned aerial vehicles using robot operating system,” Robot Operating System (ROS) The Complete Reference (Volume 2), pp. 3–39, 2017
2017
-
[28]
Optimization engine,
P. Sopasakis and E. Fresk, “Optimization engine,” https://alphaville. github.io/optimization-engine/, 2024, accessed on December 4, 2024
2024
-
[29]
Mirage: an efficient and extensible city simulation framework (systems paper),
J. Zhang, D. Jin, and Y . Li, “Mirage: an efficient and extensible city simulation framework (systems paper),” in Proceedings of the 30th International Conference on Advances in Geographic Information Systems , ser. SIGSPATIAL ’22. New York, NY , USA: Association for Computing...
2022
-
[30]
Airsim: High-fidelity visual and physical simulation for autonomous vehicles,
S. Shah, D. Dey, C. Lovett, and A. Kapoor, “Airsim: High-fidelity visual and physical simulation for autonomous vehicles,” 2017. [Online]. Available: https://arxiv.org/abs/1705.05065
2017 arXiv
-
[31]
Sanders, An Introduction to Unreal Engine 4
A. Sanders, An Introduction to Unreal Engine 4 . A K Peters/CRC Press, 2016
2016
-
[32]
Vision-and-language nav- igation: Interpreting visually-grounded navigation instructions in real environments,
P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. S ¨underhauf, I. Reid, S. Gould, and A. Van Den Hengel, “Vision-and-language nav- igation: Interpreting visually-grounded navigation instructions in real environments,” in Proceedings of the IEEE conference on computer vis...
2018
-
[33]
Room- across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding,
A. Ku, P. Anderson, R. Patel, E. Ie, and J. Baldridge, “Room- across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding,” arXiv preprint arXiv:2010.07954 , 2020
2010 arXiv
-
[34]
Aerial vision-and-dialog navigation,
Y . Fan, W. Chen, T. Jiang, C. Zhou, Y . Zhang, and X. Wang, “Aerial vision-and-dialog navigation,” in Findings of the Association for Computational Linguistics: ACL 2023. Toronto, Canada: Association for Computational Linguistics, 2023, p. 3043–3061. [Online]. Available: http...
2023
-
[35]
Vision-and- Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments ,
P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sunderhauf, I. Reid, S. Gould, and A. van den Hengel, “ Vision-and- Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments ,” in 2018 IEEE/CVF Conference on Computer Vision and Pat...
2018
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.