Pith. sign in

REVIEW 2 major objections 4 minor 33 references

The paper introduces MissionBench and claims that state-of-the-art multimodal LLMs, evaluated zero-shot as closed-loop drone agents, complete fewer than 35% of missions compared to 84.4% human performance, and that mission-level competence

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:02 UTC pith:NLZY6X4T

load-bearing objection MissionBench is a genuinely new closed-loop aerial benchmark with a solid empirical core, but the headline 34.8% success rate is softer than it looks because reporting missions are judged by an LLM that tolerates minor character errors. the 2 major comments →

arxiv 2607.22014 v1 pith:NLZY6X4T submitted 2026-07-24 cs.AI cs.CLcs.CVcs.RO

Zero-Shot Mission-Level Evaluation for Aerial MLLM Agents

Suman Navaratnarajah , Taehyoung Kim , Jona Ruthardt , Ishaan Bhimwal , Ryousuke Yamada , Yannik Blei , Wolfram Burgard , Yuki M Asano This is my paper
classification cs.AI cs.CLcs.CVcs.RO
keywords MissionBenchaerial MLLM agentszero-shot evaluationclosed-loop benchmarkembodied navigationmultimodal LLMdrone missionsscaling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

MissionBench is a closed-loop benchmark that asks general-purpose multimodal language models to fly a simulated drone from a single high-level instruction: they must decide where to go, how to position the camera, and what to report. Across 22 models, the strongest completes fewer than 35% of missions, while humans using the same interface complete 70% and humans with keyboard control complete 84.4%. The authors argue that mission-level competence is not explained by single-frame spatial perception alone; it requires integrating perception, planning, and reporting across a long-horizon episode. They also find that larger models perform better, indicating that general-purpose scaling contributes to embodied ability but remains far from human-level.

Core claim

MissionBench shows that frozen, off-the-shelf MLLMs, evaluated zero-shot as aerial agents, succeed on at most 34.8% of 30 mission-level tasks, compared to 84.4% human performance with continuous control and 70.0% when humans are forced through the same text/image interface. The oracle success rate—whether the agent ever reaches the target—exceeds the success rate for several models, meaning agents often navigate to the right place but then fail to complete the mission through misidentification, premature termination, or drift. Scaling within model families consistently improves mission progress but does not close the human-model gap, and single-frame perception proxy scores correlate only mo

What carries the argument

The central object is MissionBench itself: 120 missions across five simulated 3D environments and four task families (reporting, inspection, manipulation, patrol), plus a closed-loop protocol in which the agent receives an egocentric image, emits a structured response (bounding box, reasoning, action with magnitude, report, and a DONE flag), the simulator executes the action, and the loop repeats until DONE or budget exhaustion. It introduces mission-level success criteria—spatial thresholds for inspection and manipulation, soft LLM-judged matching for reporting, IoU over buffered trajectories for patrol—and diagnostic metrics (Success Rate, Oracle Success Rate, Mission Progress, Collision R

Load-bearing premise

Reporting missions are graded by an LLM judge that treats reports differing by one or two characters or numbers as successes (90% similarity threshold); if exact string matching is used instead, the best model's success rate drops from 34.8% to 19.1% (Appendix C.6).

What would settle it

Re-run MissionBench's 30-mission test split with a newer or larger MLLM; if any model exceeds 84.4% success, the central gap claim collapses. More locally, recompute success rates using exact string matching for reporting missions: the paper already reports this drops the best model from 34.8% to 19.1%, so the headline 'fewer than 35%' depends on the soft judge.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the 35% ceiling is real, general-purpose MLLMs are not yet reliable as zero-shot drone operators in simulated missions.
  • The consistent scaling trend suggests larger models will keep improving mission-level ability without task-specific training, which the paper flags as both an opportunity and a safety risk.
  • The OSR-SR gap implies that improving final reporting and termination decisions—knowing when to stop and what to report—may matter as much as navigation.
  • Closed-loop evaluation surfaces failure modes invisible to static perception benchmarks, so future embodied AI evaluations should adopt this protocol.
  • Existing mission-level benchmarks that decompose tasks into isolated subgoals cannot reveal these compounding failures; whole-mission evaluation is necessary.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct extension: re-running the 30-mission test split with an explicit instruction to 'if you are close but cannot read the target, move closer before reporting' may lift SR for models with OSR > SR, isolating whether the bottleneck is policy or perception.
  • The LLM-as-judge's 90% similarity threshold is the most sensitive design choice; a public leaderboard reporting both soft and exact-match success rates would make the headline claim robust.
  • Because patrol missions reward 20m-buffer IoU against a human trajectory, a coverage-based metric (area actually swept) might change model rankings on route-following.
  • MissionBench missions start with the target partially visible, so results may understate the difficulty of pure search; extending to missions with no initial cue would test search ability separately.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. MissionBench introduces a benchmark and closed-loop evaluation protocol for frozen, off-the-shelf MLLMs acting as zero-shot aerial mission agents. The benchmark contains 120 missions across five Unreal Engine 5 / Cosys-AirSim environments and four task families (reporting, inspection, manipulation, patrol). Each agent receives a single natural-language instruction, an egocentric RGB image stream, pose/action history, and must output structured reasoning, bounding box, an action with continuous magnitude, an optional report, and a DONE flag. The paper evaluates 22 open- and closed-source MLLMs and reports that the strongest model, Gemini 3.1 Pro, succeeds on 34.8% of missions versus 84.4% for human keyboard control and 70.0% for humans using the same constrained VLM interface. Additional analyses compare success rate with oracle success rate, mission progress, collision rate, step efficiency, and a single-frame perception proxy, and ablate temperature, resolution, history length, and required output fields. The central claims are that mission-level aerial capability is far from human-level, that scaling improves zero-shot embodied performance, and that mission-level success requires capabilities beyond spatial perception.

Significance. If the measurement is accepted, MissionBench is a valuable contribution: it is, to my knowledge, the first aerial benchmark to combine continuous 4-DoF control, mission-level success criteria, and explicit reporting in a single closed-loop episode, and it evaluates a broad and current set of MLLMs. The paper's methodological strengths are considerable: human baselines under both natural keyboard control and the exact constrained MLLM interface; three runs per model with standard deviations reported in Appendix B.2; an LLM-as-judge validated against 150 human-annotated reporting cases; task-specific success criteria beyond simple distance thresholds; and a reserved held-out split for future fine-tuning. The failure-mode taxonomy and the weak correlation between a static perception proxy and mission-level performance support the paper's argument that closed-loop, mission-level evaluation reveals failures invisible to single-frame perception benchmarks. However, the headline quantitative claim is sensitive to the reporting success criterion, and the main results table contains an aggregation inconsistency that must be resolved before the numbers can be taken at face value.

major comments (2)
  1. [C.6 / Table C6 / Table 3 / Abstract] The abstract's central claim that the strongest model 'succeeds on fewer than 35% of missions' is materially determined by the LLM-as-judge soft-matching rule for reporting missions. The paper itself states that replacing the judge with exact string matching drops Gemini 3.1 Pro's SR from 34.8% to 19.1%. The judge prompt in Table C6 explicitly permits 'if one or two numbers/letters differ' to count as success. For license-plate and street-name reporting, a one-character error is not a formatting artifact; it is a change in the task's correctness criterion. The 150-sample human validation (precision 1.00, recall 0.94 vs exact-match recall 0.44) is a genuine and commendable check, but it is not broken down by mission type or output-string length, and the human baselines in Table 3 are not reported under the exact-match criterion. Since reporting missions are 17/30 of the Test split, the he
  2. [Table 3 / §4.1] The stated protocol is three runs per mission on a 30-mission test split, so each mission's success count is 0, 1, 2, or 3 out of 3, and the overall SR should be a multiple of 1/90 ≈ 1.11%. Several reported SR values cannot arise from this granularity: GPT-5.4 SR=2.0%±0.0, GPT-5.4 Mini SR=1.0%±2.2, and Gemini 3.1 Flash Lite SR=6.9%±0.0 are not integer multiples of 1/90. If the aggregation is per-mission (e.g., a mission counts as successful if any run succeeds, or if a majority succeeds), that convention must be stated explicitly; if the table reports averages of per-mission success rates, the granularity is different and the standard deviations need corresponding correction. This is not a cosmetic issue: the headline 34.8% and the per-model rankings are the core empirical payload of the paper.
minor comments (4)
  1. [D.3, system prompt] Typos: 'angel' should be 'angle' in the line 'max_angle_step = 3 x {max_step_size}', and 'direclty' should be 'directly' in the format-reminder block. There is also 'in termns of' later in the per-step prompt.
  2. [§4.3 / Table B1] The perception proxy is measured on a 10-mission Proxy split, all in the Neighborhood environment and with mixed mission types, while the mission-level metrics used for correlation come from the 30-mission Test split with different environments and composition. Reporting confidence intervals or per-environment correlations would make the conclusion 'perception is not the primary bottleneck' more robust.
  3. [A.1 / Conclusion] Appendix A.1 appropriately acknowledges the limited scale and sim-to-real gap, but the main text and abstract do not qualify the 'fewer than 35%' claim with the soft-match caveat. A one-sentence qualification in the abstract or conclusion would align the headline with the evidence.
  4. [Table B5] Patrol and manipulation rows rest on only 3 missions each. For example, the human keyboard patrol SR of 88.9% corresponds to 8 of 9 episodes across 3 missions. The small per-type sample size should be noted in the main text when discussing task-family differences.

Circularity Check

0 steps flagged

No significant circularity: MissionBench's headline SR is measured against human-collected ground truth with a validated LLM-as-judge; the exact-match sensitivity is an evaluation-threshold effect, not a fitted input renamed as a prediction.

full rationale

The paper's central claims are empirical measurements, not derivations whose inputs include the outputs. Success Rate for reporting missions uses an LLM-as-judge with a 90% similarity threshold (Sec. C.6), and the paper transparently reports that exact string matching lowers Gemini 3.1 Pro's SR from 34.8% to 19.1%. This is a hand-set evaluation criterion, not a parameter fitted to reproduce the headline result; the judge is independently validated against 150 human-annotated comparisons (precision 1.00, recall 0.94). Inspection, manipulation, and patrol success are rule-based against human-collected ground-truth poses and trajectories (Secs. 3.3-3.4). The perception-proxy analysis (Sec. 4.3) uses a disjoint Proxy split rather than the Test split, so the correlation with mission performance is not an artifact of shared data. No self-citation is load-bearing: the cited simulator, benchmarks, and prior works are external and do not supply the paper's conclusions. The limitations section appropriately flags the simulated setting, scale, and task imbalance. No equation defines a predicted quantity in terms of itself, and no fitted parameter is relabeled as a prediction. The skeptical concern about threshold-dependence is a correctness/robustness issue, not a circularity issue.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central measurement depends on four hand-set evaluation parameters and on the assumption that simulation, human reference trajectories, and parser reliability faithfully represent mission success. No new physical entities are introduced.

free parameters (5)
  • dmax: per-mission max step size = one quarter of start-to-final pose distance (mission-dependent)
    Set automatically in §4.1 to make missions solvable within the action budget; it controls how finely the agent can position itself and directly affects success.
  • Success thresholds (tau_d=5m, tau_psi=15deg, patrol buffer=20m, IoU>0.5) = 5 m, 15°, 20 m, 0.5
    Hand-set criteria in §3.4 determine SR; different tolerances would change the headline 34.8%.
  • Tmax action budget = 20 steps (50 for patrol)
    Hand-set in §4.1; interacts with dmax to determine whether missions are solvable.
  • LLM-as-judge similarity threshold = 90% similarity for alphanumeric reports
    Set in §C.6; switching to exact match drops Gemini 3.1 Pro SR from 34.8% to 19.1% (Table C6).
  • Sampling temperature = 0.7
    Chosen in §4.1; ablation in Table 4a shows SR ranges 40.0–57.1 across temperatures, so the choice affects results.
axioms (4)
  • domain assumption Cosys-AirSim/UE5 simulation is a faithful enough proxy for real aerial mission difficulty
    All results are measured in simulation; authors acknowledge a sim-to-real gap in A.1. If simulation omits sensor noise/dynamics, absolute capability is overstated.
  • domain assumption The MLLM's structured output is parsed unambiguously into executable waypoints
    §3.1 assumes 'the selected high-level action is parsed into an executable waypoint'; parse failures are indistinguishable from model failures in reported metrics.
  • domain assumption Human reference trajectories and terminal states are valid ground truth for mission success
    §3.3 defines ground truth from one human pilot per mission, with multiple valid trajectories acknowledged but evaluated against single reference; alternative valid paths may be penalized.
  • ad hoc to paper Fixed start poses that make the target partially visible are an appropriate difficulty control
    §3.3 states missions start so the target is partially visible but requires deliberate planning; this choice sets difficulty and excludes unconstrained search.

pith-pipeline@v1.3.0-alltime-deepseek · 23250 in / 13161 out tokens · 124730 ms · 2026-08-01T06:02:46.421445+00:00 · methodology

0 comments
read the original abstract

Multimodal Large Language Models (MLLMs) are emerging as core reasoning modules for embodied agents, yet it remains unclear how well general-purpose models can solve long-horizon embodied tasks from a single high-level instruction. We introduce MissionBench, a benchmark for mission-level evaluation of MLLMs in aerial 3D environments. It comprises 120 missions across five simulated 3D environments and four task families. Agents must autonomously plan, navigate, and report outcomes using only egocentric observations and its action history, without aerial-specific fine-tuning. Across 22 open- and closed-source MLLMs, the strongest model succeeds on fewer than 35% of missions compared to 84.4% human performance, highlighting the difficulty of multi-step embodied tasks. Despite large variations between model families, we observe gains from scaling, indicating that larger general-purpose models possess stronger zero-shot embodied capabilities. Our analysis shows that mission-level competence requires coordinating multiple capabilities beyond spatial perception, including multi-step planning and adaptive reasoning. This motivates closed-loop evaluation and highlights both the promise and risk of scaling-driven improvements for embodied AI.

Figures

Figures reproduced from arXiv: 2607.22014 by Ishaan Bhimwal, Jona Ruthardt, Ryousuke Yamada, Suman Navaratnarajah, Taehyoung Kim, Wolfram Burgard, Yannik Blei, Yuki M Asano.

Figure 1
Figure 1. Figure 1: MissionBench evaluates MLLMs on mission-level aerial reasoning across four task families: Reporting, Inspection, Manipulation, and Patrol, and five types of environments. Each episode begins with a single natural-language directive. The agent must then iteratively decide where to go, how to position itself, and what results to report to complete the mission. ∗Equal contribution. 1 arXiv:2607.22014v1 [cs.AI… view at source ↗
Figure 2
Figure 2. Figure 2: The MissionBench closed-loop evaluation pipeline. The MLLM receives a mission instruction and egocentric images and outputs a structured response; actions are parsed into executable waypoints until the agent signals completion. The evaluation metrics Mission Progress (MP), Success Rate (SR), Oracle Success Rate (OSR), Collision Rate (CR) and Step Efficiency (Eff) are described in 3.4 and AirSimNH. Together… view at source ↗
Figure 3
Figure 3. Figure 3: Example trajectories. The human ground truth (green) is shown together with the trajectories of three MLLMs. The end point of each trajectory is indicated by a green circle (successful mission) or a red cross (failed mission). We examine the causes of these failure cases in Section B.3. (a) Sampling temperature. Temperature T SR MP 0.0 40.0 68.1 0.7 57.1 68.4 1.0 46.7 59.8 (b) Input resolution. Resolution … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 7 linked inside Pith

  1. [1]

    Do as i can, not as i say: Grounding language in robotic affordances

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022

    Pith/arXiv arXiv 2022

  2. [2]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko S \"u nderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 3674--3683, 2018

    2018

  3. [3]

    Objectnav revisited: On evaluation of embodied agents navigating to objects

    Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans. Objectnav revisited: On evaluation of embodied agents navigating to objects. arXiv preprint arXiv:2006.13171, 2020

    Pith/arXiv arXiv 2006

  4. [4]

    Neusis: A compositional neuro-symbolic framework for autonomous perception, reasoning, and planning in complex uav search missions

    Zhixi Cai, Cristian Rojas Cardenas, Kevin Leo, Chenyuan Zhang, Kal Backman, Hanbing Li, Boying Li, Mahsa Ghorbanali, Stavya Datta, Lizhen Qu, et al. Neusis: A compositional neuro-symbolic framework for autonomous perception, reasoning, and planning in complex uav search missions. IEEE Robotics and Automation Letters, 2025

    2025

  5. [5]

    Typefly: Flying drones with large language model

    Guojun Chen, Xiaojing Yu, Neiwen Ling, and Lin Zhong. Typefly: Flying drones with large language model. arXiv preprint arXiv:2312.14950, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Navila: Legged robot vision-language-action model for navigation

    An-Chieh Cheng, Yandong Ji, Zhaojing Yang, Zaitian Gongye, Xueyan Zou, Jan Kautz, Erdem B y k, Hongxu Yin, Sifei Liu, and Xiaolong Wang. Navila: Legged robot vision-language-action model for navigation. arXiv preprint arXiv:2412.04453, 2024 a

    Pith/arXiv arXiv 2024

  7. [7]

    Spatialrgpt: Grounded spatial reasoning in vision-language models

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models. Advances in Neural Information Processing Systems, 37: 0 135062--135093, 2024 b

    2024

  8. [8]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Unreal engine, 2019

    Epic Games . Unreal engine, 2019. URL https://www.unrealengine.com

    2019

  10. [10]

    Aerial vision-and-dialog navigation

    Yue Fan, Winson Chen, Tongzhou Jiang, Chun Zhou, Yi Zhang, and Xin Wang. Aerial vision-and-dialog navigation. In Findings of the Association for Computational Linguistics: ACL 2023, pp.\ 3043--3061, 2023

    2023

  11. [11]

    Openfly: A comprehensive platform for aerial vision-language navigation

    Yunpeng Gao, Chenhui Li, Zhongrui You, Junli Liu, Zhen Li, Pengan Chen, Qizhi Chen, Zhonghan Tang, Liansheng Wang, Penghui Yang, et al. Openfly: A comprehensive platform for aerial vision-language navigation. arXiv preprint arXiv:2502.18041, 2025

    arXiv 2025

  12. [12]

    Bedi: A comprehensive benchmark for evaluating embodied agents on uavs

    Mingning Guo, Mengwei Wu, Jiarun He, Shaoxian Li, Haifeng Li, and Chao Tao. Bedi: A comprehensive benchmark for evaluating embodied agents on uavs. ISPRS Journal of Photogrammetry and Remote Sensing, 232: 0 910--936, 2026

    2026

  13. [13]

    See, point, fly: A learning-free vlm framework for universal unmanned aerial navigation

    Chih Yao Hu, Yang-Sen Lin, Yuna Lee, Chih-Hai Su, Jie-Ying Lee, Shr-Ruei Tsai, Chin-Yang Lin, Kuan-Wen Chen, Tsung-Wei Ke, and Yu-Lun Liu. See, point, fly: A learning-free vlm framework for universal unmanned aerial navigation. In Conference on Robot Learning, pp.\ 4697--4708. PMLR, 2025

    2025

  14. [14]

    Cosys-airsim: a real-time simulation framework expanded for complex industrial applications

    Wouter Jansen, Erik Verreycken, Anthony Schenck, Jean-Edouard Blanquart, Connor Verhulst, Nico Huebel, and Jan Steckel. Cosys-airsim: a real-time simulation framework expanded for complex industrial applications. Annual Modeling and Simulation Conference (ANNSIM), 2023

    2023

  15. [15]

    Citynav: A large-scale dataset for real-world aerial navigation

    Jungdae Lee, Taiki Miyanishi, Shuhei Kurita, Koya Sakamoto, Daichi Azuma, Yutaka Matsuo, and Nakamasa Inoue. Citynav: A large-scale dataset for real-world aerial navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 5912--5922, 2025

    2025

  16. [16]

    Aerialvln: Vision-and-language navigation for uavs

    Shubo Liu, Hongsheng Zhang, Yuankai Qi, Peng Wang, Yanning Zhang, and Qi Wu. Aerialvln: Vision-and-language navigation for uavs. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 15384--15394, 2023

    2023

  17. [17]

    Unmanned aerial vehicles (uavs): Practical aspects, applications, open challenges, security issues, and future trends

    Syed Agha Hassnain Mohsan, Nawaf Qasem Hamood Othman, Yanlong Li, Mohammed H Alsharif, and Muhammad Asghar Khan. Unmanned aerial vehicles (uavs): Practical aspects, applications, open challenges, security issues, and future trends. Intelligent service robotics, 16 0 (1): 0 109--137, 2023

    2023

  18. [18]

    Flysearch: Exploring how vision-language models explore

    Adam Pardyl, Dominik Matuszek, Mateusz Przebieracz, Marek Cygan, et al. Flysearch: Exploring how vision-language models explore. NeurIPS Datasets and Benchmarks, 2025

    2025

  19. [19]

    Unrealcv: Virtual worlds for computer vision

    Weichao Qiu, Fangwei Zhong, Yi Zhang, Siyuan Qiao, Zihao Xiao, Tae Soo Kim, and Yizhou Wang. Unrealcv: Virtual worlds for computer vision. In Proceedings of the 25th ACM international conference on Multimedia, pp.\ 1221--1224, 2017

    2017

  20. [20]

    Airsim: High-fidelity visual and physical simulation for autonomous vehicles

    Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and service robotics: Results of the 11th international conference, pp.\ 621--635. Springer, 2017

    2017

  21. [21]

    Unmanned aerial vehicles (uavs): A survey on civil applications and key research challenges

    Hazim Shakhatreh, Ahmad H Sawalmeh, Ala Al-Fuqaha, Zuochao Dou, Eyad Almaita, Issa Khalil, Noor Shamsiah Othman, Abdallah Khreishah, and Mohsen Guizani. Unmanned aerial vehicles (uavs): A survey on civil applications and key research challenges. IEEE access, 7: 0 48572--48634, 2019

    2019

  22. [22]

    Alfred: A benchmark for interpreting grounded instructions for everyday tasks

    Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 10740--10749, 2020

    2020

  23. [23]

    UAVs meet LLMs : Overviews and perspectives toward agentic low-altitude mobility, 2025

    Yonglin Tian, Fei Lin, Yiduo Li, Tengchao Zhang, Qiyao Zhang, Xuan Fu, Jun Huang, Xingyuan Dai, Yutong Wang, Chunwei Tian, Bai Li, Yisheng Lv, Levente Kovács, and Fei-Yue Wang. UAVs meet LLMs : Overviews and perspectives toward agentic low-altitude mobility, 2025

    2025

  24. [24]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai C Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processing Systems, 37: 0 87310--87356, 2024

    2024

  25. [25]

    Towards realistic uav vision-language navigation: Platform, benchmark, and methodology

    Xiangyu Wang, Donglin Yang, Ziqin Wang, Hohin Kwan, Jinyu Chen, Wenjun Wu, Hongsheng Li, Yue Liao, and Si Liu. Towards realistic uav vision-language navigation: Platform, benchmark, and methodology. International Conference on Learning Representations (ICLR), 2025

    2025

  26. [26]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, 2022

    2022

  27. [27]

    Uav-on: A benchmark for open-world object goal navigation with aerial agents

    Jianqiang Xiao, Yuexuan Sun, Yixin Shao, Boxi Gan, Rongqiang Liu, Yanjin Wu, Weili Guan, and Xiang Deng. Uav-on: A benchmark for open-world object goal navigation with aerial agents. In Proceedings of the 33rd ACM International Conference on Multimedia, pp.\ 13023--13029, 2025

    2025

  28. [28]

    Aeroverse: Uav-agent benchmark suite for simulating, pre-training, finetuning, and evaluating aerospace embodied world models

    Fanglong Yao, Yuanchang Yue, Youzhi Liu, Xian Sun, and Kun Fu. Aeroverse: Uav-agent benchmark suite for simulating, pre-training, finetuning, and evaluating aerospace embodied world models. arXiv preprint arXiv:2408.15511, 2024

    arXiv 2024

  29. [29]

    Esarbench: A benchmark for agentic uav embodied search and rescue

    Daoxuan Zhang, Ping Chen, Jianyi Zhou, and Shuo Yang. Esarbench: A benchmark for agentic uav embodied search and rescue. arXiv preprint arXiv:2605.01371, 2026

    Pith/arXiv arXiv 2026

  30. [30]

    Navid: Video-based vlm plans the next step for vision-and-language navigation

    Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, and He Wang. Navid: Video-based vlm plans the next step for vision-and-language navigation. arXiv preprint arXiv:2402.15852, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Is your vlm sky-ready? a comprehensive spatial intelligence benchmark for uav navigation

    Lingfeng Zhang, Yuchen Zhang, Hongsheng Li, Haoxiang Fu, Yingbo Tang, Hangjun Ye, Long Chen, Xiaojun Liang, Xiaoshuai Hao, and Wenbo Ding. Is your vlm sky-ready? a comprehensive spatial intelligence benchmark for uav navigation. arXiv preprint arXiv:2511.13269, 2025

    arXiv 2025

  32. [32]

    Towards learning a generalist model for embodied navigation

    Duo Zheng, Shijia Huang, Lin Zhao, Yiwu Zhong, and Liwei Wang. Towards learning a generalist model for embodied navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 13624--13634, 2024

    2024

  33. [33]

    Navgpt-2: Unleashing navigational reasoning capability for large vision-language models

    Gengze Zhou, Yicong Hong, Zun Wang, Xin Eric Wang, and Qi Wu. Navgpt-2: Unleashing navigational reasoning capability for large vision-language models. In European Conference on Computer Vision, pp.\ 260--278. Springer, 2024

    2024