{"id":"96d5ecae-d3e5-434f-8115-7bf636642145","arxiv_id":"2505.15725","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":8.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper presents UAV-Flow, the first real-world benchmark for language-conditioned fine-grained UAV control, with 30K expert flight episodes and a simulation suite, and reports that VLA models outperform VLN baselines.","lead":"This paper introduces UAV-Flow, a real-world benchmark and dataset for teaching drones to follow short, specific spoken commands such as 'pass the tree on the left' by imitating expert pilot trajectories. It also provides a simulation testbed and deployed a vision-language-action model on a real drone, reporting that such models beat navigation-style baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quantitative VLA-over-VLN claim rests on an unvalidated simulator with manual success checks; real-world 'close imitation' is demonstrated only qualitatively.","rationale":"The reader's weakest-assumption diagnosis—that UAV-Flow-Sim is an unvalidated closed-loop proxy—is correct and is the most load-bearing concern, because all quantitative comparisons between VLN and VLA models are conducted only in that simulator. I agree that the benchmark resource (real-world dataset, task formulation, deployment framework) is a genuine contribution and that the paper should not be rejected outright. However, I would sharpen the concern by also highlighting that even the real-world deployment, which is the basis for the claim of 'direct deployment without sim-to-real gap' and 'close imitation,' is evaluated only through a few qualitative demo trajectories, with no success metric or trajectory-error measurement. The manual success check in Sec 3.2 and the small 273-trajectory simulation test set further weaken the quantitative ranking. These issues are methodological and addressable, so a CONDITIONAL verdict is appropriate. My agreement_with_reader is 'partial' rather than 'agree' because the reader emphasizes simulation fidelity, whereas I see the missing quantitative real-world evaluation as an equally load-bearing gap for the 'close imitation' claim; both are part of the same underlying validation deficiency. A concrete test—running the same Sec 4.2 models on the real platform with pre-registered geometric success criteria and comparing SR/NDTW to simulator results—would settle whether the simulation-based ranking transfers and whether the imitation-quality claim can be quantified.","tokens_in":12268,"tokens_out":4567,"duration_ms":43169,"concrete_test":"Deploy the same Pi-0-UAV and OpenVLA-UAV models evaluated in Sec 4.2 on the real UAV for a pre-registered set of trials (e.g., 30 trials per each of the 10 Flow task types). Score success using explicit geometric criteria (e.g., endpoint within a specified distance of the target side, orbit radius within a tolerance, correct side of obstacle) and compute SR and NDTW against the expert reference trajectories. If the real-world SR/NDTW values are comparable to the simulator results from Sec 4.2 and the OpenVLA-vs-Pi-0 ranking matches, then the comparative VLA-over-VLN conclusion survives the simulation-fidelity concern; if the ranking flips or real-world performance is substantially worse, the central comparative claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—that VLA models are superior to VLN baselines in the Flow setting—is supported only by experiments in UAV-Flow-Sim (Sec 2.3, Sec 4.2). That simulation is asserted to have high fidelity to real flight because it 'closely mimics the position-mode control used in real-world UAV remote controllers,' but no validation is provided comparing simulated trajectories, visual grounding, or control dynamics against real UAV behavior. The paper itself later concedes that 'simulation environments still exhibit discrepancies from the real world in both visual perception and flight control dynamics' (Sec 2.3), directly undermining the high-fidelity premise. The closed-loop success metric in Sec 3.2 is based on 'manual inspection of whether the trajectory semantically satisfies the instruction,' with no pre-registered rubric, no inter-annotator agreement, and no error bars or repeated seeds reported. The simulation test set is only 273 annotated trajectories (Sec 2.4), so per-task sample sizes are small. Meanwhile, the real-world deployment in Sec 4.3 is qualitative: representative flight demos are visualized, but no success rate, trajectory error, NDTW, or comparison to expert pilot trajectories is reported. The paper's own Limitations section states that 'it remains challenging to conduct systematic real-world experiments and establish consistent evaluation metrics.' Consequently, both headline conclusions—'VLA models are superior to VLN baselines' and 'close imitation of precise, expert-level flight trajectories'—are not supported by the quantitative evidence presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces UAV-Flow, a benchmark for the 'Flying-on-a-Word' (Flow) task, defined as short-range, language-conditioned fine-grained UAV trajectory control via imitation learning. The main artifact is a real-world dataset of 30,692 human-piloted flight trajectories with synchronized FPV video, 6-DoF state, and atomic language instructions collected across three campuses, together with a 10,109-trajectory UnrealCV simulation dataset and a 273-trajectory annotated closed-loop test set. The paper also contributes a ground-drone collaborative deployment framework with a look-ahead action-chunk mechanism and a closed-loop simulation evaluation using success rate (SR) and NDTW metrics. The headline empirical claims are that VLA models (OpenVLA-UAV, Pi-0-UAV) outperform VLN baselines (Seq2Seq-UAV, CMA-UAV, Travel-UAV) on Flow, and that the system supports direct real-world deployment without a sim-to-real gap, demonstrated by qualitative flight demos.","tokens_in":12526,"tokens_out":5524,"duration_ms":49246,"significance":"The real-world dataset is a substantial new resource: to my knowledge, no comparable public dataset pairs atomic language instructions with centimeter-level expert UAV trajectories in open outdoor environments, and the paper states that data, code, and flight demos will be released. The task formulation separating motion intent understanding from spatial context grounding is reasonable, and adapting OpenVLA and Pi-0 to aerial control through a ground-station look-ahead mechanism is a useful engineering contribution. However, the two headline conclusions rest on evidence of unequal strength: the VLA-over-VLN result comes from a self-built simulator with a manual success metric and no reported variance, while the 'without sim-to-real gap' claim is contradicted by the paper's own caveats and by the qualitative nature of the real-world evaluation. The benchmark itself remains valuable if these claims are appropriately qualified and the simulator is validated.","major_comments":[{"comment":"The central quantitative claim that VLA models are superior to VLN baselines is supported only by closed-loop simulation scores whose success metric is determined by manual inspection of whether the trajectory 'semantically satisfies the instruction,' with no error bars, no repeated seeds, no pre-specified rubric, and no inter-annotator agreement reported. With a test set of only 273 annotated trajectories (Sec. 2.4), per-task sample sizes are small, so the reported SR and NDTW differences may not be statistically reliable. Please report means and variances over multiple evaluation runs, a pre-registered success rubric, and per-task sample sizes, or explicitly restrict the superiority claim to the simulation setting.","section":"Sec. 4.2, Sec. 3.2, Sec. 2.4"},{"comment":"The paper's own statements undermine the abstract's claims of 'direct deployment without sim-to-real gap' and 'close imitation' of expert trajectories. Sec. 2.3 concedes that 'simulation environments still exhibit discrepancies from the real world in both visual perception and flight control dynamics,' and Sec. 4.3 states that 'it remains challenging to adopt quantitative evaluation metrics in real-world scenarios'; the Limitations section similarly says systematic real-world evaluation remains difficult. The real-world deployment is therefore evidenced only by a few visualized demos. I ask for quantitative real-world trajectory error (e.g., endpoint error or NDTW relative to expert reference trajectories) on at least a small evaluation set, or for the central claims to be limited to the simulation domain.","section":"Sec. 2.3, Sec. 4.3, Limitations"},{"comment":"Pi-0-UAV uses an action chunk size (Horizon Steps) of 10, and this value is never swept. Since the VLN baselines do not use action chunking, the VLA-over-VLN comparison could be sensitive to this design choice; a different chunk size might change the relative ordering on specific task types. Please include a sensitivity analysis over chunk size and report whether the qualitative conclusion is robust.","section":"Sec. 4.2, Table 4, Appendix B"},{"comment":"The adapted VLN baselines (Seq2Seq-UAV, CMA-UAV) are trained with a single MSE loss and without state input, while the VLA baselines use flow matching or tokenized action prediction with different optimization budgets; the paper also notes that adding state input to the RN N models hurt performance. This makes it hard to separate 'paradigm superiority' from implementation quality. Please provide a matched-capacity comparison or additional evidence that these adaptations are the strongest reasonable representatives of the VLN paradigm.","section":"Sec. 4.1, Appendix C"}],"minor_comments":[{"comment":"The heading 'Flying-on-a-word (Flow) Colosso' appears to contain a typo; 'Colosseo' is used in the paper title and elsewhere.","section":"Sec. 3 heading"},{"comment":"The token 'UAV' is inconsistently typeset as 'UA V' in many places, which hurts readability.","section":"Throughout"},{"comment":"The radar charts report SR and NDTW without numerical values; a companion table with exact numbers and per-task sample sizes would make the benchmark results reproducible and easier to compare.","section":"Fig. 7 and Fig. 8"},{"comment":"The 273-trajectory simulation test set is not described in terms of how it was split from the 10,109 simulated trajectories or whether the test instructions overlap with the training instruction templates; this should be clarified.","section":"Sec. 2.4"},{"comment":"The LLM-based 'Open Vocabulary Command Set' is described as generated with GPT models, but no human consistency check or filtering statistics are reported for these generated instructions; please specify how they were validated.","section":"Sec. 2.2"},{"comment":"The claim that open-vocabulary training 'enhances language generalization and even improves performance on certain tasks' is made without supporting numerical comparisons or significance tests; please provide the underlying numbers.","section":"Sec. 4.2"}],"recommendation":"major_revision","confidential_remarks":"The benchmark artifact is substantial and likely to be useful to the community, and the authors are appropriately transparent in the Limitations section. However, all baselines, the simulator, and the evaluation metrics come from the same group, so independent validation would greatly increase confidence in the VLA-over-VLN conclusion. The Travel baseline is prior work by the same authors; the paper should ensure this is explicitly stated in the related-work discussion. My main concern is that the abstract overstates what the evidence supports: the quantitative comparison is simulation-only and the real-world demonstration is qualitative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is the resource: a 30K-episode real-world dataset of language-annotated, fine-grained UAV trajectories, collected across three campuses by experienced pilots, with synchronized video and 6-DoF state. That is new. Prior work in aerial VLN is long-horizon and mostly simulated; this paper carves out a different task, Flow, focused on short-range, reactive flight controlled by atomic language commands. The task formulation is clean, and the ground-drone deployment framework is a sensible engineering answer to onboard compute limits. The paper also deserves credit for being upfront about its limitations and for acknowledging that simulation still deviates from reality in perception and dynamics.\n\nWhere it goes soft is the evidence for the central quantitative claim. The result that VLA models beat VLN baselines comes entirely from UAV-Flow-Sim, a self-built simulator. Success rate is determined by manual inspection of rendered trajectories, with no rubric, no inter-annotator agreement, and no error bars or repeated seeds. The simulation test set is only 273 trajectories, so per-task samples are small. The paper asserts the simulator closely mimics real position-mode control, but offers no validation against real flight trajectories or visual grounding. The real-world deployment is qualitative only: a few demo flights, no metrics. The Limitations section concedes exactly this, saying systematic real-world evaluation remains challenging. So the two headline claims, VLA superiority and close imitation of expert trajectories, are not yet backed by the quantitative evidence. They may be true, but the paper does not demonstrate it.\n\nIs this fatal? For the benchmark itself, no. The dataset and task are the product, and they do not depend on the baseline comparison. A future user can adopt the benchmark and run their own evaluation. The weak simulator validation is a problem for the paper's conclusions, but it can be fixed by releasing artifacts, adding error bars, and tempering the claims. The paper provides enough pipeline detail to be credible, and the authors' own acknowledgment of sim-to-real gaps shows they are not hiding the issue.\n\nWho is this for? People working on language-conditioned UAV control, VLA, or imitation learning. It deserves a serious referee, and I would send it to peer review, but with the expectation of heavy revision: release the data and code, validate the simulator, and report statistics. I would engage with it, but I would not cite the VLA-over-VLN result until the evaluation is stronger.","headline":"A genuinely new real-world dataset and task formulation for language-conditioned fine-grained UAV control, but the headline VLA-over-VLN result rests on an unvalidated simulator and should be treated as provisional until the evaluation is tightened.","tokens_in":701,"tokens_out":746,"would_cite":true,"duration_ms":20389,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces UAV-Flow, the first real-world benchmark for language-conditioned fine-grained UAV control, and claims that vision-language-action models outperform vision-language-navigation baselines at imitating expert pilot…","keywords":["Flying-on-a-Word","UAV imitation learning","vision-language-action","vision-language navigation","benchmark","real-world dataset","language-conditioned control","fine-grained trajectory control"],"falsifier":"Run a head-to-head closed-loop evaluation on real outdoor flights, comparing Pi-0-UAV and OpenVLA-UAV against Seq2Seq-UAV and CMA-UAV on a fixed set of object-interactive commands, using NDTW and trajectory-semantic-success metrics computed from RTK ground truth; if the VLN-derived models achieve comparable or higher scores, the paper's central claim fails.","tokens_in":12077,"feed_emoji":"🚁","tokens_out":3075,"duration_ms":26594,"temperature":0.7,"pith_summary":"The paper formalizes Flying-on-a-Word (Flow), a task where a UAV must translate a short, atomic language instruction into a fine-grained, short-range flight trajectory that reacts to the visual scene. To support this task, the authors build UAV-Flow, a real-world dataset of 30,692 expert-pilot flight episodes collected across three large campuses, paired with a closed-loop simulation suite of 10,109 episodes. The central claim is that UAVs can learn to imitate expert-level flight patterns from language instructions and be deployed directly on real hardware without a sim-to-real gap. Quantitative experiments in simulation compare adapted vision-language-navigation (VLN) models against vision-language-action (VLA) models, and the paper argues that VLA models are superior in the Flow setting because they provide stronger visual grounding and fine-grained control. Real-world deployment, shown qualitatively with flight demos, demonstrates that the proposed ground-drone collaborative framework can run large models onboard with a look-ahead mechanism for continuous motion.","feed_headline":"VLA models beat VLN baselines for fine-grained drone control","feed_subtitle":"First real-world benchmark with 30K flight episodes lets drones imitate expert pilot trajectories from natural language.","key_machinery":"The core machinery is the Flow policy formulation πθ : (St, Ot, I) → at, mapping the UAV's 6-DoF state, egocentric visual observation, and natural language instruction to low-level control actions at each time step. Two task types—primitive motion commands and object-interactive commands—are used to evaluate motion intent understanding and spatial context grounding separately. For real-world deployment, the paper introduces a ground-drone collaborative framework with a Globally-Aligned Continuous Motion scheme: the UAV streams FPV video and state to a ground station, and a look-ahead mechanism fuses predicted target points with the current state to filter out already-passed targets and ensure delay-free flight. Evaluation in the closed-loop simulation environment uses Success Rate (SR), judged by manual inspection of whether the trajectory semantically satisfies the instruction, and Normalized Dynamic Time Warping (NDTW) comparing predicted and reference 6D trajectories.","core_discovery":"The paper's central discovery is that language-conditioned fine-grained UAV control can be cast as an imitation learning problem, and that in this setting VLA models—originally built for robotic manipulation—consistently outperform VLN models adapted from navigation. The benchmark itself, UAV-Flow, provides the first real-world language-vision-action dataset for this task, with each trajectory recorded by professional pilots following atomic instructions such as 'pass through the tree from the left side' or 'move 4.0 meters at 30 degrees to the right.' The authors find that RNN-based VLN models fail to stop at the right point and inherit motion direction, while VLA models such as OpenVLA-UAV and Pi-0-UAV show stronger spatial perception and motion execution. They also show that training on open-vocabulary command sets, generated by LLM-based language diversification, does not hurt success rates and can even improve language generalization.","pith_inferences":["One implicit consequence is that the Flow task formulation, with its short-range reactive trajectories, may serve as a lower-level primitive that can later be composed with long-horizon planning systems—the paper's own limitations section notes this integration as future work.","The paper's simulation results suggest that the choice of visual tokenization capacity matters: Travel-UAV, built on a compact vision encoder that uses only 17 tokens, underperforms on 'turn to face the target' tasks, implying that fine-grained visual grounding is a bottleneck that could be tested by ablating vision encoder size in VLA models.","A testable extension would be to measure whether the same VLA superiority holds under varying weather, lighting, and dynamic obstacles in real-world conditions, since the quantitative evidence is currently limited to the simulator and qualitative real-world demos.","The NDTW metric with 6D state vectors could be adapted to assess trajectory safety or energy efficiency, not just similarity to a reference path, extending the benchmark beyond imitation fidelity."],"forward_implications":["If the central claim holds, language-driven UAV control shifts from long-horizon navigation benchmarks to reactive, fine-grained flight control, opening a new evaluation paradigm for embodied language agents in 3D space.","VLA models originally developed for tabletop manipulation can be transferred to aerial platforms with relatively minor structural adaptation, suggesting that manipulation-oriented action prediction is a stronger starting point than navigation-oriented discrete action selection for reactive control.","The ground-drone collaborative deployment strategy, with look-ahead chunk-wise action prediction, provides a practical template for running large vision-language models on resource-constrained flying platforms in open environments.","The open-vocabulary command set generated via LLM diversification could make UAV control systems more robust to natural language variation, improving generalization beyond fixed command templates.","The benchmark's combination of real-world training data and simulation-based closed-loop evaluation offers a reusable pipeline for other fine-grained embodied control tasks where sim-to-real gaps are severe."],"supporting_citations":[{"why":"Provides the Seq2Seq-VLN baseline architecture that the paper adapts to continuous UAV pose regression as Seq2Seq-UAV.","marker":"[10]"},{"why":"Provides the CMA-VLN baseline with cycle attention, adapted as CMA-UAV in the benchmark comparison.","marker":"[21]"},{"why":"Supplies the Travel-MLLM baseline that the paper adapts into Travel-UAV by restructuring text input and output to predict 6-DoF poses.","marker":"[7]"},{"why":"Provides the OpenVLA vision-language-action model that serves as one of the two main VLA baselines.","marker":"[22]"},{"why":"Provides the Pi-0 flow-matching vision-language-action model, the second main VLA baseline and the one deployed on real hardware.","marker":"[23]"},{"why":"Supplies the UnrealCV simulation environment used to build the UAV-Flow-Sim dataset and the closed-loop evaluation suite.","marker":"[16]"},{"why":"Defines the DTW-based metric that the paper extends to NDTW for measuring trajectory similarity in the Flow evaluation.","marker":"[26]"},{"why":"Provides AerialVLN, the prior aerial VLN dataset that the paper contrasts to highlight the difference between long-horizon discrete navigation and the Flow setting.","marker":"[5]"}],"fun_headline_variants":["VLA beats VLN in new real-world benchmark for drone control","First benchmark for drones that fly on a word","Language-guided micro-flight: VLA models outperform navigation models","Drones imitate expert pilots from natural language: UAV-Flow","New UAV benchmark reveals VLA advantage in fine-grained flight"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quantitative comparison between VLA and VLN models is performed in a self-built simulator that is assumed to faithfully capture the closed-loop reactive flight dynamics and visual grounding of real UAV flight, while real-world deployment is evaluated only qualitatively.","fun_headline_variants_meta":{"raw":{"variants":["VLA beats VLN in new real-world benchmark for drone control","First benchmark for drones that fly on a word","Language-guided micro-flight: VLA models outperform navigation models","Drones imitate expert pilots from natural language: UAV-Flow","New UAV benchmark reveals VLA advantage in fine-grained flight"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000418,"raw_usage":{"total_tokens":2161,"prompt_tokens":958,"completion_tokens":1203,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":1121}},"tokens_in":574,"tokens_out":1203,"duration_ms":10140,"temperature":1.0,"reasoning_tokens":1121,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:11:19.198316+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a head-to-head closed-loop evaluation on real outdoor flights, comparing Pi-0-UAV and OpenVLA-UAV against Seq2Seq-UAV and CMA-UAV on a fixed set of object-interactive commands, using NDTW and trajectory-semantic-success metrics computed from RTK ground truth; if the VLN-derived models achieve comparable or higher scores, the paper's central claim fails.","supporting_citations":[{"cited_title":"Beyond the nav-graph: Vision-and-language navigation in continuous environments","cited_arxiv_id":null,"evidence_quote":"Provides the CMA-VLN baseline with cycle attention, adapted as CMA-UAV in the benchmark comparison."},{"cited_title":"Unrealcv: Virtual worlds for computer vision","cited_arxiv_id":null,"evidence_quote":"Supplies the UnrealCV simulation environment used to build the UAV-Flow-Sim dataset and the closed-loop evaluation suite."},{"cited_title":"General evaluation for instruction conditioned navigation using dynamic time warping","cited_arxiv_id":null,"evidence_quote":"Defines the DTW-based metric that the paper extends to NDTW for measuring trajectory similarity in the Flow evaluation."},{"cited_title":"Aerialvln: Vision- and-language navigation for uavs","cited_arxiv_id":null,"evidence_quote":"Provides AerialVLN, the prior aerial VLN dataset that the paper contrasts to highlight the difference between long-horizon discrete navigation and the Flow setting."}],"review_version":1}