{"id":"833af534-eca0-4a27-ab72-c0753f9ef2df","arxiv_id":"2506.07509","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An open-source ROS2/PX4 framework using locally hosted LLMs and VLMs enables natural language drone commands, with the best simulated mission success rate at 40%.","lead":"This paper introduces an open-source system that lets people control PX4 drones using natural language, with AI models running locally instead of in the cloud. It reports that in simulations the best model combination completed search-and-approach missions 40% of the time, while some language models generated commands more reliably than others.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 100% valid-command results measure format compliance only; command magnitudes and VLM detection accuracy are explicitly unevaluated, so the 40% mission success rate is the only correctness signal and cannot support the conclusion's phrase 'accurate natural language-based command execution'.","rationale":"The reader's weakest assumption is the same load-bearing concern I would raise: the headline validity metrics are syntactic and do not measure whether the generated commands are numerically correct or whether the VLM correctly detects objects. The task in Eqs. (1)-(3) requires the LLM to select specific Turn and Move magnitudes from the current pose and goal coordinate, and the VLM output feeds directly into that decision. The paper candidly admits it did not evaluate arithmetic precision or detection correctness, and it also candidly explains that failures occurred due to suboptimal command values and VLM false positives/negatives. That is enough to invalidate the conclusion's phrase 'accurate natural language-based command execution' as a statement supported by the presented evidence. The framework contribution itself is real: open-source code, model configurations, and prompt templates are released, and the paper explicitly lists limitations such as NED/ENU coordinate transforms, RGB8 encoding, and lack of latency/path-optimality measurements. Giving credit where it is due, this is a useful integration and a small feasibility study. The issue is not the framework but the strength of the claim drawn from a 40% success rate over 20 episodes per model pairing with no baseline and no error bars. A scripted greedy baseline would settle whether the low success rate is due to the LLM/VLM choices or to intrinsic task difficulty; if the baseline succeeds at a much higher rate, then the 100% valid-command metric is clearly insufficient to support 'accurate' execution. The conditional verdict remains appropriate: the paper should either tone down the conclusion or add quantitative action-error and detection-accuracy evaluations.","tokens_in":7374,"tokens_out":3709,"duration_ms":48265,"concrete_test":"Run the same 20-episode protocol with a scripted greedy baseline that, at each step, computes the exact Turn/Move toward the goal from ground-truth pose and target location and executes it without an LLM or VLM. If the greedy baseline reaches near 100% success while the best LLM/VLM pairing reaches 40%, the gap directly measures the cost of LLM/VLM command and detection errors and demonstrates that syntactic validity is not a proxy for accurate execution. Also report per-step numeric action error (actual vs. geometrically optimal) for the Gemma episodes to localize failures.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical support consists of 100% syntactically valid Turn/Move commands, 97-100% valid binary VLM responses, and a best mission success rate of 40%. The first two metrics are format checks, not correctness checks. Section IV explicitly states: 'the LLMs were not evaluated based on the arithmetic precision of angle calculations or movement distance estimations' and 'Object detection correctness was not considered.' Yet the task defined by Eqs. (1)-(3) requires numerically appropriate Turn(theta) and Move(d) values and reliable target detection to reach the goal. The paper itself attributes mission failures to 'suboptimal command values' and false positives/negatives in object detection. Therefore a model can generate 100% 'valid' commands while consistently steering the drone away from or past the target; the 40% success rate, measured over only 20 episodes per pairing and without a baseline, is the only outcome that bears on accuracy. The conclusion's 'accurate natural language-based command execution' is thus not established by the reported evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an open-source agentic framework for natural-language control of PX4-based drones, integrating ROS2, Isaac Sim, and locally hosted Ollama models. It benchmarks four LLM families for command generation and three VLM families for scene understanding, both in simulation and on a custom quadcopter platform. The headline results are that Gemma3, Qwen2.5, and Llama-3.2 produce 100% syntactically valid Turn/Move commands; the VLM valid binary response rates range from 97% to 100%; and the best mission success rate is 40%, achieved by the Gemma3 LLM-VLM pairing over 20 episodes per model combination. A qualitative real-world demonstration is also reported using Gemma models only. The authors are candid about the small number of episodes, the lack of statistical testing, and the fact that command-value precision and object-detection correctness were not evaluated.","tokens_in":7554,"tokens_out":3702,"duration_ms":44881,"significance":"The paper's main strength is engineering integration: a modular, open-source pipeline that connects PX4 flight control, ROS2, and local open-source language and vision models, with source code, model configurations, and prompt templates released publicly. If the empirical claims were substantiated, this would be a useful step toward democratizing natural-language UAV control without cloud APIs. However, at present the headline validity metrics are format-compliance checks rather than correctness checks, and the only task-level correctness signal, a best mission success rate of 40% over 20 episodes with no baseline, is too thin to support the conclusion that the models execute commands accurately. The paper's own limitations section and small-sample caveats are appropriate, but they underscore that the central empirical claim is not yet established.","major_comments":[{"comment":"The 100% LLM valid-command metric measures syntactic format compliance only. The paper states that 'the LLMs were not evaluated based on the arithmetic precision of angle calculations or movement distance estimations' and 'Object detection correctness was not considered.' Yet the task defined by Eqs. (1)-(3) requires numerically appropriate Turn(theta) and Move(d) values and reliable target detection for the quadcopter to approach the goal. The paper itself attributes mission failures to 'suboptimal command values' and false positive/negative object detections. Therefore, the 40% mission success rate is the only reported quantity that actually bears on command accuracy, and the conclusion's phrase 'accurate natural language-based command execution' is not supported by the validity metrics.","section":"IV. Results and Discussion, Table I"},{"comment":"The best mission success rate is 40% over only 20 episodes per model pairing, with no confidence intervals, no significance tests, and no baseline (e.g., random actions or a simple heuristic like 'move toward the goal'). The authors acknowledge that the modest differences between Gemma, Qwen, and Llama 'may reflect random variation.' Without a baseline or statistical treatment, the claimed association between valid-command rate and mission success is not established. At minimum, the paper should report per-pairing success counts and exact binomial confidence intervals, and compare against a nontrivial baseline to show the LLM/VLM contributes beyond task randomness.","section":"IV. Results and Discussion, Table I"},{"comment":"The VLM 'valid detection' metric is also a format check: it records whether the model answered 'Yes' or 'No,' not whether the answer matched the true presence of the target object. The paper explicitly excludes object-detection correctness. Since the navigation policy relies on the VLM's output to decide whether to approach or search, a VLM that always answered 'Yes' could achieve high validity without providing useful information. The paper should report precision/recall or F1 against ground-truth target presence per frame or per episode, which would also quantify the false positives and negatives cited as causes of mission failure.","section":"IV. Results and Discussion, VLM evaluation"},{"comment":"The real-world component is qualitative only: Figure 4 illustrates one successful sequence, and the text says 'only Gemma was chosen' without presenting quantitative success rates, episode counts, or failure modes for the physical platform. If the abstract's claim of evaluation 'both in simulation and on a custom quadcopter platform' is to be quantified, the real-world trials need at least the same success-rate reporting as the simulation, or the paper should explicitly label the real-world portion as a feasibility demonstration rather than a performance evaluation.","section":"IV. Results and Discussion, real-world deployment"}],"minor_comments":[{"comment":"The text contains several instances of 'UA V' with an erroneous space (abstract, Index Terms, and Section II); these should be corrected to 'UAV.'","section":"Throughout"},{"comment":"Please clarify whether the '0.5 m radius' success criterion is in the horizontal plane at the fixed 1 m altitude or in full 3D space; this affects how close the quadcopter must be to the target.","section":"III. Methodology, Mission success criterion"},{"comment":"State explicitly how an episode is scored when the human operator switches to POSITION mode during a real-world trial: as a failure, an abort, or a restart? This is needed to interpret real-world success rates.","section":"III. Methodology, safety fallback"},{"comment":"The paper would benefit from an in-text example of the exact Turn/Move grammar and a sample prompt, in addition to the linked repository, to make the validity criteria reproducible without consulting external material.","section":"IV. Results and Discussion, evaluation details"},{"comment":"The limitations paragraph already acknowledges missing latency, token-usage, and path-optimality analyses; adding one or two of these quantitative metrics in the revised version would strengthen the practical contribution.","section":"IV. Results and Discussion, limitations"}],"recommendation":"major_revision","confidential_remarks":"This is a reasonable systems-integration paper with a public code release, but the empirical evaluation is currently below the evidentiary bar for the claims made. The central fix is to ground the evaluation in task-level correctness: measure the error between generated command values and the geometrically required values, report object-detection accuracy against ground truth, add a baseline and confidence intervals, and either quantify the real-world trials or clearly label them as demonstrations. These are all achievable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real contribution as an integration and a small open benchmark, and the authors are honest about many of its limits. But the central numerical claims do not support the conclusion's phrase \"accurate natural language-based command execution.\"\n\nWhat is new: combining PX4, ROS2, and locally hosted Ollama models into one open-source framework, with a comparative table of four LLM families and three VLM families for a search-and-approach drone task. That is a legitimate new application; prior work leaned on ChatGPT or ground robots. The code, configs, and prompts are public, and the sim-to-real attempt is more than many similar papers do. I also give the authors credit for explicitly stating that arithmetic precision of turn/move values and VLM detection correctness were not evaluated, and for noting that 20 episodes per pairing makes the success-rate differences not statistically meaningful. That level of candor is rare.\n\nWhere it softens: the stress-test concern lands. The 100% valid-command metric is a format check, not a correctness check. A model can emit perfectly formatted Turn(90) and Move(3) commands that point away from the target. The paper itself attributes failures to \"suboptimal command values\" and detection false positives/negatives, which confirms that the validity metrics are not the accuracy signal. The only grounded correctness signal is the mission success rate, which peaks at 40% (Gemma pair) and is 0-35% elsewhere. Without a baseline (random actions, heuristic controller, or closed-loop classical planner) and without error bars, the quantitative comparison is suggestive, not conclusive. The real-world demo is one qualitative sequence, fine as a feasibility illustration. The conclusion should be toned down to \"feasibility\" rather than \"accurate execution.\"\n\nMinor issues: the task is narrow (fixed altitude, two primitives, one object class set), and the paper's related-work section is decent but doesn't deeply engage with the VLN benchmarks it cites. Those are minor given the scope.\n\nWho this is for: researchers wanting a local, cloud-free baseline for language-driven drone control, or someone building on PX4/ROS2 and deciding which open models to try. It deserves a serious referee: the integration is real, the negative result for DeepSeek is informative, and the honesty about limitations makes it a good starting point for a stronger benchmark paper. I would send it to review, but I'd ask for a revised conclusions section and at least one sanity baseline before acceptance.","headline":"A genuinely useful open-source integration paper whose headline metrics (100% valid commands) measure syntax only; the real evidence is a 40% mission success rate over 20 episodes, so the accuracy claim outruns the data.","tokens_in":8128,"tokens_out":855,"would_cite":false,"duration_ms":13466,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that a fully local open-source stack can translate natural-language instructions into PX4 flight commands, with the best pairing completing 40 percent of search-and-approach missions.","keywords":["PX4","UAV","LLM","VLM","ROS2","natural language control","embodied agent","Ollama"],"falsifier":"Replay the same 20-episode benchmark while logging, at every step, the true bearing and distance to the goal and comparing them with the issued Turn and Move magnitudes; if a random generator of valid commands matches the 40 percent success rate, or if issued magnitudes are uncorrelated with the ideal values, then the reported validity scores do not establish accurate natural-language command execution.","tokens_in":7145,"feed_emoji":"🚁","tokens_out":9842,"duration_ms":116613,"temperature":0.7,"pith_summary":"This paper tries to establish that a person can fly a PX4 quadcopter by typing plain-English instructions, using only open-source language and vision models hosted locally rather than paid cloud APIs. The authors build an agentic loop in which a large language model receives mission context, a short history of recent actions, and a vision model's answer about what the camera sees, then replies with one of two motion primitives, Turn or Move. Over 20 episodes per model pairing, Gemma3, Qwen2.5, and Llama-3.2 all produced syntactically valid commands in 100 percent of trials, while DeepSeek-LLM managed only 38 percent, and the best mission success rate was 40 percent, achieved by the Gemma3 language-and-vision pairing. The significance, if the claim holds, is that drone dialogue control does not require a closed-source or well-resourced cloud backend, and that open models can be benchmarked and swapped according to mission and hardware constraints.","feed_headline":"Local AI models can command a PX4 drone from plain language","feed_subtitle":"Best mission success is 40% even with 100% valid command syntax and near-perfect object-presence responses.","key_machinery":"The load-bearing mechanism is a closed agentic loop: a ROS2 wrapper exposes locally hosted models through Ollama, a visual-question-answering node converts each camera frame into a Yes/No presence answer, an action-history node supplies the five most recent valid commands, and a path-planning node turns the model's output into PX4 OFFBOARD actions. The loop is deliberately constrained to two deterministic motion primitives, `Turn($\\theta$)` with $\\theta \\in [-90^\\circ, 90^\\circ]$ and `Move($d$)` with $d \\in [-3.0, 3.0]$ m, which makes command generation a grammar-restricted task that any compliant model output can execute. The fixed-history, low-temperature, chain-of-thought-suppressed prompting is what keeps model outputs deterministic enough to be scored on validity.","core_discovery":"The paper's central claim is that a fully local stack—PX4 autopilot, ROS2 middleware, and Ollama-served open models—can close the loop from natural-language dialogue to physical drone flight in both simulation and on a custom quadcopter. On the constrained action space of `Turn($\\theta$)` with $\\theta$ in $[-90^\\circ, 90^\\circ]$ and `Move($d$)` with $d$ in $[-3.0, 3.0]$ m, most tested LLMs emitted grammatically valid commands at near-perfect rates, and all VLMs answered object-presence queries with valid Yes/No responses 97–100 percent of the time. The authors therefore claim that local open-source models are a viable backbone for natural-language UAV control, with the 40 percent mission success ceiling attributed to imperfect angle and distance magnitudes, false object detections, and occasional repeated commands rather than to a failure of language-to-command mapping.","pith_inferences":["Beyond the paper's benchmark, the 40 percent ceiling is a natural target for ablation: swapping the LLM for a script that draws random valid Turn/Move commands would reveal how much of mission success actually comes from linguistic reasoning rather than from the constrained action space and obstacle layout.","A further extension the paper only gestures at is replacing the VLM's Yes/No presence queries with an object detector that returns bounding boxes; that would give the planner spatial grounding and would likely separate false-detection failures from command-magnitude failures.","Because the whole stack is local, the framework implicitly supports privacy-sensitive and connectivity-denied deployments such as indoor inspection in factories or search in disaster zones, an operational benefit that deserves its own evaluation."],"forward_implications":["If the claim holds, natural-language drone control can run entirely on local hardware over a local network, with no per-token cost, no cloud API dependency, and no transmission of camera imagery to a remote service.","A 100 percent valid-command rate does not by itself imply reliable mission execution: even perfect syntax and near-perfect detection leave the best mission success at 40 percent, so the bottleneck is command quality, not command form.","Model selection is decisive: DeepSeek-LLM's 38 percent command validity collapses mission success to 0–5 percent, while the Gemma, Qwen, and Llama pairings cluster in the 30–40 percent range.","The real-world evidence is a single demonstration with the Gemma pairing, meaning the sim-to-real conclusion is that transfer is possible at least once, not that all local model families transfer equally."],"supporting_citations":[{"why":"Supplies the prior PX4-plus-ChatGPT simulation baseline this framework moves away from by using local open models.","marker":"[17]"},{"why":"Shows ChatGPT generating PX4 commander commands in Gazebo, the closest prior art for LLM-to-PX4 command generation.","marker":"[18]"},{"why":"Provides the reasoning-action-observation loop and dynamically callable tools reused as the agentic pattern for ROS-based control.","marker":"[19]"},{"why":"Establishes transformer-based language-conditioned robot control, the lineage from which LLM-generated action commands descend.","marker":"[8]"},{"why":"Represents robotic actions as text tokens in a vision-language-action model, underpinning the idea that an LLM can output executable command strings.","marker":"[9]"},{"why":"Introduces grounding language-model steps in robot affordances, the conceptual basis for constraining generated actions to feasible primitives.","marker":"[10]"}],"fun_headline_variants":["Local open models translate chat to drone commands, but missions fail 60%","Natural-language drone control via open-source stack tops out at 40% success","From chat to quadcopter: local LLMs drive PX4, yet only 40% missions succeed","Open-source AI pilots PX4 drone via dialogue, but mission success remains 40%","Pure local stack controls PX4 drone with natural language; mission rate: 40%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a syntactically valid Turn or Move command is a meaningful sign of correct natural-language execution; yet the paper explicitly did not check the arithmetic of angle and distance values or the correctness of the VLM's object detections.","fun_headline_variants_meta":{"raw":{"variants":["Local open models translate chat to drone commands, but missions fail 60%","Natural-language drone control via open-source stack tops out at 40% success","From chat to quadcopter: local LLMs drive PX4, yet only 40% missions succeed","Open-source AI pilots PX4 drone via dialogue, but mission success remains 40%","Pure local stack controls PX4 drone with natural language; mission rate: 40%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000754,"raw_usage":{"total_tokens":3320,"prompt_tokens":879,"completion_tokens":2441,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":2327}},"tokens_in":495,"tokens_out":2441,"duration_ms":16862,"temperature":1.0,"reasoning_tokens":2327,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:31:48.930232+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replay the same 20-episode benchmark while logging, at every step, the true bearing and distance to the goal and comparing them with the issued Turn and Move magnitudes; if a random generator of valid commands matches the 40 percent success rate, or if issued magnitudes are uncorrelated with the ideal values, then the reported validity scores do not establish accurate natural-language command execution.","supporting_citations":[{"cited_title":"Im- plementation of natural language uav control using openai’s chatgpt in a simulated university environment,","cited_arxiv_id":null,"evidence_quote":"Supplies the prior PX4-plus-ChatGPT simulation baseline this framework moves away from by using local open models."},{"cited_title":"From words to flight: Integrat- ing openai chatgpt with px4/gazebo for natural language-based drone control,","cited_arxiv_id":null,"evidence_quote":"Shows ChatGPT generating PX4 commander commands in Gazebo, the closest prior art for LLM-to-PX4 command generation."}],"review_version":1}