{"id":"839ee9f4-cced-4f67-b8d3-e5c4f5ca20b2","arxiv_id":"2505.20783","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FM-Planner shows Llama-3.1-8B can match A* on simple drone obstacle courses and, after LoRA fine-tuning plus YOLO perception, completes three real indoor flights, though with no statistical baselines.","lead":"The paper benchmarks eight language models and five vision-language models for drone path planning, then builds a Llama-3.1 planner with a YOLOv8 vision encoder and tests it on a real indoor drone. It reports that the text-driven LLM matches A* on simple static obstacle courses, but the evaluation is too small and uncontrolled to support the broader claims of robust, real-time navigation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Perception-to-LLM obstacle localization is unvalidated; without error bounds against the 1-1.2 m clearance margins, the physical 'robust obstacle awareness' claim is not established.","rationale":"The paper is a legitimate engineering study: it benchmarks eight LLMs and five VLMs, makes code available, and reports real flights with a LoRA-fine-tuned Llama-3.1-8B and YOLOv8. The simulation claim that Llama matches A* in the two tested scenarios is plausible, though it lacks repetitions and error bars. The weight-bearing part of the conclusion is the physical LLM-Vision system: the only link between the environment and the LLM's P_o input is YOLOv8+RealSense. If that link is biased, noisy, or misses obstacles, the 'real-time obstacle awareness' conclusion collapses even if the LLM reasons correctly about the coordinates it is given. The paper provides no measurement of this link, and the three successful flights with only ground-truth obstacle positions in Table V cannot measure it. This is not a disagreement with field consensus; it is an internal validity gap in the deployment claim. A calibration-and-repetition experiment would directly settle whether the concern lands. Because the reader already identified this same weakest assumption and issued a CONDITIONAL verdict, I do not see a reason to move the verdict; the same condition should stand.","tokens_in":12917,"tokens_out":6436,"duration_ms":73673,"concrete_test":"Run a calibration pass in the same 8x10x5 m flight volume: place obstacles at OptiTrack-known positions, capture D435i frames from the actual start poses of Tasks 1-3, run the YOLOv8+depth pipeline, and compare the estimated per-axis P_o values against OptiTrack ground truth across a grid of ranges and orientations. If the maximum Euclidean localization error is safely below the 1.2 m planning margin (e.g., under 0.3 m in all cases), the concern is resolved. If not, feed the estimated P_o values into the LLM planner and re-run each physical task at least 10 times; any collision or margin-exceeding error would refute the robustness claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the LLM-Vision planner has 'real-time obstacle awareness' depends on the obstacle positions P_o fed to the LLM being accurate. Section III-C says YOLOv8 plus RealSense depth data is used to 'estimate the 3D position', and Section IV-G says obstacle information was extracted from RGB and depth images during flight. Yet the paper provides no detection accuracy, depth-error analysis, calibration check, or failure cases for this perception pipeline. The safety margins in the planner—1 m in Section IV-F and 1.2 m in Section III-A—are the only buffer between a mislocalized obstacle and a collision. A missed obstacle or a position bias larger than the margin would make the LLM plan through the obstacle while the system still reports a successful run. The three physical tasks in Table V are single-shot, and Table V lists the ground-truth obstacle coordinates, not the estimated coordinates actually used by the planner, so these successes cannot bound the perception error. Without an error model for YOLOv8+D435i, the physical experiments support only the narrow claim that the drone succeeded in three particular configurations, not the conclusion that fine-tuned LLMs have robust, real-time obstacle awareness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes FM-Planner, a foundation-model-based global path planning framework for drones. The authors benchmark eight LLMs and five VLMs in a Gazebo simulation with static cubic obstacles, using A*, RRT, and Q-learning as baselines and reporting success rate (SR), average completion time (ACT), path length (PL), and an efficiency-success score (ESS). Based on the benchmark, they fine-tune Llama-3.1-8B-Instruct with LoRA on 5,000 synthetic waypoint sequences generated by RRT and integrate it with a YOLOv8 vision encoder to estimate obstacle positions from RGB-D images. They validate the resulting LLM-Vision planner in three indoor flight tasks with a QAV250 drone and report that all three tasks succeeded with an average reasoning time of about 9.5 s per query. The headline claims are that Llama-3.1-8B-Instruct matches the A* planner on reliability and efficiency in simulation, and that the fine-tuned LLM-Vision system demonstrates robust spatial reasoning and real-time obstacle awareness.","tokens_in":13133,"tokens_out":6209,"duration_ms":58889,"significance":"The paper addresses a timely question—whether foundation models can perform global path planning for drones—and the base-LLM benchmark is a useful contribution: it compares eight models against classical planners in a standardized setting, and the independent comparison with external baselines makes the headline simulation result non-circular. The open project site and detailed hyperparameter table also support reproducibility. However, the significance of the physical-validation claim is currently limited. The simulation results are reported as point estimates without variance or trial counts, the VLM evaluation is qualitative, and the real-world experiments provide no error analysis for the perception pipeline or comparison with baselines. As presented, the evidence does not support the conclusion that the system has robust, real-time obstacle awareness.","major_comments":[{"comment":"The central safety claim rests on an unvalidated perception pipeline. Section III-C states that YOLOv8 plus depth information is used to 'estimate the 3D position' of obstacles, and Section IV-G reports that obstacle information was extracted from RGB and depth images during flight. However, the paper provides no detection accuracy, depth-error analysis, calibration check, or failure cases for this pipeline. Table V reports ground-truth obstacle coordinates, not the estimated coordinates actually fed to the LLM. With the only buffers being the 1.2 m margin in Section III-A and the 1 m clearance in Section IV-F, a misdetection or depth bias larger than these margins would produce unsafe waypoints while the run is still counted as a success. The three single-shot physical tasks therefore support only the narrow claim that the drone succeeded in three particular configurations, not the paper's conclusion of robust obstacle awareness.","section":"III-C, IV-G, Table V"},{"comment":"The VLM benchmark is not quantified. Section IV-E2 describes the VLM-generated trajectories only qualitatively (e.g., 'GPT-4o and Claude produced relatively coherent and feasible paths'), and Fig. 5 shows paths without any numerical metrics. Since the abstract and introduction claim a 'comprehensive benchmarking study' of LLM and VLM approaches, the VLM portion needs at least the same SR/PL/ESS metrics and trial counts as the LLM portion to support the comparison and the claim that VLMs lack spatial reasoning.","section":"IV-E"},{"comment":"The simulation benchmark reports single values without variance, number of trials, or random seeds. The text mentions '10 trials' only in Section IV-D1, and Table II shows one 0/10 entry, but the SR differences that drive the paper's conclusions (e.g., 100% vs. 90% for Llama vs. Qwen-7B in S1) are not statistically distinguishable at n=10. Please report per-trial data, confidence intervals, and seeds, or increase the number of trials, for all reported SR/ACT/PL/ESS values.","section":"IV-D, Tables I-III"},{"comment":"The 'real-time obstacle awareness' claim is not supported by the reported 9.5 s reasoning time and the described architecture. Section IV-G states that the LLM is executed at each planning step, but it does not specify how often the YOLOv8 perception updates the obstacle list or whether the global path is recomputed during flight. If the waypoint sequence is generated once before takeoff, then the system is not providing real-time obstacle awareness in any feedback sense; it is a one-shot global planner with offline perception. The conclusion should be qualified accordingly, or the paper should report the update rate and show that the planner reacts to perception changes during flight.","section":"IV-F, IV-G, V"},{"comment":"The fine-tuning evaluation is incomplete. The LLM-Vision model is trained on 5,000 synthetic scenarios whose waypoints were computed by RRT (Section IV-F), yet the physical experiments in Section IV-G do not compare against RRT, A*, or the unfine-tuned LLM, and no analysis is given for generalization to obstacle layouts outside the training distribution. In addition, the statement that 'RRT planner provides optimal shortest paths' is incorrect: RRT is not an optimal planner. The training-data claim should be corrected, and the physical validation needs a baseline comparison and an out-of-distribution test to support the conclusion that the fine-tuned model has robust spatial reasoning.","section":"IV-F"}],"minor_comments":[{"comment":"The phrase 'DeepSeek-R1 fails in all 10 trials, underscoring the limitations of purely reactive methods' mischaracterizes DeepSeek-R1, which is a large reasoning model, not a purely reactive method.","section":"IV-D1"},{"comment":"The ESS definition ESS=SR/ACT with units %/s is unusual; please state explicitly how it normalizes for path length and why it is preferred over standard metrics.","section":"IV-B"},{"comment":"The statement that a safety margin of 1.2 m is 'maintained around the obstacle along the edges' is ambiguous; specify whether the margin is measured from the obstacle surface or center.","section":"III-A"},{"comment":"Table IV lists Epochs=60 but the text says 'trained over 100 epochs' and Fig. 6 shows convergence after 60; please reconcile this inconsistency.","section":"IV-F, Table IV"},{"comment":"The abstract says 'eight representative LLM and VLM approaches' but the paper actually evaluates eight LLMs and five VLMs; please rephrase to avoid undercounting the models.","section":"Abstract"},{"comment":"The statement that the model was 'requested from the Hugging Face server' suggests the LLM is not running fully onboard the Jetson; clarify the deployment split between onboard and remote inference and its effect on the real-time claim.","section":"IV-G"},{"comment":"The VLM simulation uses 1 m obstacles while the LLM simulation uses 2 m obstacles; please justify or align these settings so the two benchmarks are comparable.","section":"III-B, IV-A"}],"recommendation":"major_revision","confidential_remarks":"The base-LLM benchmark is the strongest part of the paper and is worth publishing if the statistical reporting is fixed. The biggest risk is the physical-validation section: as it stands, the conclusion overreaches the evidence. I would encourage the editor to require the perception error analysis and a baseline comparison before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The paper's real value is the empirical sweep: eight LLMs and five VLMs scored on the same drone path-planning tasks, with A*, RRT and Q-learning as external baselines. That is genuinely useful model-selection data, and the base-LLM comparison is not circular. The integrated LoRA-tuned Llama-3.1-8B plus YOLOv8 system also flies in three real configurations, which is a real feasibility demonstration.\n\nWhere it gets soft. The simulation tables give single numbers with no variance or trial counts beyond an implied 10; the VLM results are visual and qualitative; the physical experiments are three single-shot runs with no baseline. More important, the 'real-time obstacle awareness' conclusion depends on YOLOv8 plus RealSense depth giving obstacle positions accurate enough for the 1–1.2 m clearance margins, and the paper gives no detection accuracy, depth-error analysis, or calibration check. Table V reports ground-truth obstacle coordinates, not the estimated coordinates the planner actually used. So the three successful flights support the narrow claim that the drone made it in those three setups; they don't establish robust perception. The stress-test note lands.\n\nAlso worth flagging: the text calls DeepSeek-R1 'purely reactive'—it isn't—and says 'the RRT planner provides optimal shortest paths'—that's RRT*, not classic RRT. And the fine-tuned model was trained on RRT-generated waypoints, then reported as successful in flight, but there is no simulation comparison of the fine-tuned model against the untuned LLM or against A* on the same scenarios, so the fine-tuning gain is not quantified.\n\nNet: this is a serious engineering paper with a new benchmark and a working integrated system. It deserves a referee, but the revision needs repeated trials with variance, a perception error analysis, a physical baseline, and a narrower conclusion. The benchmark alone is worth citing—this is a paper for groups picking an LLM for waypoint generation, and for anyone building a vision-plus-LLM drone loop.","headline":"A useful empirical benchmark and a real flight demo, but the paper's central 'robust obstacle awareness' claim outruns the evidence—especially the unvalidated YOLO+depth perception chain.","tokens_in":13702,"tokens_out":3925,"would_cite":true,"duration_ms":37160,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned Llama matches A* on drone path planning","keywords":["foundation model","path planning","autonomous drone","large language model","vision-language model","LoRA fine-tuning","waypoint generation","obstacle avoidance"],"falsifier":"Measure the actual 3D position error of the YOLOv8 detection pipeline against a ground-truth motion-capture of obstacle locations in the same arena; if the error is comparable to or larger than the 1-1.2 m safety margin, then the planner's collision-free guarantee fails. A simpler version: perturb the obstacle coordinates fed to the LLM by 0.5-1.0 m in simulation and record whether success rate collapses.","tokens_in":12697,"feed_emoji":"🚁","tokens_out":4455,"duration_ms":40790,"temperature":0.7,"pith_summary":"The paper sets out to show that foundation models can do global path planning for drones, not just local reactive control. It benchmarks eight LLMs and five VLMs on simulated obstacle-avoidance tasks and finds that pure LLMs operating on textual coordinates outperform VLMs that must reason from map images. Its central result is an LLM-Vision planner: a LoRA-fine-tuned Llama-3.1-8B-Instruct fed by a YOLOv8 vision encoder, which reaches 100 percent success in the two- and three-obstacle simulation scenarios and completes all three real-world flights with about 9.5 seconds of average reasoning time. If the result holds, it means a modest, fine-tuned language model paired with an off-the-shelf object detector can replace classical planners such as A* in known static indoor environments.","feed_headline":"Fine-tuned Llama matches A* on drone path planning","feed_subtitle":"A LoRA-tuned 8B LLM plus YOLOv8 vision guides a real drone around obstacles, with 100% simulation success.","key_machinery":"The load-bearing object is the tokenized prompt $T_{\\mathrm{input}} = \\langle \\text{Prompt}, P_s, P_g, \\{P_o^n\\}_{n=1}^N\\rangle$, which serializes start, goal, and obstacle coordinates into natural language for the LLM. A YOLOv8 vision encoder turns the drone's RGB-D stream into class-labeled 2D bounding boxes and estimated 3D positions, which are tokenized into that prompt. The LLM is fine-tuned with Low-Rank Adaptation (LoRA), keeping the base weights frozen and learning only low-rank updates $\\Delta W = BA$, which adds $O(dr + rk)$ parameters instead of $O(dk)$. A Euclidean-interpolation module inserts waypoints every 0.5 m and keeps a 1.2 m safety margin around obstacles; in the physical system the margin is 1-1.2 m. This pipeline converts sparse language-generated waypoints into smooth, flight-ready position setpoints.","core_discovery":"The paper claims that a fine-tuned LLM integrated with a vision encoder demonstrates robust spatial reasoning and real-time obstacle awareness, making it well-suited for practical drone global planning tasks. In the simulation benchmarks, the fine-tuned Llama-3.1-8B-Instruct matches the optimal A* planner on both reliability and efficiency, recording a 100 percent success rate and efficiency-success scores of 1.250 and 1.111 on the two- and three-obstacle scenarios; it is the only neural planner to do so. In physical experiments, the LLM-Vision planner guided a QAV250 racing drone through all three test tasks with different start positions and obstacle configurations, with a mean reasoning time near 9.5 seconds. The authors interpret this as evidence that textual spatial inputs plus separately perceived obstacle locations generalize better than pure VLM visual reasoning.","pith_inferences":["The paper's benchmark is restricted to two static obstacle configurations; a natural stress test is whether the fine-tuned model generalizes to arbitrary obstacle counts and positions without re-tuning.","The absence of quantified perception error leaves open whether the same pipeline would be safe in tighter spaces where the 1.2 m margin is unavailable.","Because the synthetic fine-tuning data were generated with RRT, the policy may inherit sampling artifacts rather than learning true geometric reasoning; this would show up in corridors narrower than the training distribution."],"forward_implications":["A small LoRA-fine-tuned LLM can serve as a global path planner on edge hardware, generating collision-free waypoints without explicit dynamics models.","Classical planners like A* remain the optimality reference, but the LLM route adds the ability to absorb natural-language mission constraints into the same planning step.","The planner's reliability degrades sharply at control rates above 50 Hz, so the practical speed envelope is bounded by re-planning latency.","VLMs alone produced non-optimal or colliding paths in simulation, indicating that visual grounding must be supplied separately to the LLM rather than inside a single VLM."],"supporting_citations":[{"why":"Supplies the A* optimal grid-search baseline that the Llama model matches in reliability and efficiency.","marker":"[5]"},{"why":"Supplies the RRT sampling-based baseline and is also the planner used to compute waypoints in the synthetic fine-tuning dataset.","marker":"[6]"},{"why":"LEVIOSA is the prior LLM-to-waypoint trajectory work that FM-Planner extends with benchmarking and real-time perception.","marker":"[28]"},{"why":"Supplies the Q-learning baseline planner against which the neural planners are compared.","marker":"[33]"}],"fun_headline_variants":["Fine-tuned Llama ties A* in drone path planning","LLM-Vision drone planner: 100% success, matches A*","8B LLM plus vision guides real drone, 100% sim success","Drone navigation with fine-tuned LLM matches optimal A*","Real drone tests confirm LLM planner matches A* in sim"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole safety argument rests on the unvalidated premise that the YOLOv8-plus-depth obstacle positions are accurate enough that the 1-1.2 m clearance margins truly keep the drone collision-free.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned Llama ties A* in drone path planning","LLM-Vision drone planner: 100% success, matches A*","8B LLM plus vision guides real drone, 100% sim success","Drone navigation with fine-tuned LLM matches optimal A*","Real drone tests confirm LLM planner matches A* in sim"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1606,"prompt_tokens":904,"completion_tokens":702,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":609}},"tokens_in":520,"tokens_out":702,"duration_ms":6270,"temperature":1.0,"reasoning_tokens":609,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:47:35.638412+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the actual 3D position error of the YOLOv8 detection pipeline against a ground-truth motion-capture of obstacle locations in the same arena; if the error is comparable to or larger than the 1-1.2 m safety margin, then the planner's collision-free guarantee fails. A simpler version: perturb the obstacle coordinates fed to the LLM by 0.5-1.0 m in simulation and record whether success rate collapses.","supporting_citations":[{"cited_title":"Uav path planning based on the combination of a-star algorithm and rrt-star algorithm,","cited_arxiv_id":null,"evidence_quote":"Supplies the A* optimal grid-search baseline that the Llama model matches in reliability and efficiency."},{"cited_title":"An efficient rrt-based framework for planning short and smooth wheeled robot motion under kinodynamic constraints,","cited_arxiv_id":null,"evidence_quote":"Supplies the RRT sampling-based baseline and is also the planner used to compute waypoints in the synthetic fine-tuning dataset."},{"cited_title":"Leviosa: Natural language-based uncrewed aerial vehicle trajectory generation,","cited_arxiv_id":null,"evidence_quote":"LEVIOSA is the prior LLM-to-waypoint trajectory work that FM-Planner extends with benchmarking and real-time perception."},{"cited_title":"An adaptive conversion speed q-learning algorithm for search and rescue uav path planning in unknown environments,","cited_arxiv_id":null,"evidence_quote":"Supplies the Q-learning baseline planner against which the neural planners are compared."}],"review_version":1}