{"id":"7381ead6-2153-4bd1-9444-9a244f447981","arxiv_id":"2505.07236","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"UAV-CodeAgents combines multi-agent ReAct reasoning with a fine-tuned vision-language model for pixel-level grounding to generate UAV missions from satellite imagery and text prompts.","lead":"UAV-CodeAgents is a multi-agent system that uses large language and vision models to turn satellite images and natural language instructions into drone flight plans. It reports a 93% success rate on 30 fire-detection mission images, but the evaluation uses the authors' own benchmark and includes no comparison to prior systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 93% success rate is unverifiable as reported: no success criterion is defined, TTD excludes false positives/negatives, and simulated fire ground truth comes from the same VLM under evaluation.","rationale":"The reader correctly identified the evaluation as a weak point, focusing on the small self-referential benchmark and 2D simulation. My pass lands on a more internal problem: the success metric itself is undefined and the only quantitative metric, TTD, explicitly excludes false positives and false negatives. This makes the headline 93% success rate and the 96.96 s average completion time impossible to interpret, independent of how representative the 30 images are. The lack of a success criterion is load-bearing because the paper's central claim is precisely that the system succeeds at 93% of missions. The proposed check — rerunning with ground-truth labels and a fixed success rule on all 30 samples — would settle whether the claim is real or an artifact of self-evaluation. I therefore recommend UNVERDICTED rather than REJECT, because the architecture may be sound; the evidence as presented is simply insufficient to verify the central empirical claim.","tokens_in":7532,"tokens_out":4062,"duration_ms":40860,"concrete_test":"Ask the authors to release the 30 benchmark images with independent ground-truth fire annotations (point/polygon locations) and a stated per-sample success rubric, then rerun both temperature settings on all 30 images. Count a sample as successful only if the generated waypoint sequence reaches every ground-truth fire location within a pre-specified pixel radius and the final report contains no false positives; recompute TTD on the full 30-sample set rather than after excluding failures. If the recalculated success rate is materially below 93% or the T=0.5 vs T=0.7 difference vanishes, the temperature claim and the headline success rate should be revised.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"Section V.B defines the only quantitative metric, Time-to-Detection, with N = 30 test cases 'excluding those with false positives or false negatives.' This is internally problematic: the headline 93% success rate (28/30 at T=0.5) is never tied to an explicit per-sample success criterion, yet the time metric is computed after discarding the very samples that would indicate failure. The paper reports no TTD value and no N used after exclusions, so the average 96.96 s may cover only a self-selected subset. More importantly, the 2D simulator linearly interpolates between waypoints over a static satellite image; the only 'fire confirmed' signals come from VLM inference on cropped frames produced by that same pipeline. There is no independent ground-truth label saying where fires are, no threshold for a successful detection, and no baseline (e.g., random waypoints or simple image thresholding) to calibrate the metric. Consequently, '28 successful samples' cannot be interpreted as 93% planning success; it may mean only that the agent chain terminated and returned a final answer.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes UAV-CodeAgents, a multi-agent framework built on LLMs/VLMs and the ReAct loop, intended to convert natural-language mission queries plus satellite imagery into executable UAV waypoint plans. The system combines an Airspace Manager Agent that interprets queries and grounds objects at the pixel level, a UAV Agent that simulates flights and performs VLM-based fire detection, and a reactive thinking loop for plan revision. The authors evaluate the system on 30 images from an augmented version of their own UAV-VLA benchmark using a single vague query, report a 93% success rate and 96.96 s average mission creation time at temperature 0.5, compare to temperature 0.7, and separately fine-tune Qwen2.5VL-7B on 9,000 annotated satellite images, reporting per-category mean distance and coverage. The paper claims that lower temperature yields higher planning reliability and reduced execution time, and that the fine-tuned model achieves strong spatial grounding.","tokens_in":7773,"tokens_out":3443,"duration_ms":36868,"significance":"If the reported results held, the framework would be a useful demonstration of VLM-driven UAV mission planning from high-level language and satellite imagery, with potential value for disaster-response scenarios. The paper also promises a release of the codebase and a new benchmark dataset, which are constructive reproducibility contributions, and the fine-tuning dataset for pixel-pointing is a plausible resource. However, the significance is currently not established empirically: the headline success rate is uninterpretable without a success rubric, the only quantitative metric is defined but never reported, no baselines are provided, the simulator is a linear-interpolation abstraction, and the grounding evaluation lacks a train/test split or pre-fine-tuning comparison. The architecture itself is a reasonable integration of existing components, but the evidence as presented does not yet demonstrate that the framework outperforms simpler alternatives or generalizes beyond the 30 selected images.","major_comments":[{"comment":"The 93% success rate is not tied to any explicitly defined per-sample success criterion. Equation (1) defines Time-to-Detection but says N = 30 test cases 'excluding those with false positives or false negatives,' yet the paper never states what constitutes a success, a false positive, or a false negative on a sample. Table II reports 'Successful samples' (28 and 26) with no rubric, so a reader cannot verify whether success means merely that the agent chain terminated or that the UAV visited the true fire location. Please define the success criterion, apply it to all 30 samples, and report the number of samples excluded after each exclusion rule.","section":"Section V.B, Eq. (1), and Table II"},{"comment":"The evaluation is conducted only on 30 images from the authors' own augmented UAV-VLA benchmark, and no baseline is compared. Without a baseline such as the original UAV-VLA system, a random-waypoint policy, a simple image-thresholding detector, or a direct single-agent prompt on the same benchmark, the observed 28/30 success cannot be attributed to the ReAct loop, the multi-agent design, or the pixel-pointing mechanism. The temperature comparison is also only between two settings of the same system. Please add at least one external baseline and report per-scene results to enable calibration of the 93% figure.","section":"Section IV.A and V.C"},{"comment":"Time-to-Detection (TTD) is defined in Eq. (1) but never reported anywhere in the paper. The reported 'Avg. Elapsed Time' is a different quantity, and it is not accompanied by the standard deviation, the number of samples remaining after exclusions, or the operational exclusion criteria. The text also does not state whether the 96.96 s average includes all samples or only the 28 'successful' ones. Please report TTD explicitly for each temperature, with the sample count after exclusions, and explain how failures are handled (e.g., censored at max time) rather than silently discarded.","section":"Section V.B and V.C"},{"comment":"The fine-tuning results are reported only as per-category mean distance and mean coverage, without an evaluation split, a comparison to the zero-shot Qwen2.5VL-7B before fine-tuning, or any statement of the units or normalization of pixel distances. Table III also contains duplicated category names such as 'ponds' versus 'pond', 'roads' versus 'road', and 'farm' versus 'farms', which makes the table ambiguous. Without a held-out test set and a pre-fine-tuning baseline, the claim of 'strong spatial grounding' is not supported. Please add a train/test split, report the pre-fine-tuning numbers, and clarify the category naming.","section":"Section V.D and Table III"},{"comment":"The simulation linearly interpolates waypoints over a static satellite image and generates crops at interpolated positions; there is no sensor model, altitude, heading, or temporal dynamics. In addition, the 'fire confirmed' signal comes from VLM inference on the same cropped frames produced by the same pipeline, so the only ground truth in the loop is the same model under evaluation. This creates a circularity that makes the 93% success rate hard to interpret as a real detection result. Please use independent ground-truth annotations of fire locations and a more realistic simulator, or clearly state the result as a planning-completion rate rather than a fire-detection success rate.","section":"Section IV.B and V.B"}],"minor_comments":[{"comment":"The bullet 'We design a UAV-oriented reactive thinking loop...' appears twice verbatim in the contribution list; please remove the duplicate.","section":"Section I (Contributions)"},{"comment":"The text states that pixel-pointing is 'not a novel method,' yet the abstract and contributions present it as a core component. Please clarify what is new here, for instance whether the novelty lies in the fine-tuning dataset or in the particular integration with ReAct.","section":"Section III.D"},{"comment":"With N = 30, the difference between 28 and 26 successful samples is small, and no confidence interval or statistical test is reported. A p-value or confidence interval would help assess whether the temperature effect is meaningful.","section":"Section V.C and Table II"},{"comment":"The 'Avg. Elapsed Time' is not precisely defined. Please state whether it includes API inference time, agent communication overhead, simulation time, or all of these, since the abstract refers to 'mission creation time.'","section":"Section V.A"},{"comment":"The paper promises release of the codebase and benchmark but does not state the license, access conditions, or expected release timeline. Including these details would strengthen the reproducibility claim.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript is a system paper whose central empirical claims rest on a very small, self-selected evaluation with no external baseline and no defined success rubric. The issues are fixable in principle, but they are load-bearing: the 93% success rate and the temperature comparison are the main quantitative results. I would not recommend rejection outright, since the framework and the promised benchmark could be valuable, but the revision must add a clear success definition, report TTD and sample counts honestly, include at least one baseline, and add proper splits and comparisons for the fine-tuning experiment. I also note that most of the closely related references are from the same research group; this is not improper, but it makes independent baseline comparison even more necessary."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a system paper that stacks known components—ReAct loop, multi-agent orchestration on smolagents, a fine-tuned Qwen2.5VL for pixel-pointing—into a UAV mission planning pipeline. Credit where due: it is honest that pixel-pointing is not novel, describes the architecture clearly, and the idea of iteratively refining a plan from a vague query like “I’ve heard there are fires in our area” is genuinely useful for language-driven UAV work. The fine-tuning experiment with 9,000 annotated satellite images is a real effort, and the promise to release code and a benchmark is good practice.\n\nBut the evaluation does not support the abstract’s claims. The central number—93% success rate—rests on 30 images from the authors’ own UAV-VLA benchmark, with no baseline, no success rubric, and no external ground truth. The Time-to-Detection metric literally excludes false positives and false negatives, so it measures latency on a self-selected easy subset. The 2D simulator linearly interpolates between waypoints over a static satellite image, and the only “fire confirmed” signal comes from the same VLM under test. That makes the reported 28/30 hard to interpret as planning success; it may only mean the agent chain terminated and returned an answer. The SFT results also lack evaluation splits and baseline numbers, so the claimed spatial grounding improvement is unverified. These are not minor caveats—they are load-bearing for the empirical contribution.\n\nThat said, the architecture itself is sensible and the paper is on its own terms coherent. The self-referential evaluation is a weakness, not evidence of bad faith; the authors seem to know the limits of their setup, they just didn’t do the hard work of comparing against anything or defining success cleanly. A competent referee could push them to add a random-waypoint baseline, fix the success criterion, report TTD after exclusions, and ideally test on an independent benchmark.\n\nWho should read this: researchers working on VLM-based UAV planning or language-to-waypoint systems. They’ll get the architectural framing and the fine-tuning recipe, but they should treat the headline numbers as pilot results, not verified performance.\n\nRecommendation: send it to peer review. The system is plausible, the problem is relevant, and the paper deserves a chance to be pushed into a properly evaluated form. As-is, it would need major revision before publication.","headline":"Plausible multi-agent VLM system for UAV mission planning, but the headline 93% success rate is not supported by the evaluation as reported.","tokens_in":8256,"tokens_out":877,"would_cite":false,"duration_ms":10194,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-agent system with a reactive reason-act loop and pixel-level vision grounding converts vague natural-language reports into UAV waypoint plans, succeeding on 93% of tested fire scenes.","keywords":["multi-agent systems","UAV mission planning","vision-language models","ReAct reasoning","pixel-level grounding","satellite imagery","fire detection","autonomous navigation"],"falsifier":"Take a held-out set of satellite images of real fire incidents that differ from the 9,000 training images, run the same vague query through the system, and compare each generated waypoint plan against the known fire locations; if the system does not put waypoints near the fires on a comparable fraction of scenes—or if smoke and cloud cover in fresh imagery break the pixel grounding—the central claim fails.","tokens_in":7383,"feed_emoji":"🚁","tokens_out":7804,"duration_ms":73201,"temperature":0.7,"pith_summary":"This paper tries to establish that a team of cooperating software agents—one that interprets language and images, one that simulates the drone—can convert a vague, incomplete human report and a satellite image into a concrete drone mission without manual waypoint entry. The authors' key claim is that two mechanisms make this possible: a reactive reason-and-act loop, in which the agents observe, describe, reason, decide, and act again after seeing the drone's simulated perspective, and a pixel-pointing step that grounds phrases like \"building\" or \"pond\" to coordinates on the map. They report that on 30 fire-detection scenes, the system succeeds on 28 (93%) with an average planning time of about 97 seconds at a conservative decoding temperature. If the result holds up, it matters because it would let non-experts task aerial robots for time-critical jobs like fire response by simply describing what they know.","feed_headline":"93% of benchmark fire scenes become flyable drone plans","feed_subtitle":"A multi-agent reason-act loop and pixel-level map grounding produce a mission in about 97 seconds.","key_machinery":"The central mechanism is the coupling of a ReAct (Reason + Act) loop with a vision-grounded pixel-pointing step. In the loop, the manager agent observes the scene, asks a vision-language model for a detailed description, reasons about how the scene relates to the mission query, decides on waypoints or revisions, and acts by dispatching instructions to the UAV agent; after the simulated flight, frames flow back for another round. Pixel-pointing is what connects a semantic label (such as \"the warehouse near the forest\") to actual coordinates on a satellite image, using a 7B vision-language model fine-tuned on 9,000 annotated satellite images. The same mechanism lets the planner rank candidate targets by estimated fire probability and route the UAV to the highest-probability locations first.","core_discovery":"On its own terms, the paper claims that a modular multi-agent arrangement—an airspace manager that reasons over language and images, plus an embodied UAV agent that simulates flight and reports observations—can turn a vague, ambiguous instruction into a flyable mission, and that the two ingredients doing the heavy lifting are the reactive thinking loop and pixel-level grounding. The reported evidence is that, on 30 fire-scene images drawn from an augmented version of the UAV-VLA benchmark, the system plans 28 missions successfully (93%) with a mean generation time of 96.96 seconds at temperature 0.5, compared with 26 successes and 105.29 seconds at temperature 0.7. The paper also claims that fine-tuning a 7B vision-language model on 9,000 annotated satellite images gives it reliable spatial grounding across categories such as buildings, roads, ponds, parking lots, and farmland, with mean pixel distances and coverage values reported per category.","pith_inferences":["Beyond the paper's fire-detection setting, the same pixel-pointing-plus-ReAct pipeline could be applied to flood mapping, crop monitoring, or post-disaster damage assessment, provided the grounding model is fine-tuned on those visual categories.","Because the 93% figure comes from a 2D simulator that linearly interpolates between waypoints, real flights are likely to be harder; the most direct test is to fly the generated plans on a physical drone and measure whether the target is actually reached.","An ablation that removes the reactive loop while keeping the same agents would isolate how much of the success comes from iterative re-planning rather than from the underlying vision-language grounding.","The fine-tuned grounding model could be packaged as a standalone geospatial phrase-to-coordinate service, independent of the multi-agent mission planner."],"forward_implications":["A natural-language query like \"I've heard there are fires in our area\" can seed a complete UAV waypoint mission, so operators do not need to place waypoints by hand.","The reactive loop should let the same architecture recover from ambiguous or incomplete initial instructions by re-querying the visual scene and revising waypoints mid-mission.","The pixel-pointing grounding makes semantic map categories (buildings, roads, ponds, farmland) directly addressable as flight targets, not just as caption text.","Lower decoding temperature becomes a usable reliability knob for mission-critical planning: the authors find 0.5 outperforms 0.7 on both success rate and latency.","Because agents are modular and communicate by messages, the system can scale to more UAVs and can mix heavy and light models across ground and onboard agents."],"supporting_citations":[{"why":"Supplies the UAV-VLA benchmark whose augmented 30-image set is used for all success-rate measurements.","marker":"[14]"},{"why":"Prior cognitive UAV VLA system with a reasoning module, providing the comparison baseline for reasoning-based planning.","marker":"[13]"},{"why":"Earlier vision-language-path-action system for large-scale route generation that this work extends with multi-agent ReAct.","marker":"[15]"},{"why":"Shows how language instructions can be grounded in aerial navigation, basis for the language-to-waypoint idea.","marker":"[7]"},{"why":"Provides a realistic UAV vision-language navigation benchmark and simulator, informing the simulated evaluation design.","marker":"[9]"},{"why":"Applies domain reasoning with a modular agent to wildfire analysis, motivating the fire-domain agent design.","marker":"[22]"},{"why":"Foundational contrastive vision-language model that underlies the grounding capability used for pixel pointing.","marker":"[5]"}],"fun_headline_variants":["Multi-agent reason-act loop turns vague orders into UAV missions","Pixel-level map grounding gives UAVs 93% flyable plan success","Reactive thinking loop plans drone missions in 97 seconds","UAV-CodeAgents: 93% success with vision-language planning","Satellite image reasoning yields flyable drone plans 93% of time"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that the 30 benchmark images, with flights simulated by straight-line interpolation over satellite imagery, represent the real fire-detection situations where the system would be used.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent reason-act loop turns vague orders into UAV missions","Pixel-level map grounding gives UAVs 93% flyable plan success","Reactive thinking loop plans drone missions in 97 seconds","UAV-CodeAgents: 93% success with vision-language planning","Satellite image reasoning yields flyable drone plans 93% of time"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000227,"raw_usage":{"total_tokens":1479,"prompt_tokens":959,"completion_tokens":520,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":429}},"tokens_in":575,"tokens_out":520,"duration_ms":4751,"temperature":1.0,"reasoning_tokens":429,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:20:57.712883+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of satellite images of real fire incidents that differ from the 9,000 training images, run the same vague query through the system, and compare each generated waypoint plan against the known fire locations; if the system does not put waypoints near the fires on a comparable fraction of scenes—or if smoke and cloud cover in fresh imagery break the pixel grounding—the central claim fails.","supporting_citations":[{"cited_title":"UA V-VLA: Vision-language-action system for large scale aerial mission generation,","cited_arxiv_id":null,"evidence_quote":"Supplies the UAV-VLA benchmark whose augmented 30-image set is used for all success-rate measurements."},{"cited_title":"AerialVLN: Vision-and-language navigation for UA Vs,","cited_arxiv_id":null,"evidence_quote":"Shows how language instructions can be grounded in aerial navigation, basis for the language-to-waypoint idea."},{"cited_title":"Learning transferable visual models from natural language supervi- sion,","cited_arxiv_id":null,"evidence_quote":"Foundational contrastive vision-language model that underlies the grounding capability used for pixel pointing."}],"review_version":1}