{"id":"cfb69a76-406e-4275-972e-028ca83e66f0","arxiv_id":"2412.11672","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"LLM-DaaS fine-tunes small language models to convert natural-language drone delivery requests into structured tasks, reporting high G-Eval scores on a synthetic dataset.","lead":"This paper builds a drone delivery system that uses fine-tuned language models to turn free-text customer requests into structured delivery tasks, then plans routes with A* or Dijkstra under weather conditions. The work is an application of existing LLM fine-tuning to a niche logistics domain, with evaluation based on synthetic data and AI judges.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed near-perfect extraction accuracy is not established: only LLM-judge G-Eval scores are reported, with no exact-match metric and appendix scores inconsistent with the headline summary.","rationale":"The reader's verdict of REJECT is, in my view, correct. The paper's contribution is a fine-tuning recipe for DaaS request structuring plus weather-aware routing simulation. The pathfinding comparison (§5.2) is anecdotal and contradictory (A* beats Dijkstra on Route 46 but loses on Route 47), so it cannot support efficiency claims. The strongest claimed result is the LLM extraction accuracy, and that is where the load-bearing weakness lies. The evaluation uses G-Eval, an LLM judge, rather than exact-match against ground truth; the appendix shows only two test cases per model and aggregate numbers are inconsistent with the text. This is not a matter of disagreement with consensus; it is an internal evidentiary gap. The concrete test—exact-match accuracy on the 1,000 test requests—would settle whether the claim is real. If the repository is unavailable, that itself is a further reason the claim is currently unverifiable. There is no ad hominem here; the concern is about the metric and the reporting, not the authors' conduct. I therefore recommend leaving the reader's REJECT in place.","tokens_in":8016,"tokens_out":4225,"duration_ms":36526,"concrete_test":"Obtain the code/data repository promised in §8 and re-run the evaluation on the 1,000 post-fine-tuning test requests used in §4.3. Parse each model's generated structured request and compare it to the ground-truth structured request field-by-field (request_id, start_node, destination_node, payload), computing per-field and full-tuple exact-match accuracy and 95% confidence intervals. Then compute G-Eval for the same outputs. If full-tuple exact-match accuracy is substantially lower than the reported G-Eval scores (e.g., more than 5 percentage points lower) or if the repository is unavailable for this check, the near-perfect accuracy claim does not survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of near-perfect structured extraction (Abstract, §5.1) depends on G-Eval scores from the DeepEval framework (§4.3.4). G-Eval is an LLM-based chain-of-thought judge, not a direct measurement of field-level correctness. The paper never reports exact-match accuracy on the four target fields (request_id, start_node, destination_node, payload) for the 1,000 post-fine-tuning test requests. Moreover, the appendix tables are internally inconsistent with the headline: §5.1 states LLaMA 3.2 and Phi-3.5 both reached 0.9977, yet Appendix Table 5 shows only per-test-case LLaMA finetuned scores of 0.831 and 0.845, and Table 4 shows Phi 3.5 at 0.9600 and 0.9977 for two cases. No aggregate scores, standard deviations, or confidence intervals are given, and the two shown cases per model are a tiny, selective sample. Because the test requests themselves were generated by ChatGPT from the structured records (§4.2), the evaluation loop is self-referential: an LLM generates inputs, the model maps them, and another LLM scores the result. The load-bearing assumption is that G-Eval scores correspond to real extraction accuracy on unseen human requests; without an exact-match check on a held-out set, the reported 'near-to-perfect accuracy' is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes LLM-DaaS, a framework that fine-tunes several small LLMs (Phi-3.5, LLaMA 3.2, Gemma 2b, Qwen-2.5) to convert free-text user requests into structured DaaS task records containing request_id, start_node, destination_node, and payload. The structured requests are then used by a drone selection/composition model that incorporates real-time weather data and pathfinding algorithms (A* and Dijkstra) for routing. The paper reports G-Eval scores close to 0.99 for several fine-tuned models and compares A* and Dijkstra on two routes. The authors claim near-perfect extraction accuracy and improved operational efficiency under uncertain weather.","tokens_in":8374,"tokens_out":4961,"duration_ms":45539,"significance":"If the claimed extraction accuracy and operational improvements were validated, the work could contribute to automating drone service request handling. The idea of fine-tuning small LLMs for structured extraction in a specific domain is relevant, and the comparison of several models is a useful starting point. The paper also addresses an important practical problem: bridging free-text user input and structured drone task specifications. However, the current evaluation methodology is fundamentally circular (LLM-generated test data scored by an LLM judge), the reported scores are internally inconsistent, and the operational claims lack quantitative support. As presented, the results do not substantiate the central claims.","major_comments":[{"comment":"The evaluation is self-referential: the 5,000 structured requests are paraphrased into free-text by ChatGPT (§4.2), and the fine-tuned models are tested on 1,000 of these same ChatGPT-generated requests, with scores assigned by G-Eval, an LLM-based judge (§4.3.4). No exact-match accuracy on the four target fields is reported, and there is no held-out set of human-written requests. High G-Eval scores could therefore reflect stylistic mimicry of the generator rather than genuine extraction ability. The paper must report exact-match accuracy on a non-synthetic test set and justify the use of G-Eval as a substitute for exact field matching.","section":"§4.2, §5.1"},{"comment":"The aggregate G-Eval scores are not supported by the appendix tables. §5.1 states that LLaMA 3.2 and Phi-3.5 both achieved 0.9977, but Table 5 shows finetuned LLaMA 3.2 scores of 0.831 and 0.845, and Table 4 shows finetuned Phi-3.5 scores of 0.9600 and 0.9977. No average, standard deviation, or number of test cases is given, so the reader cannot verify the headline claim. The two shown test cases per model are a selective sample and cannot substantiate near-99% aggregate performance.","section":"§5.1 vs Appendix A"},{"comment":"The field request_id is listed as one of the four main extraction targets, but the free-text user requests in Table 1 contain no request identifier. The G-Eval explanations in Appendix A never mention request_id, which suggests that either the metric does not actually evaluate this field or the task is ill-posed. The paper must clarify how request_id can be extracted from an input that does not contain it, and whether it is part of the scored output.","section":"§4.3.1, Table 1"},{"comment":"The pathfinding evaluation is anecdotal and contradictory: A* outperforms Dijkstra on Route 46, but Dijkstra outperforms A* on Route 47. Only two routes are compared, with no statistical analysis and no explanation of why A* becomes suboptimal on Route 47. Moreover, the weather-adjusted speed function f(v, WS, WD) introduced in §3.1 is never defined, so the claimed weather-adaptive routing behavior is not reproducible. This evidence does not support the paper's operational efficiency claims.","section":"§5.2, §3.1"},{"comment":"The drone selection and composition model is described only qualitatively in §4.4. No experimental results are presented for drone selection, composition, or the integrated end-to-end system. The abstract claims 'significantly improve task accuracy, operational efficiency,' but the only quantitative results in Section 5 are the G-Eval scores and the two route comparisons. The operational efficiency claim is therefore unsupported.","section":"§4.4, §5"}],"minor_comments":[{"comment":"The appendix tables contain formatting and typographical errors, including 'explainations' in Table 2's header and inconsistent spacing in 'T able 2' and similar headings.","section":"Tables 2–5"},{"comment":"Several references are malformed or incomplete, for example [2] 'Alwateer, M., Loke, Fernando, S.' and [6] 'Dhulkefl, Elaf, Durdu, Akif, Terzioğlu, Hakan' use inconsistent author formatting and missing initials.","section":"References"},{"comment":"The Data and Code Availability section says the repository can be found 'repo' without a URL or identifier, making it impossible for readers to access the code and dataset.","section":"§8"},{"comment":"The fine-tuning setup is not described: no learning rate, number of epochs, LoRA configuration, or training framework is reported, which limits reproducibility.","section":"§4.3.2"},{"comment":"Figures 1 and 2 are referenced in the text but their captions are not shown in the manuscript body, making it difficult to interpret the system architecture and skyway map.","section":"Figures 1 and 2"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be an early-stage draft with an evaluation loop that cannot support the central claims. The lack of exact-match metrics, the internal inconsistency between the text and appendix scores, and the unresolved request_id issue are severe. The pathfinding and selection results are also too sparse to justify the operational claims. I would not consider this for publication without a fundamentally new evaluation, including a human or rule-based exact-match check on non-synthetic requests and quantitative results for the full DaaS pipeline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main thing you should know: the paper's central claim of near-to-perfect extraction accuracy is not established by the evidence. The evaluation loop is self-referential—ChatGPT generates the free-text requests, the fine-tuned LLM structures them, and another LLM (G-Eval) scores the result. They never report exact-match accuracy on the four target fields, and the appendix tables show LLaMA 3.2 finetuned scores of 0.831 and 0.845, yet the text claims 0.9977. That mismatch alone should make you suspicious of the headline numbers.\n\nWhat is genuinely new: the paper creates a synthetic dataset of 5,000 structured delivery requests paired with ChatGPT-generated free-text variants, and it compares four LLMs (Gemma 2b, LLaMA 3.2, Phi-3.5, Qwen-2.5) before and after fine-tuning on the extraction task. That dataset and the fine-tuning comparison are a legitimate extension to the DaaS domain, and the integration of real-time weather data into path planning is a reasonable systems piece. The simulation and the pathfinding comparison (A* vs Dijkstra) are anecdotal—two routes with contradictory winners—but they are clearly described as illustrative, so I would call that a minor weakness, not a fatal one.\n\nThe load-bearing flaw is the evaluation. G-Eval from DeepEval is a chain-of-thought LLM judge, not a measurement of field-level correctness. With no exact-match metric, no error bars, and only a few cherry-picked test cases in the appendix, the reader cannot verify that the models actually extract request_id, start_node, destination_node, and payload correctly. Given that the test requests were generated from the same structured templates used for training, high G-Eval scores may just reflect the model learning to mimic the generator's style. The authors also mention a code repository but provide no link, so the dataset and code are not independently checkable.\n\nWho this is for: applied AI researchers working on LLM-based structured data extraction in logistics or drone operations. The idea is worth exploring, but the current paper does not support its claims. A serious referee should require an exact-match evaluation on a human-written holdout set (or at least an LLM-agnostic metric), a transparent reporting of per-field performance, and a resolution of the appendix inconsistency. If the authors can provide that, the work could become a solid applied contribution. As it stands, the central result is unverified.","headline":"The framework is plausible and the dataset is a useful contribution, but the near-perfect accuracy claim rests on a circular LLM-judge evaluation that the appendix numbers contradict.","tokens_in":791,"tokens_out":711,"would_cite":false,"duration_ms":22078,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned LLMs convert free-text drone delivery requests into structured tasks with near-99% G-Eval scores.","keywords":["Drone-as-a-Service","fine-tuned LLM","free-text request processing","structured data extraction","weather-aware route planning","A* algorithm","G-Eval"],"falsifier":"Collect a held-out set of free-text delivery requests from people who did not help generate the training data, have human annotators mark the correct start node, destination node, and payload, and compare each fine-tuned model's per-field exact-match accuracy with its G-Eval score. Exact-match accuracy materially below the reported near-99% G-Eval scores would refute the central claim of near-to-perfect conversion.","tokens_in":7844,"feed_emoji":"🚁","tokens_out":12595,"duration_ms":106453,"temperature":0.7,"pith_summary":"LLM-DaaS proposes to make Drone-as-a-Service conversational: a fine-tuned language model reads a customer's free-text delivery request and emits a structured task containing a request id, start node, destination node, and payload weight. The paper's central claim is that fine-tuning on a dataset of 4,000 structured-plus-paraphrase pairs makes small LLMs reliable enough at this extraction to score near 99% on the G-Eval metric, with the best models reported at 0.9977. The same system pairs the extracted task with real-time weather data and A*/Dijkstra route planning, adjusting drone speed to wind and re-routing around bad conditions. If the claim holds, a DaaS operator could accept plain-language requests and run the rest of the delivery pipeline automatically, reducing the human labor and error currently needed to turn customer words into drone commands.","feed_headline":"Fine-tuned LLMs parse drone requests near 99%","feed_subtitle":"A text-to-task pipeline plus weather-aware routing lets customers book drone deliveries in plain language.","key_machinery":"The load-bearing mechanism is a fine-tuned LLM used as a text-to-structure front end. The paper defines the DaaS model as a tuple DaaS = ⟨W, P, I⟩ of weather, paths, and interaction log, and the pathfinding algorithm as PA = ⟨H, A*, D, Wad j⟩, where H is Euclidean distance, A* and Dijkstra compute shortest paths, and Wad j is a weather-adjusted speed v_adj = f(v, wind speed, wind direction). The fine-tuned LLM converts natural language into the four fields the scheduler needs; the selection/composition layer then chooses a drone or an inter-drone handoff based on battery, payload, speed, range, and local weather. The evaluation is G-Eval, an LLM-based judge that scores outputs with chain-of-thought reasoning against custom criteria.","core_discovery":"On its own terms, the paper's discovery is that the hard part of a DaaS system is not routing—A* and Dijkstra both complete the task—but the semantic gap between how a person asks for a delivery and what a scheduler can execute. The authors show that this gap can be closed by fine-tuning a small LLM on a dual-format dataset: 5,000 structured delivery requests (request_id, start_node, destination_node, payload) and generated conversational versions of the same requests. After fine-tuning on 4,000 of those pairs and testing on the remaining 1,000, the reported G-Eval scores jump from roughly 0.13-0.32 for the base models to 0.831-0.9977 after training, with Phi-3.5 and LLaMA 3.2 reported at 0.9977 and Qwen-2.5 at 0.9887. The system then feeds the structured request into a selection module that matches drones by battery, payload capacity, speed, and range, and into a weather-aware planner that adjusts speed as a function of wind speed and direction and re-plans with A* or Dijkstra. In the paper's two-route comparison under identical weather, A* beat Dijkstra on Route 46 (1h51 vs 1h57 over 113.91 km vs 119.08 km) while Dijkstra beat A* on Route 47 (2h59 vs 3h04 over 179.08 km vs 187.00 km), so neither algorithm dominates.","pith_inferences":["Editorial extension: the training data comes from prompt-generated paraphrases built from structured requests, so high G-Eval scores may overstate performance on the long tail of human phrasing; a field-exact-match test on independently written requests would settle this.","Editorial extension: G-Eval is an LLM-based judge whose scores summarize the judge's assessment, and those scores can be high even when a field value is wrong in a way the judge tolerates. Reporting per-field exact-match precision alongside G-Eval would make the near-99% claim easier to falsify.","Editorial extension: the A*/Dijkstra comparison covers only two routes under one weather setting; the mixed winner suggests route selection should be made per mission. The paper's four-year simulated weather logs provide the material for a broader comparison that the paper does not present."],"forward_implications":["A customer can describe pickup, dropoff, and package weight in free text and the backend receives a structured request ready for drone assignment, removing manual data entry.","The fine-tuning recipe works on small models, so the extraction step can run with lower latency and memory than a large general-purpose chatbot.","Weather-adjusted routing yields concrete savings in at least one simulated route (about 5 km and 6 minutes on Route 46), so operational decisions should use current weather, not fixed shortest-path tables.","Because the structured request is a small fixed tuple, the same extraction module can be paired with different selection/composition policies or fleet simulators."],"supporting_citations":[{"why":"This survey is cited for the claim that LLMs can produce structured outputs from natural-language input, the premise for using an LLM as the request parser.","marker":"[5]"},{"why":"This work is cited for automating task-oriented LLM-based chatbots, the approach the fine-tuning step builds on.","marker":"[14]"},{"why":"This paper is cited for uncertainty-aware DaaS composition, the groundwork for the weather-adaptive selection and composition module.","marker":"[8]"},{"why":"This paper is cited for evidence that weather conditions impact drone operations and require real-time path adaptation.","marker":"[13]"},{"why":"This paper is cited for drone flight scheduling under battery and temperature uncertainty, supporting the simulation of battery and weather constraints.","marker":"[12]"},{"why":"This paper is cited for simulation-based DaaS and on-drone decision-making, the basis for the selection and composition model.","marker":"[2]"}],"fun_headline_variants":["LLM-DaaS: From plain text to drone missions","Fine-tuned LLM parses drone requests to tasks","Weather-aware LLM drives drone-as-a-service","Chat to drone: LLM converts requests to flights"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that prompt-generated free-text requests are representative of how real users will phrase delivery requests, and that G-Eval, an LLM judge, measures true extraction correctness; if either assumption fails, the near-99% accuracy may not transfer to real deployments.","fun_headline_variants_meta":{"raw":{"variants":["LLM-DaaS: From plain text to drone missions","Fine-tuned LLM parses drone requests to tasks","Weather-aware LLM drives drone-as-a-service","Chat to drone: LLM converts requests to flights"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000777,"raw_usage":{"total_tokens":3530,"prompt_tokens":1136,"completion_tokens":2394,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":752,"completion_tokens_details":{"reasoning_tokens":2330}},"tokens_in":752,"tokens_out":2394,"duration_ms":18133,"temperature":1.0,"reasoning_tokens":2330,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:41:11.889856+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a held-out set of free-text delivery requests from people who did not help generate the training data, have human annotators mark the correct start node, destination node, and payload, and compare each fine-tuned model's per-field exact-match accuracy with its G-Eval score. Exact-match accuracy materially below the reported near-99% G-Eval scores would refute the central claim of near-to-perfect conversion.","supporting_citations":[{"cited_title":"Journal of Artificial Intelligence (2023)","cited_arxiv_id":null,"evidence_quote":"This survey is cited for the claim that LLMs can produce structured outputs from natural-language input, the premise for using an LLM as the request parser."},{"cited_title":"In: International Conference on AI (2023)","cited_arxiv_id":null,"evidence_quote":"This work is cited for automating task-oriented LLM-based chatbots, the approach the fine-tuning step builds on."},{"cited_title":"IEEE Transactions on Systems, Man, and Cybernetics (2020)","cited_arxiv_id":null,"evidence_quote":"This paper is cited for uncertainty-aware DaaS composition, the groundwork for the weather-adaptive selection and composition module."},{"cited_title":"Journal of UAV Path Planning (2022)","cited_arxiv_id":null,"evidence_quote":"This paper is cited for evidence that weather conditions impact drone operations and require real-time path adaptation."},{"cited_title":"Journal of UAV Systems (2022)","cited_arxiv_id":null,"evidence_quote":"This paper is cited for drone flight scheduling under battery and temperature uncertainty, supporting the simulation of battery and weather constraints."},{"cited_title":"Simulation Systems (2021)","cited_arxiv_id":null,"evidence_quote":"This paper is cited for simulation-based DaaS and on-drone decision-making, the basis for the selection and composition model."}],"review_version":1}