REVIEW 5 major objections 5 minor 4 cited by
LLM-DaaS: LLM-driven Drone-as-a-Service Operations from Text User Requests
T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Fine-tuned LLMs convert free-text drone delivery requests into structured tasks with near-99% G-Eval scores.
desk verdict The framework is plausible and the dataset is a useful contribution, but the near-perfect accuracy claim rests on a circular LLM-judge evaluation that the appendix numbers contradict. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a fine-tuned LLM used as a text-to-structure front end. The paper defines the DaaS model as a tuple DaaS = ⟨W, P, I⟩ of weather, paths, and interaction log, and the pathfinding algorithm as PA = ⟨H, A*, D, Wad j⟩, where H is Euclidean distance, A* and Dijkstra compute shortest paths, and Wad j is a weather-adjusted speed v_adj = f(v, wind speed, wind direction). The fine-tuned LLM converts natural language into the four fields the scheduler needs; the selection/composition layer then chooses a drone or an inter-drone handoff based on battery, payload, speed, range, and local weather. The evaluation is G-Eval, an LLM-based judge that scores outputs with chain-of-thought reasoning against custom criteria.
What would settle it
Collect a held-out set of free-text delivery requests from people who did not help generate the training data, have human annotators mark the correct start node, destination node, and payload, and compare each fine-tuned model's per-field exact-match accuracy with its G-Eval score. Exact-match accuracy materially below the reported near-99% G-Eval scores would refute the central claim of near-to-perfect conversion.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that the hard part of a DaaS system is not routing—A* and Dijkstra both complete the task—but the semantic gap between how a person asks for a delivery and what a scheduler can execute. The authors show that this gap can be closed by fine-tuning a small LLM on a dual-format dataset: 5,000 structured delivery requests (request_id, start_node, destination_node, payload) and generated conversational versions of the same requests. After fine-tuning on 4,000 of those pairs and testing on the remaining 1,000, the reported G-Eval scores jump from roughly 0.13-0.32 for the base models to 0.831-0.9977 after training, with Phi-3.5 and LLaMA 3.2 reported at 0.9977 and Qwen-2.5 at 0.9887. The system then feeds the structured request into a selection module that matches drones by battery, payload capacity, speed, and range, and into a weather-aware planner that adjusts speed as a function of wind speed and direction and re-plans with A* or Dijkstra. In the paper's two-route comparison under identical weather, A* beat Dijkstra on Route 46 (1h51 vs 1h57 over 113.91 km vs 119.08 km) while Dijkstra beat A* on Route 47 (2h59 vs 3h04 over 179.08 km vs 187.00 km), so neither algorithm dominates.
Load-bearing premise
The load-bearing premise is that prompt-generated free-text requests are representative of how real users will phrase delivery requests, and that G-Eval, an LLM judge, measures true extraction correctness; if either assumption fails, the near-99% accuracy may not transfer to real deployments.
Editorial extensions
If this is right
- A customer can describe pickup, dropoff, and package weight in free text and the backend receives a structured request ready for drone assignment, removing manual data entry.
- The fine-tuning recipe works on small models, so the extraction step can run with lower latency and memory than a large general-purpose chatbot.
- Weather-adjusted routing yields concrete savings in at least one simulated route (about 5 km and 6 minutes on Route 46), so operational decisions should use current weather, not fixed shortest-path tables.
- Because the structured request is a small fixed tuple, the same extraction module can be paired with different selection/composition policies or fleet simulators.
Reading between the lines
- Editorial extension: the training data comes from prompt-generated paraphrases built from structured requests, so high G-Eval scores may overstate performance on the long tail of human phrasing; a field-exact-match test on independently written requests would settle this.
- Editorial extension: G-Eval is an LLM-based judge whose scores summarize the judge's assessment, and those scores can be high even when a field value is wrong in a way the judge tolerates. Reporting per-field exact-match precision alongside G-Eval would make the near-99% claim easier to falsify.
- Editorial extension: the A*/Dijkstra comparison covers only two routes under one weather setting; the mixed winner suggests route selection should be made per mission. The paper's four-year simulated weather logs provide the material for a broader comparison that the paper does not present.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes LLM-DaaS, a framework that fine-tunes several small LLMs (Phi-3.5, LLaMA 3.2, Gemma 2b, Qwen-2.5) to convert free-text user requests into structured DaaS task records containing request_id, start_node, destination_node, and payload. The structured requests are then used by a drone selection/composition model that incorporates real-time weather data and pathfinding algorithms (A* and Dijkstra) for routing. The paper reports G-Eval scores close to 0.99 for several fine-tuned models and compares A* and Dijkstra on two routes. The authors claim near-perfect extraction accuracy and improved operational efficiency under uncertain weather.
Significance. If the claimed extraction accuracy and operational improvements were validated, the work could contribute to automating drone service request handling. The idea of fine-tuning small LLMs for structured extraction in a specific domain is relevant, and the comparison of several models is a useful starting point. The paper also addresses an important practical problem: bridging free-text user input and structured drone task specifications. However, the current evaluation methodology is fundamentally circular (LLM-generated test data scored by an LLM judge), the reported scores are internally inconsistent, and the operational claims lack quantitative support. As presented, the results do not substantiate the central claims.
major comments (5)
- [§4.2, §5.1] The evaluation is self-referential: the 5,000 structured requests are paraphrased into free-text by ChatGPT (§4.2), and the fine-tuned models are tested on 1,000 of these same ChatGPT-generated requests, with scores assigned by G-Eval, an LLM-based judge (§4.3.4). No exact-match accuracy on the four target fields is reported, and there is no held-out set of human-written requests. High G-Eval scores could therefore reflect stylistic mimicry of the generator rather than genuine extraction ability. The paper must report exact-match accuracy on a non-synthetic test set and justify the use of G-Eval as a substitute for exact field matching.
- [§5.1 vs Appendix A] The aggregate G-Eval scores are not supported by the appendix tables. §5.1 states that LLaMA 3.2 and Phi-3.5 both achieved 0.9977, but Table 5 shows finetuned LLaMA 3.2 scores of 0.831 and 0.845, and Table 4 shows finetuned Phi-3.5 scores of 0.9600 and 0.9977. No average, standard deviation, or number of test cases is given, so the reader cannot verify the headline claim. The two shown test cases per model are a selective sample and cannot substantiate near-99% aggregate performance.
- [§4.3.1, Table 1] The field request_id is listed as one of the four main extraction targets, but the free-text user requests in Table 1 contain no request identifier. The G-Eval explanations in Appendix A never mention request_id, which suggests that either the metric does not actually evaluate this field or the task is ill-posed. The paper must clarify how request_id can be extracted from an input that does not contain it, and whether it is part of the scored output.
- [§5.2, §3.1] The pathfinding evaluation is anecdotal and contradictory: A* outperforms Dijkstra on Route 46, but Dijkstra outperforms A* on Route 47. Only two routes are compared, with no statistical analysis and no explanation of why A* becomes suboptimal on Route 47. Moreover, the weather-adjusted speed function f(v, WS, WD) introduced in §3.1 is never defined, so the claimed weather-adaptive routing behavior is not reproducible. This evidence does not support the paper's operational efficiency claims.
- [§4.4, §5] The drone selection and composition model is described only qualitatively in §4.4. No experimental results are presented for drone selection, composition, or the integrated end-to-end system. The abstract claims 'significantly improve task accuracy, operational efficiency,' but the only quantitative results in Section 5 are the G-Eval scores and the two route comparisons. The operational efficiency claim is therefore unsupported.
minor comments (5)
- [Tables 2–5] The appendix tables contain formatting and typographical errors, including 'explainations' in Table 2's header and inconsistent spacing in 'T able 2' and similar headings.
- [References] Several references are malformed or incomplete, for example [2] 'Alwateer, M., Loke, Fernando, S.' and [6] 'Dhulkefl, Elaf, Durdu, Akif, Terzioğlu, Hakan' use inconsistent author formatting and missing initials.
- [§8] The Data and Code Availability section says the repository can be found 'repo' without a URL or identifier, making it impossible for readers to access the code and dataset.
- [§4.3.2] The fine-tuning setup is not described: no learning rate, number of epochs, LoRA configuration, or training framework is reported, which limits reproducibility.
- [Figures 1 and 2] Figures 1 and 2 are referenced in the text but their captions are not shown in the manuscript body, making it difficult to interpret the system architecture and skyway map.
Circularity Check
No circular derivation; the LLM-extraction claim rests on a synthetic benchmark and an LLM judge, which are external-validity and reporting concerns, not constructed equivalence.
full rationale
The claimed near-perfect extraction accuracy is supported by G-Eval scores on 1,000 test requests generated by ChatGPT from the same structured-request templates used to build the fine-tuning set (§4.2, §4.3). This is a genuine threat to external validity: the benchmark may measure how well the models invert ChatGPT's paraphrasing style rather than how they handle real user language, and G-Eval is an LLM-based judge rather than an exact field-match metric. However, this does not make the derivation circular in the defined sense. The test labels are not derived from the models' outputs, the models are not trained to optimize G-Eval, and the reported scores are discriminative (base models score 0.13–0.32, fine-tuned 0.83–0.9977). The inconsistency between the appendix per-case scores and the Section 5.1 aggregate claims is a reporting weakness, not a constructed equivalence. The only self-citation (Hamdi et al. [8], an author of this paper) is background on uncertainty-aware DaaS composition and is not the load-bearing basis of any prediction. The central derivation chain is self-contained; the concerns are about benchmark realism and reporting transparency, not circularity.
Assumptions & free parameters
free parameters (3)
- Speed adjustment function f(v, WS, WD)
- LLM fine-tuning hyperparameters
- Simulation parameters (battery degradation, maintenance schedule, weather thresholds)
assumptions (5)
- standard math Euclidean distance is an admissible heuristic for A* in the skyway network
- domain assumption Weather conditions (wind, temperature, precipitation) affect drone speed and route safety
- domain assumption The skyway network is a graph of nodes and edges with known coordinates
- ad hoc to paper ChatGPT-generated paraphrases of structured requests are representative of real user language
- ad hoc to paper G-Eval scores from an LLM judge are a valid measure of extraction correctness
Cite this review
Pith. "Pith review of LLM-DaaS: LLM-driven Drone-as-a-Service Operations from Text User Requests." pith.science (2026). https://pith.science/paper/H7BS6WCB
@misc{pith2026241211672,
author = {Pith},
title = {Pith review of: LLM-DaaS: LLM-driven Drone-as-a-Service Operations from Text User Requests},
year = {2026},
howpublished = {\url{https://pith.science/paper/H7BS6WCB}},
note = {Machine review of arXiv:2412.11672}
}
read the original abstract
We propose LLM-DaaS, a novel Drone-as-a-Service (DaaS) framework that leverages Large Language Models (LLMs) to transform free-text user requests into structured, actionable DaaS operation tasks. Our approach addresses the key challenge of interpreting and structuring natural language input to automate drone service operations under uncertain conditions. The system is composed of three main components: free-text request processing, structured request generation, and dynamic DaaS selection and composition. First, we fine-tune different LLM models such as Phi-3.5, LLaMA-3.2 7b and Gemma 2b on a dataset of text user requests mapped to structured DaaS requests. Users interact with our model in a free conversational style, discussing package delivery requests, while the fine-tuned LLM extracts DaaS metadata such as delivery time, source and destination locations, and package weight. The DaaS service selection model is designed to select the best available drone capable of delivering the requested package from the delivery point to the nearest optimal destination. Additionally, the DaaS composition model composes a service from a set of the best available drones to deliver the package from the source to the final destination. Second, the system integrates real-time weather data to optimize drone route planning and scheduling, ensuring safe and efficient operations. Simulations demonstrate the system's ability to significantly improve task accuracy, operational efficiency, and establish LLM-DaaS as a robust solution for DaaS operations in uncertain environments.
Figures
Forward citations
Cited by 4 Pith papers
-
Arabic Large Language Models for Medical Text Generation
Fine-tuning Mistral-7B on a new Arabic medical Q&A dataset yields BERTScore F1 of 68.50 percent, higher than LLaMA-2, GPT-2 Medium, AraGPT2, and BLOOM in this study.
-
An Agentic Flow for Finite State Machine Extraction using Prompt Chaining
A three-stage LLM prompt-chaining system extracts FSM rulebooks from RFC documents, achieving F1 scores near 85% on FTP and RTSP.
-
A Multi-Layered Large Language Model Framework for Disease Prediction
LLAMA3-based NER preprocessing plus LoRA fine-tuning of CAMeL-BERT gives 83% disease type and 69% severity accuracy on Arabic telehealth posts, though without significance testing.
-
Retrieval Augmented Generation Based LLM Evaluation For Protocol State Machine Inference With Chain-of-Thought Reasoning
A RAG-plus-chain-of-thought agent improves BLEU, ROUGE, and WER scores for RTSP packet seed generation over Gemma-2-9B and Llama-3-8B baselines, though the evaluation does not test actual fuzzing effectiveness.
Reference graph
Works this paper leans on
-
[1]
IEEE Robotics and Automation Letters (2021)
Alkouz, B., Bouguettaya, A., Mistry, S.: Swarm-based drone-as-a-service (sdaas) for delivery. IEEE Robotics and Automation Letters (2021)
work page 2021
-
[2]
Alwateer, M., Loke, Fernando, S.: Drones-as-a-service: A simulation-based analysis for on-drone decision-making. Simulation Systems (2021)
work page 2021
-
[3]
Banerjee, D., Singh, P., Avadhanam, A., Srivastava, S.: Benchmarking llm-powered chatbots: Methods and metrics. Journal of AI Systems (2022)
work page 2022
-
[4]
Journal of AI and System Engineering (2020)
Choi, S.C., Sung, N.M., Park, J.H., Ahn, I.Y., Kim, J.: Enabling drone as a ser- vice: onem2m-based uav/drone management system. Journal of AI and System Engineering (2020)
work page 2020
-
[5]
Journal of Artificial Intelligence (2023)
Dam, S., Hong, C.S., Qiao, Y., Zhang, C.: A complete survey on llm-based ai chatbots. Journal of Artificial Intelligence (2023)
work page 2023
-
[6]
In: IEEE Aerospace Conference (2019)
Dhulkefl, Elaf, Durdu, Akif, Terzioğlu, Hakan: Dijkstra algorithm using uav path planning. In: IEEE Aerospace Conference (2019)
work page 2019
-
[7]
Journal of Transportation Research (2019)
Dorling, K., Heinrichs, J., Messier, G.G., Magierowski, S.: Vehicle routing problems for drone delivery. Journal of Transportation Research (2019)
work page 2019
-
[8]
IEEE Transactions on Systems, Man, and Cybernetics (2020)
Hamdi, A., Salim, F.D., Kim, D.Y., Neiat, A.G., Bouguettaya, A.: Drone-as-a- service composition under uncertainty. IEEE Transactions on Systems, Man, and Cybernetics (2020)
work page 2020
Show all 17 references
-
[9]
Journal of AI Path Planning (2020)
He, Y., Hou, T., Wang, M.: A new method for unmanned aerial vehicle path planning in complex environments. Journal of AI Path Planning (2020)
2020
-
[10]
Journal of Business and Automation (2021)
Iyelolu, Toluwalase, Agu, Edith, Ijomah, Tochukwu: Ai-driven chatbots in crm: New opportunities for customer engagement. Journal of Business and Automation (2021)
2021
-
[11]
Journal of Business Automation (2021)
Khneyzer, C., Boustany, Z.: Ai-driven chatbots in crm: Economic and managerial implications. Journal of Business Automation (2021)
2021
-
[12]
Journal of UAV Systems (2022)
Kim, S.J., Lim, G.J., Cho, J.: Drone flight scheduling under uncertainty on battery duration and air temperature. Journal of UAV Systems (2022)
2022
-
[13]
Journal of UAV Path Planning (2022)
Patel, S., et al.: Drone flight scheduling under uncertainty. Journal of UAV Path Planning (2022)
2022
-
[14]
In: International Conference on AI (2023)
Sánchez Cuadrado, J., Pérez-Soler, S., Guerra, E., De Lara, J.: Automating the development of task-oriented llm-based chatbots. In: International Conference on AI (2023)
2023
-
[15]
Journal of Sys- tem Design (2021)
Wang, L., et al.: Drones-as-a-service: A simulation-based analysis. Journal of Sys- tem Design (2021)
2021
-
[16]
Journal of AI and ML Research (2023)
Yeom, J., Lee, Byun, H.: Tc-llama 2: Fine-tuning llm for technology and commer- cialization applications. Journal of AI and ML Research (2023)
2023
-
[17]
Yonetani, R., Taniai, T., Barekatain, M., Nishimura, M., Kanezaki, A.: Path plan- ning using neural a* search. In: International Conference on Robotics and Au- tomation (2023) Appendix A:G-Eval Scores and Explanations for Different Models T able 2.Gemma 2b G-Eval scores and ex...
2023
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.