Pith. sign in

REVIEW 5 major objections 6 minor 81 references

LLM-Agents Driven Automated Simulation Testing and Analysis of small Uncrewed Aerial Systems

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AutoSimTest claims that four collaborating LLM agents can automate small-drone simulation testing from scenario design through flight-log analysis.

desk verdict A plausible multi-agent framework for sUAS simulation testing with one solid result (7/7 sensor failure detection), but the generalizability claim rests on unexecuted scripts and self-referential validation. read the letter →

arxiv 2501.11864 v1 pith:4GXQYD7G submitted 2025-01-21 cs.SE

Venkata Sai Aswath Duvvuru , Bohan Zhang , Michael Vierhauser , Ankit Agrawal This is my paper · ORCID
classification cs.SE
keywords simulationtestingsmalluncrewedaerialsystemsLLMagentsretrieval-augmentedgenerationscenarioflightloganalysisPX4ArduPilot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AutoSimTest is a pipeline of four collaborating LLM-powered agents that take a developer's high-level testing goal and carry it through scenario design, execution, and analysis for small drones. It claims that a developer who types something like 'test this drone in foggy city surveillance' can receive a realistic scenario blueprint, valid mission and environment scripts for PX4 or ArduPilot controllers, executed simulations, and an automatically generated flight-log report. The paper argues this replaces the largely manual work of imagining scenarios, configuring simulators, writing missions, and interpreting thousands of flight parameters, with results that reduce effort and broaden test coverage. It also reports that the analytics agent detected all seven injected sensor failures in flight logs and that interviewed developers saw the tool as useful.

What carries the argument

The load-bearing object is the agent pipeline plus its two knowledge stores. The S-Agent uses retrieval-augmented generation over a corpus of real-world drone incident reports to ground generated scenarios; the M-Agent and Env-Agent act as structured translators that emit JSON mission files and simulator settings, constrained by prompt rules and validated by rule-based checks before execution; and the Analytics-Agent pairs a flight-controller parameter knowledge base with a vision-capable language model that reads plotted time series and produces text reports. The rule-based validators are what let the pipeline claim executability without a human checking every script.

What would settle it

Take any of the 25 generated mission and environment scripts that passed validation and run it, unmodified, in the corresponding simulator; if a script fails to start, produces invalid waypoints, or violates a stated test property, the executability claim collapses. Similarly, present the analytics agent with a flight log containing a sensor failure not drawn from its knowledge base and check whether the report identifies the failure.

Watch

Extended reading notes

Core claim

The central claim is that one multi-agent framework can automate the full sUAS simulation-testing loop across different flight controllers and simulators. Using a retrieval-augmented knowledge base of real drone incidents, the scenario agent proposes environment, mission, and test-property blueprints; two translator agents convert those blueprints into executable mission scripts and simulator configuration files; and an analytics agent turns flight logs into a parameter-focused report and answers follow-up questions interactively. Evaluations with two systems under test, one built on PX4 and one on ArduPilot, showed the generated scripts ran unchanged, the 25 blueprints across five use cases were judged relevant and diverse, and the analytics agent identified all seven injected sensor failures. The authors therefore argue that simulation testing of drones can shift from manual, expert-dependent effort to automated, iterative, high-coverage testing.

Load-bearing premise

The claim that the framework generalizes across use cases rests on treating JSON-schema and rule-based validation as proof that scripts are executable, because only the two city-surveillance scenarios were actually run in simulation.

Editorial extensions

If this is right

  • PX4 and ArduPilot systems under test can be driven by automatically generated missions, removing the manual step of hand-writing waypoint plans in ground-control tools.
  • Because blueprints are sourced from recorded incidents, test suites can reach scenarios developers would not think to specify, such as a drone that re-targets from a person to a dog.
  • Flight-log analysis shifts from browsing hundreds of controller parameters to asking high-level questions, with the agent selecting and plotting the few relevant series.
  • The framework is fast and structured enough to slot into CI/CD pipelines, producing an analysis report after each change to the system under test.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's evidence, a fair next test is to execute all 25 generated scenarios in simulators and check for runtime or semantic failures, since current validity evidence is rule-based rather than execution-based.
  • The architecture separates scenario knowledge from the underlying generation model, so swapping in a different LLM should change style but not pipeline behavior; this is a testable extension the paper leaves implicit.
  • The RAG-plus-agents structure could transfer to autonomous ground vehicles, as the paper notes, but the transfer requires rebuilding the incident corpus, script schemas, and parameter knowledge base rather than reusing them directly.
  • The analytics agent's detection of seven injected sensor failures does not show how it handles compound failures or faults with no obvious parameter spike; adding such cases would test the analysis claim more sharply.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes AutoSimTest, a multi-LLM-agent framework for automating simulation testing of small uncrewed aerial systems. The framework spans three phases: scenario blueprint generation by an S-Agent using RAG over real-world incident data; generation and validation of mission and environment scripts by M-Agent and Env-Agent; and automated/interactive flight-log analysis by an Analytics-Agent using a PX4/ArduPilot parameter knowledge base. Evaluation includes two executed city-surveillance scenarios on PX4/AirSim and ArduPilot/SITL, generation of 25 scenario blueprints with mission/environment scripts across five use cases (only script-validated, not executed), RAG-based quality metrics for the S-Agent and Analytics-Agent, detection of seven injected sensor failures in PX4 logs, and a perception study with four sUAS developers. The paper claims significant improvements in efficiency and scope of sUAS testing.

Significance. If the claims were fully supported, AutoSimTest would be a notable contribution to automated CPS/sUAS testing, particularly the Analytics-Agent's top-down log analysis and the end-to-end pipeline from natural-language scenario descriptions to executed missions. Strengths include the concrete demonstration on two real flight controllers, the successful detection of all seven injected sensor failures, a publicly available codebase, and the authors' candid acknowledgment of several limitations in Section IX. However, the evaluation evidence as presented does not substantiate the headline efficiency/scope claims: there is no baseline or direct measurement, and the five-use-case generalization rests on scripts that were not executed and in at least one case involve scenario elements the authors admit are not simulatable. The central idea is promising, but the manuscript needs additional evidence and a recalibration of its claims.

major comments (5)
  1. [§VII-A2, §VI-A, §VIII-2] The generalizability claim in RQ1 ('AUTO SIMTEST can be applied to test across diverse sUAS use cases') is not supported by the evidence. Only the two city-surveillance scenarios were actually executed in simulation (Section VII-A1); the remaining 23 mission and environment scripts were judged valid only through JSON-schema and rule-based validation (Section VII-A2). That validation checks syntactic format and value ranges, not whether the simulation can instantiate the scenario or whether the declared test property is observable in the logs. The paper itself concedes in Section VIII-2 that key scenario elements, such as the movement pattern of a drowning person in a river search-and-rescue scenario, are not supported by AirSim or similar tools. The executed scenarios show feasibility for waypoint-based city surveillance but do not establish that all five use cases were meaningfully tested.
  2. [§V-B, Table II, §VI-A] The Rule-Based Validator is self-referential with respect to the agent prompts. The rules used for validation (e.g., wind = [0,50] mph, altitude <= 400 ft, velocity = [0,30] mph) are the same rules included in the agents' prompt design, as shown in Table II. Passing such a validator demonstrates only that the LLM followed the output constraints written into the prompt; it does not demonstrate executability on a real simulator or fidelity to the scenario's semantic intent. This matters because the all-scripts-valid result in Section VII-A2 is the sole evidence for the five-use-case generalization. In addition, the JSON-schema used for the five use cases defines a 'hypothetical SuT' rather than the actual PX4/ArduPilot systems, so even the syntactic check is not tied to the SuTs used in the executed cases.
  3. [§VII, Abstract, §XI] The claimed efficiency improvement is never directly measured. The abstract states that AutoSimTest 'significantly improves the efficiency and scope of the sUAS testing process' and Section XI claims it is 'substantially reducing the time and effort required for testing and validation,' but no baseline comparison is provided. There is no measurement of time saved, effort reduction, or comparison against manual testing or existing tools such as QGroundControl, MissionPlanner, or DroneReqValidator. The evaluation consists of feasibility demonstrations, RAG-based quality metrics, script validation, and a four-participant perception study; none of these directly quantify efficiency. This is a load-bearing gap for the paper's central claim.
  4. [§VII-C, §IX] The perception study is used to support the contribution that AutoSimTest provides 'valuable support to both novice and experienced developers,' but the evidence is preliminary. Only four developers participated, and only three interacted with the S-Agent; no formal user study with predefined tasks or usability metrics was conducted. The paper itself acknowledges in Section IX that 'a formal user study is needed to obtain deeper insights.' As presented, the RQ3 results are anecdotal and should be framed as preliminary insights rather than a demonstrated contribution.
  5. [§VII-B4, Table VIII] The Analytics-Agent's sensor failure detection is the strongest positive result, but its scope is narrow. Each of the seven PX4 flight logs contains exactly one injected failure (Section VI-A), and there are no control logs without failures to assess the false-positive rate. The 'detected' verdicts in Table VIII are the agent's own textual reports, and no independent quantitative accuracy metric (e.g., precision/recall over a labeled log corpus) is reported. The claim that the agent can 'automatically identify common sensor failures' is credible for the seven specific logs tested, but the evaluation does not yet support broader statements about detection reliability.
minor comments (6)
  1. [§VII-B] The text states that Llama 3 is 'a 3 trillion parameter model'; Llama 3 released models have at most 400B parameters, so this should be corrected or the intended model clarified.
  2. [§VIII-2] There is a typo: 'Airim' should be 'AirSim'.
  3. [§VIII-1] The phrase 'analyzing parameters of autopilots of autonomous cars' is awkward; consider 'analyzing autopilot parameters of autonomous cars.'
  4. [§IV-A, §VI-A] The notation for the flight controller is inconsistent: 'Px4' and 'PX4' are both used. Please standardize to 'PX4'.
  5. [Table II] In the A-Agent (Interactive) column, the Rules entry repeats 'Analysis Completeness'; the intended distinction between automated and interactive rules should be clarified.
  6. [§VII-A1] The claim that mission scripts were used 'without making any changes' in both simulation tools would be easier to verify if the executed mission and environment scripts were included in the supplementary materials alongside the unexecuted use-case scripts.

Circularity Check

1 steps flagged · score 6.0 of 10

RQ1's 'valid in all 25' generalizability result is partly self-referential: the rule-based validator checks the same rules that were written into the agent prompts, so executability is concluded partly by construction.

  1. self definitional [Section VII-A2, Validation of Generated Executable Scripts]
    "We used Rule-based Validators for both the Env-Agent and M-Agent to verify if the generated scripts are valid and executable using the Rules (cf. Table II) that were enforced as part of the agents’ prompt."

    The validator applies exactly the Rules that were already part of the M-Agent and Env-Agent prompts (Table II lists the same Rules column used for generation and validation). Thus 'valid and executable' is defined as 'satisfies the prompt-enforced rules'; the all-25-pass result is a self-consistency check rather than independent evidence of executability. The paper then uses this result to answer RQ1 ('valid simulation and test scripts were generated in all 25 test executions') and to generalize beyond the two city-surveillance scenarios actually run in AirSim and ArduPilot-SITL. Since the validation criterion is identical to the generation constraint, the generalizability conclusion is largely forced by construction.

full rationale

AutoSimTest's core pipeline is not definitionally circular: scenario blueprints use an external incident knowledge base, two city-surveillance missions were actually executed on PX4-AirSim and ArduPilot-SITL, the Analytics-Agent was tested against seven injected sensor failures, and developer feedback was collected externally. The main circularity is localized to RQ1's generalizability evidence. The M-Agent and Env-Agent prompts include a Rules component and a Sample-of-Expected-Output JSON schema; the Rule-Based Validator then checks the same rules and schema. The paper's statement that 'all scripts were valid and passed the validity checks, indicating that M-Agent and Env-Agent can generate valid mission scripts and simulator settings' therefore treats a self-consistency check as proof of executability. Because only two of the 25 scenarios were actually simulated, the 'valid in all 25 test executions' answer to RQ1 is partly forced by construction. The author self-rating of scenario quality and the RAGAs LLM-as-judge are validity threats rather than circular derivations, and self-citations to prior simulator work are not load-bearing for the central claims. Overall, the result is partially circular in its generalization claim, while the feasibility and analytics contributions retain independent grounding.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No new physical or speculative entities are introduced; the LLM agents are implemented software components with in-paper demonstrations. The main added assumptions are the validation rule thresholds and the domain assumptions listed above, all of which the central feasibility and generalizability claims depend on.

free parameters (1)
  • Validation rule thresholds = wind 0-50 mph, altitude <= 400 ft, velocity 0-30 mph, light intensity 0-10, delivery distance < 2 miles
    Hand-chosen constants used in the prompt Rules and the Rule-Based Validator for Env-Agent and M-Agent. Every 'valid script' outcome depends on these choices, and they are not derived from an external standard in the paper.
assumptions (4)
  • domain assumption Rule-based and JSON-schema validation is sufficient to establish that a generated script is executable and faithful to the scenario blueprint.
    Used in Section VI-A and VII-A2 to generalize from two executed SuT cases to 25 generated scripts across five use cases.
  • domain assumption The incident knowledge base built from AAIB, Wikipedia, and NASA ASRS reports is representative enough to generate useful scenarios for the five chosen use cases.
    The RAG relevance and diversity results depend on this coverage, and the paper attributes the lower Precision Agriculture scores to limited incident data in Section VII-B1.
  • domain assumption Flight logs with artificially injected sensor failures are representative of real-world failures and are sufficient for judging the correctness of the Analytics-Agent.
    Section VI-A, SuT Flight Logs; the central RQ2 detection result uses only these 7 logs.
  • ad hoc to paper LLM-as-a-critic evaluation with RAGAs and Llama 3 is a valid proxy for the factual quality of generated scenario and analysis text.
    The paper itself flags this as a construct validity threat in Section IX, and no external ground truth is used for these metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM-Agents Driven Automated Simulation Testing and Analysis of small Uncrewed Aerial Systems." pith.science (2026). https://pith.science/paper/4GXQYD7G

@misc{pith2026250111864,
  author       = {Pith},
  title        = {Pith review of: LLM-Agents Driven Automated Simulation Testing and Analysis of small Uncrewed Aerial Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4GXQYD7G}},
  note         = {Machine review of arXiv:2501.11864}
}
read the original abstract

Thorough simulation testing is crucial for validating the correct behavior of small Uncrewed Aerial Systems (sUAS) across multiple scenarios, including adverse weather conditions (such as wind, and fog), diverse settings (hilly terrain, or urban areas), and varying mission profiles (surveillance, tracking). While various sUAS simulation tools exist to support developers, the entire process of creating, executing, and analyzing simulation tests remains a largely manual and cumbersome task. Developers must identify test scenarios, set up the simulation environment, integrate the System under Test (SuT) with simulation tools, formulate mission plans, and collect and analyze results. These labor-intensive tasks limit the ability of developers to conduct exhaustive testing across a wide range of scenarios. To alleviate this problem, in this paper, we propose AutoSimTest, a Large Language Model (LLM)-driven framework, where multiple LLM agents collaborate to support the sUAS simulation testing process. This includes: (1) creating test scenarios that subject the SuT to unique environmental contexts; (2) preparing the simulation environment as per the test scenario; (3) generating diverse sUAS missions for the SuT to execute; and (4) analyzing simulation results and providing an interactive analytics interface. Further, the design of the framework is flexible for creating and testing scenarios for a variety of sUAS use cases, simulation tools, and SuT input requirements. We evaluated our approach by (a) conducting simulation testing of PX4 and ArduPilot flight-controller-based SuTs, (b) analyzing the performance of each agent, and (c) gathering feedback from sUAS developers. Our findings indicate that AutoSimTest significantly improves the efficiency and scope of the sUAS testing process, allowing for more comprehensive and varied scenario evaluations while reducing the manual effort.

Figures

Figures reproduced from arXiv: 2501.11864 by the authors.

Figure 1
Figure 1. Overview of our AUTOSIMTEST Framework with the 3 main Phases: Scenario Blueprint Construction , Manual Validation and Feedback (blue), Scenario Specification, Validation, and Execution (green), and Scenario Analysis (yellow). AUTOSIMTEST as a first step towards automating the simula￾tion testing process for sUAS. We leverage our experience in simulation and field testing within this domain to construct and implement… view at source ↗
Figure 2
Figure 2. Env-Agent output { "mission": { "cruiseSpeed": 15, "hoverSpeed": 5, "items": [ { "AMSLAltAboveTerrain": null, "Altitude": 50, "AltitudeMode": 1, "autoContinue": true, "command": 22, "frame": 3, "params": [15,0,0, null, 47.398039859999997, 8.5455725400000003, 50 ], "type": "SimpleItem" }, {...}, {...}, {...} ], "plannedHomePosition": [ 47.397742, 8.545594, 488 ], },} [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Schematic overview of the main parts of the [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: User Interface for developers to interact with [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

81 extracted references · 70 canonical work pages

  1. [1]

    drone flight control for dji, autel drones

    Dronelink. drone flight control for dji, autel drones. https://www. dronelink.com/. (Accessed on 07/29/2024)

    2024

  2. [2]

    https://www.skydio.com/ solutions/public-safety/search-and-rescue-drones

    Drones for search and rescue — skydio. https://www.skydio.com/ solutions/public-safety/search-and-rescue-drones. (Accessed 01-07- 2024)

    work page 2024

  3. [3]

    https://www.gradio.app/

    Gradio. https://www.gradio.app/. (Accessed on 07/21/2024)

    work page 2024

  4. [4]

    https://en

    List of unmanned aerial vehicle-related incidents - wikipedia. https://en. wikipedia.org/wiki/List of unmanned aerial vehicle-related incidents. (Accessed on 07/17/2024)

    work page 2024

  5. [5]

    https://huggingface

    microsoft/phi-3-mini-128k-instruct · hugging face. https://huggingface. co/microsoft/Phi-3-mini-128k-instruct. (Accessed on 07/21/2024)

    work page 2024

  6. [6]

    https://huggingface

    microsoft/phi-3-vision-128k-instruct · hugging face. https://huggingface. co/microsoft/Phi-3-vision-128k-instruct. (Accessed on 07/21/2024)

    work page 2024

  7. [7]

    https://asrs.arc.nasa

    NASA Unmanned Aircraft Systems (UAS) Reports. https://asrs.arc.nasa. gov/docs/rpsts/uas.pdf. (Accessed on 07/17/2024)

    work page 2024

  8. [8]

    https://docs.px4.io/main/ en/debug/failure injection.html

    System failure injection — px4 guide (main). https://docs.px4.io/main/ en/debug/failure injection.html. (Accessed on 11/07/2024)

    work page 2024

Show all 81 references
  1. [9]

    https: //www.gov.uk/aaib-reports?keywords=UAS

    UAS - Air Accidents Investigation Branch reports - GOV .UK. https: //www.gov.uk/aaib-reports?keywords=UAS. (Accessed on 07/17/2024)

    2024

  2. [10]

    Abdin, S

    M. Abdin, S. A. Jacobs, A. A. Awan, J. Aneja, A. Awadallah, H. Awadalla, N. Bach, A. Bahree, A. Bakhtiari, H. Behl, et al. Phi- 3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219 , 2024

    2024 arXiv

  3. [11]

    Agrawal, S

    A. Agrawal, S. J. Abraham, B. Burger, C. Christine, L. Fraser, J. M. Hoeksema, S. Hwang, E. Travnik, S. Kumar, W. Scheirer, et al. The next generation of human-drone partnerships: Co-designing an emergency response system. In Proc. of the 2020 CHI Conf. on Human Factors in Com...

    2020

  4. [12]

    Agrawal, P

    A. Agrawal, P. Zech, and M. Vierhauser. Coupled Requirements-Driven Testing of CPS: From Simulation to Reality . In 2024 IEEE 32nd International Requirements Engineering Conference (RE) , pages 337– 344, Los Alamitos, CA, USA, Jun 2024. IEEE Computer Society

    2024

  5. [13]

    Agrawal, B

    A. Agrawal, B. Zhang, Y . Shivalingaiah, M. Vierhauser, and J. Cleland- Huang. A requirements-driven platform for validating field operations of small uncrewed aerial vehicles. In Proc. of the 31st Int’l Requirements Engineering Conf., pages 29–40. IEEE, 2023

    2023

  6. [14]

    M. N. Al Islam, M. T. Chowdhury, A. Agrawal, M. Murphy, R. Mehta, D. Kudriavtseva, J. Cleland-Huang, M. Vierhauser, and M. Chechik. Configuring mission-specific behavior in a product line of collaborating small unmanned aerial systems. Journal of Systems and Software , 197:111...

    2023

  7. [15]

    M. N. Al Islam, Y . Ma, P. Alarcon, N. Chawla, and J. Cleland-Huang. Resam: Requirements elicitation and specification for deep-learning anomaly models with applications to uav flight controllers. In Proc. of the 30th International Requirements Engineering Conf. , pages 153–

  8. [16]

    Al-Mousa, B

    A. Al-Mousa, B. H. Sababha, N. Al-Madi, A. Barghouthi, and R. Younisse. Utsim: A framework and simulator for uav air traffic integration, control, and communication. International Journal of Advanced Robotic Systems , 16(5):1729881419870937, 2019

    2019

  9. [17]

    B. S. Ali. Traffic management for drones flying in the city. International Journal of Critical Infrastructure Protection , 26:100310, 2019

    2019

  10. [18]

    Amazon Drone Delivery

    Amazon.com. Amazon Drone Delivery. https://www.aboutamazon.com/ news/operations/amazon-delivering-the-future-2023-announcements,

    2023

  11. [19]

    Anand, S

    H. Anand, S. A. Rees, Z. Chen, A. J. Poruthukaran, S. Bearman, L. G. P. Antervedi, and J. Das. Openuav cloud testbed: a collaborative design studio for field robotics. In Proc. of the 17th Int’l Conference on Automation Science and Engineering , pages 724–731. IEEE, 2021

    2021

  12. [20]

    Anweiler and D

    S. Anweiler and D. Piwowarski. Multicopter platform prototype for environmental monitoring. Journal of Cleaner Production , 155:204– 211, 2017

    2017

  13. [21]

    SITL Simulator

    ArduPilot. SITL Simulator. http://ardupilot.org/dev/docs/ sitl-simulator-software-in-the-loop.html, 2018. (Accessed 01-07- 2024)

    2018

  14. [22]

    Ardupilot – open source autopilot software

    Ardupilot. Ardupilot – open source autopilot software. https://ardupilot. org, 2020. [Last accessed 01-06-2024]

    2020

  15. [23]

    MissionPlanner

    Ardupilot. MissionPlanner. https://ardupilot.org/planner, 2022. (Ac- cessed on 07/17/2024)

    2022

  16. [24]

    Arora, T

    C. Arora, T. Herda, and V . Homm. Generating test scenarios from nl requirements using retrieval-augmented llms: An industrial study. arXiv preprint arXiv:2404.12772, 2024

    2024 arXiv

  17. [25]

    Atkins, A

    E. Atkins, A. Khalsa, and M. Groden. Commercial low-altitude uas operations in population centers. In Proc. of the 9th Aviation Tech- nology, Integration, and Operations Conference and Aircraft Noise and Emissions Reduction Symposium , page 7070, 2009

    2009

  18. [26]

    S. Bag, S. K. Kumar, and M. K. Tiwari. An efficient recommendation generation using relevant jaccard similarity. Information Sciences , 483:53–64, 2019

    2019

  19. [27]

    Benarbia and K

    T. Benarbia and K. Kyamakya. A literature review of drone-based package delivery logistics systems and their implementation feasibility. Sustainability, 14(1):360, 2021

    2021

  20. [28]

    Bondi, D

    E. Bondi, D. Dey, A. Kapoor, J. Piavis, S. Shah, F. Fang, B. Dilkina, R. Hannaford, A. Iyer, L. Joppa, et al. Airsim-w: A simulation environment for wildlife conservation with uavs. In Proc. of the 1st ACM Conf. on Computing and Sustainable Societies , pages 1–12, 2018

    2018

  21. [29]

    Chandrasekaran, Y

    J. Chandrasekaran, Y . Lei, R. Kacker, and D. R. Kuhn. A combinatorial approach to testing deep neural network-based autonomous driving systems. In Proc. of the 2021 Int’l Conf. on Software Testing, Verification and Validation WS, pages 57–66. IEEE, 2021

    2021

  22. [30]

    Chang, S

    C. Chang, S. Wang, J. Zhang, J. Ge, and L. Li. Llmscenario: Large language model driven scenario generation. IEEE Transactions on Systems, Man, and Cybernetics: Systems , 2024

    2024

  23. [31]

    B. Chen, Z. Zhang, N. Langren ´e, and S. Zhu. Unleashing the potential of prompt engineering in large language models: a comprehensive review. arXiv preprint arXiv:2310.14735 , 2023

    2023 arXiv

  24. [32]

    Cleland-Huang, A

    J. Cleland-Huang, A. Agrawal, M. N. A. Islam, E. Tsai, M. Van Spey- broeck, and M. Vierhauser. Requirements-driven configuration of emergency response missions with small aerial vehicles. In Proc. of the 24th ACM Conf. on Systems and Software Product Line: Volume A-Volume A, p...

    2020

  25. [33]

    X. Dai, C. Ke, Q. Quan, and K.-Y . Cai. Rflysim: Automatic test platform for uav autopilot systems with fpga-based hardware-in-the-loop simulations. Aerospace Science and Technology , 114:106727, 2021

    2021

  26. [34]

    Dileep, A

    M. Dileep, A. Navaneeth, S. Ullagaddi, and A. Danti. A study and analysis on various types of agricultural drones and its applications. In Proc. of the 5th Int’l Conf. on Research in Computational Intelligence and Communication Networks , pages 181–185. IEEE, 2020

    2020

  27. [35]

    Erdelj and E

    M. Erdelj and E. Natalizio. Uav-assisted disaster management: Appli- cations and open issues. In Proc. of the 2016 Int’l Conf. on Computing, Networking and Communications , pages 1–5. IEEE, 2016

    2016

  28. [36]

    S. Es, J. James, L. Espinosa-Anke, and S. Schockaert. Ragas: Au- tomated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217, 2023

    2023 arXiv

  29. [37]

    Giovagnola, J

    J. Giovagnola, J. B. M. Meg ´ıas, M. M. Fern ´andez, M. P. Cu ´ellar, and D. P. M. Santos. Airloop: A simulation framework for testing of uav services. IEEE Access, 11:23309–23325, 2023

    2023

  30. [38]

    Gonz ´alez-Rocha, L

    J. Gonz ´alez-Rocha, L. Bilyeu, S. D. Ross, H. Foroutan, S. J. Jacquemin, A. P. Ault, and D. G. Schmale. Sensing atmospheric flows in aquatic environments using a multirotor small uncrewed aircraft system (suas). Environmental Science: Atmospheres, 3(2):305–315, 2023

    2023

  31. [39]

    J. Gu, Z. Han, S. Chen, A. Beirami, B. He, G. Zhang, R. Liao, Y . Qin, V . Tresp, and P. Torr. A systematic survey of prompt engineering on vision-language foundation models. arXiv preprint arXiv:2307.12980 , 2023

    2023 arXiv

  32. [40]

    S. Guan, H. Sirianni, G. Wang, and Z. Zhu. suas monitoring of coastal environments: A review of best practices from field to lab. Drones, 6(6):142, 2022

    2022

  33. [41]

    W. Hua, X. Yang, M. Jin, Z. Li, W. Cheng, R. Tang, and Y . Zhang. Trustagent: Towards safe and trustworthy llm-based agents through agent constitution. In Trustworthy Multi-modal Foundation Models and AI Agents (TiFA), 2024

    2024

  34. [42]

    M. N. A. Islam, J. Cleland-Huang, and M. Vierhauser. Adam: Adaptive monitoring of runtime anomalies in small uncrewed aerial systems. In Proc. of the 19th Int’l Symp. on Software Engineering for Adaptive and Self-Managing Systems, pages 44–55, 2024

    2024

  35. [43]

    Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y . Xu, E. Ishii, Y . J. Bang, A. Madotto, and P. Fung. Survey of hallucination in natural language generation. ACM Computing Surveys , 55(12):1–38, 2023

    2023

  36. [44]

    Z. A. Jibon, M. A. Adnan, N. T. Nora, M. M. H. Akash, and F. Ahammed. Development of an autonomous uav for seed and fertilizer distribution in precision agriculture. In Proc. of the 14th Int’l Conf. on Computing Communication and Networking Technologies , pages 1–5. IEEE, 2023

    2023

  37. [45]

    Johnson and S

    E. Johnson and S. Mishra. Flight simulation for the development of an experimental uav. In Proc. of the AIAA Modeling and Simulation Technologies Conference and Exhibit , page 4975, 2002

    2002

  38. [46]

    Johnson, M

    J. Johnson, M. Douze, and H. J ´egou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data , 7(3):535–547, 2019

    2019

  39. [47]

    Lachapelle, J

    G. Lachapelle, J. Henriksen, and T. Melgara. Seasonal effect of tree foliage on gps signal availability and accuracy for vehicular navigation. In Proceedings of the 7th International Technical Meeting of the Satellite Division of The Institute of Navigation (ION GPS 1994), pag...

    1994

  40. [48]

    C. Lee, R. Roy, M. Xu, J. Raiman, M. Shoeybi, B. Catanzaro, and W. Ping. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428 , 2024

    2024 arXiv

  41. [49]

    Lewis, E

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschel, et al. Retrieval- augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems , 33:9459–9474, 2020

    2020

  42. [50]

    Locascio, M

    D. Locascio, M. Levy, K. Ravikumar, B. German, S. I. Briceno, and D. N. Mavris. Evaluation of concepts of operations for suas package delivery. In Proc. of the 16th AIAA Aviation Technology, Integration, and Operations Conf. , page 4371, 2016

    2016

  43. [51]

    Y . Ma, M. N. Al Islam, J. Cleland-Huang, and N. V . Chawla. Detecting anomalies in small unmanned aerial systems via graphical normalizing flows. IEEE Intelligent Systems , 38(2):46–54, 2023

    2023

  44. [52]

    Malik, M

    S. Malik, M. A. Khan, Aadam, H. El-Sayed, F. Iqbal, J. Khan, and O. Ullah. Carla+: An evolution of the carla simulator for complex environment using a probabilistic graphical model. Drones, 7(2):111, 2023

    2023

  45. [53]

    V . C. Martinez, B. Ince, P. K. Selvam, I. Petrunin, M. Seo, E. Anastas- sacos, P. G. Royall, A. Cole, A. Tsourdos, and S. Knorr. Detect and avoid considerations for safe suas operations in urban environments. pages 1–10, 2021

    2021

  46. [54]

    Marvin, N

    G. Marvin, N. Hellen, D. Jjingo, and J. Nakatumba-Nabende. Prompt engineering in large language models. In Proc. of the Int’l Conf. on Data Intelligence and Cognitive Informatics , pages 387–402. Springer, 2023

    2023

  47. [55]

    M. L. McHugh. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276–282, 2012

    2012

  48. [56]

    Messmer, B

    M. Messmer, B. Kiefer, L. A. Varga, and A. Zell. Uav-assisted maritime search and rescue: A holistic approach. arXiv preprint arXiv:2403.14281, 2024

    2024 arXiv

  49. [57]

    Meta. Llama 3. https://llama.meta.com/llama3/, 2024. [Online: accessed August-01-2024]

    2024

  50. [58]

    C. Olea, H. Tucker, J. Phelan, C. Pattison, S. Zhang, M. Lieb, and J. White. Evaluating persona prompting for question answering tasks. In Proc. of the 10th Int’l Conf. on Artificial Intelligence and Soft Computing, 2024

    2024

  51. [59]

    Open Robotics. Gazebo. https://gazebosim.org, 2023. (Accessed 01-14- 2023)

    2023

  52. [60]

    Papineni, S

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages 311–318, 2002

    2002

  53. [61]

    Open Source Flight Controller

    PX4. Open Source Flight Controller. https://px4.io, 2021. (Acessed 01-06-2024)

    2021

  54. [62]

    PX4 Flight Review

    PX4 Autopilot. PX4 Flight Review. https://review.px4.io/, 2024. [Online: accessed August-01-2024]

    2024

  55. [63]

    QGroundControl

    QGroundControl – Drone Control. QGroundControl. http:// qgroundcontrol.com, 2022. (Accessed on 07/17/2024)

    2022

  56. [64]

    Reimers and I

    N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proc. of the 2019 Conf. on Empirical Meth- ods in Natural Language Processing . Association for Computational Linguistics, 11 2019

    2019

  57. [65]

    Scherer, S

    J. Scherer, S. Yahyanejad, S. Hayat, E. Yanmaz, T. Andre, A. Khan, V . Vukadinovic, C. Bettstetter, H. Hellwagner, and B. Rinner. An autonomous multi-uav system for search and rescue. In Proc. of the first workshop on micro aerial vehicle networks, systems, and applications fo...

    2015

  58. [66]

    S. Shah, D. Dey, C. Lovett, and A. Kapoor. Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and Service Robotics: Results of the 11th International Conference , pages 621–635. Springer, 2018

    2018

  59. [67]

    L. Shi, B. Qi, J. Luo, Y . Zhang, Z. Liang, Z. Gao, W. Deng, and L. Sun. Aegis: An advanced llm-based multi-agent for intelligent functional safety engineering. arXiv preprint arXiv:2410.12475 , 2024

    2024 arXiv

  60. [68]

    Strobelt, A

    H. Strobelt, A. Webson, V . Sanh, B. Hoover, J. Beyer, H. Pfister, and A. M. Rush. Interactive and visual prompt engineering for ad-hoc task adaptation with large language models. IEEE transactions on visualization and computer graphics , 29(1):1146–1156, 2022

    2022

  61. [69]

    D. R. Thomas. A general inductive approach for qualitative data analysis. 2003

    2003

  62. [70]

    Vierhauser, M

    M. Vierhauser, M. N. A. Islam, A. Agrawal, J. Cleland-Huang, and J. Mason. Hazard analysis for human-on-the-loop interactions in suas systems. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software ...

    2021

  63. [71]

    J. Wang, J. Chen, Y . Sun, X. Ma, D. Wang, J. Sun, and P. Cheng. Robot: Robustness-oriented testing for deep learning systems. In Proc. of the Int’l Conf. on Software Engineering , pages 300–311. IEEE, 2021

    2021

  64. [72]

    J. Wang, Z. Liu, L. Zhao, Z. Wu, C. Ma, S. Yu, H. Dai, Q. Yang, Y . Liu, S. Zhang, et al. Review of large vision models and visual prompt engineering. Meta-Radiology, page 100047, 2023

    2023

  65. [73]

    White, Q

    J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. El- nashar, J. Spencer-Smith, and D. C. Schmidt. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382, 2023

    2023 arXiv

  66. [74]

    Yadav, B

    M. Yadav, B. Vashisht, N. Vullaganti, S. Jalota, S. Yadav, G. Singh, A. Kumar, and S. Kothiyal. Iot-enabled unmanned aerial vehicle: An emerging trend in precision farming. Artificial Intelligence and Smart Agriculture: Technology and Applications , pages 271–292, 2024

    2024

  67. [75]

    Zhang and A

    B. Zhang and A. Agrawal. Dronewis: Automated simulation testing of small unmanned aerial system in realistic windy conditions. In Pro- ceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE ’24, page 2358–2361, New York, NY , USA,

  68. [76]

    Zhang, Y

    B. Zhang, Y . Shivalingaiah, and A. Agrawal. Dronereqvalidator: Fa- cilitating high fidelity simulation testing for uncrewed aerial systems developers. In Proc. of the 38th IEEE/ACM Int’l Conf. on Automated Software Engineering, pages 2082–2085. IEEE, 2023

    2023

  69. [77]

    Zhang, C

    J. Zhang, C. Xu, and B. Li. Chatscene: Knowledge-enabled safety- critical scenario generation for autonomous vehicles. In Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition , pages 15459–15469, 2024

    2024

  70. [78]

    Y . Zhao, P. Singh, H. Bhathena, B. Ramos, A. Joshi, S. Gadiyaram, and S. Sharma. Optimizing llm based retrieval augmented generation pipelines in the financial domain. In Proc. of the 2024 Conf. of the North American Chapter of the Association for Computational Linguistics: H...

    2024

  71. [79]

    Y . Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910, 2022

    2022 arXiv

  72. [2023]

    (Accessed 01-07-2024)

    2024

  73. [2024]

    Association for Computing Machinery

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.