pith. sign in

arxiv: 2606.23760 · v2 · pith:JYXD5KNMnew · submitted 2026-06-22 · 💻 cs.RO · cs.AI· cs.MA· cs.SE

Engineering Reliable Autonomous Systems: Challenges and Solutions

Marie Farrell , Matt Luckcuck , Angelo Ferrando , Rafael C. Cardoso , Natasha Alechina , Marco Autili , Diana Benjumea Hernandez , Luciana Brasil Rebelo dos Santos
show 23 more authors
Daniela Briola Ana Cavalcanti Christian Colombo Louise A. Dennis Clare Dixon Michael Fisher Mario Gleirscher Taylor Johnson Charles Lesire Livia Lestingi Sven Linker Brian Logan Colin Paterson Fabio Papacchini Patrizio Pelliccione Pedro Ribeiro Maike Schwammberger Silvia Lizeth Tapia Tarifa Hazel Taylor Jim Woodcock Mengwei Xu Yi Yang Huan Zhang
This is my paper

Pith reviewed 2026-06-26 08:36 UTC · model grok-4.3

classification 💻 cs.RO cs.AIcs.MAcs.SE
keywords autonomous systemsverification and validationsoftware architecturesreliabilityengineering challengesworkshop reportroadmapsafe systems
0
0 comments X

The pith

A 2024 workshop produced a catalogue of challenges in verification, real-world engineering, and architectures for reliable autonomous systems, along with solution pathways.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper reports on the Lorentz Center Workshop held in June 2024 that brought together researchers, industry practitioners, and sector representatives to discuss engineering reliable autonomous systems. It focuses on three topics: verification and validation techniques, engineering real-world autonomous systems, and software architectures for safety. The main result is a catalogue of challenges in these areas plus pathways to solutions, noting that some challenges can use existing academic methods not yet common in practice while others need new research. This matters because autonomous systems are becoming more prevalent and require practical techniques to ensure reliability.

Core claim

The workshop's main outcome is a catalogue of challenges in verification and validation of autonomous systems, engineering real-world autonomous systems, and software architectures for safe autonomous systems, together with a pathway to solutions. Some challenges can already be tackled by techniques that are well known in academia but have not yet become regularly used in practice. Other challenges remain unresolved and require further research. This roadmap is intended to support future research and industrial collaboration.

What carries the argument

The catalogue of challenges across the three workshop topics together with the derived pathway to solutions from the collected discussions.

If this is right

  • Some verification techniques already known in academia can be adopted in practice to address listed challenges without new research.
  • Unresolved challenges in software architectures will require dedicated research efforts guided by the roadmap.
  • Industrial partners can use the catalogue to prioritize collaboration on specific reliability issues.
  • Future research projects can reference the pathways to align efforts on verification, engineering, and architecture topics.
  • The roadmap can help coordinate between academic communities and practitioners in sectors with distinctive autonomous system challenges.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The structure of the catalogue could be reused to organize roadmaps in adjacent areas such as multi-agent systems or human-robot interaction.
  • Case studies applying the suggested pathways in deployed systems would provide concrete tests of their practicality.
  • Periodic updates to the catalogue through follow-on workshops could track which challenges have been resolved over time.
  • The pathways might inform standards development by highlighting where existing techniques are ready for wider use.

Load-bearing premise

The discussions at this single 2024 workshop accurately capture the most important open challenges across the field and that the suggested pathways represent feasible next steps.

What would settle it

A broader survey or additional workshops across more sectors that identify a substantially different set of top challenges or pathways would show the catalogue is incomplete or misdirected.

Figures

Figures reproduced from arXiv: 2606.23760 by Ana Cavalcanti, Angelo Ferrando, Brian Logan, Charles Lesire, Christian Colombo, Clare Dixon, Colin Paterson, Daniela Briola, Diana Benjumea Hernandez, Fabio Papacchini, Hazel Taylor, Huan Zhang, Jim Woodcock, Livia Lestingi, Louise A. Dennis, Luciana Brasil Rebelo dos Santos, Maike Schwammberger, Marco Autili, Marie Farrell, Mario Gleirscher, Matt Luckcuck, Mengwei Xu, Michael Fisher, Natasha Alechina, Patrizio Pelliccione, Pedro Ribeiro, Rafael C. Cardoso, Silvia Lizeth Tapia Tarifa, Sven Linker, Taylor Johnson, Yi Yang.

Figure 1
Figure 1. Figure 1: Radar plot of our subjective assessment of the case studies in Section 2 against our [PITH_FULL_IMAGE:figures/full_fig_p018_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Radar plot of our subjective assessment of the case studies in Section 2 against our [PITH_FULL_IMAGE:figures/full_fig_p019_2.png] view at source ↗
Figure 1.1
Figure 1.1. Figure 1.1: We specify the Assume-Guarantee contracts for each node (denoted by A (i) and G (o) respectively). These are then used to guide the verification approach applied to each node, denoted by dashed lines, such as software testing for a black-box implementation of the Vision node. The solid arrows represent data flow between nodes and that the assumptions of the next node should follow from the guarantee of t… view at source ↗
Figure 4
Figure 4. Figure 4: Most deployed autonomous systems operate under significant environmental, stake [PITH_FULL_IMAGE:figures/full_fig_p021_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Template blank roadmap One area of particular focus in the coming years is the verification of vision-based perception systems. Since many autonomous systems that interact with the real, physical world rely on object detection and recognition (including autonomous vehicles and domestic robotic assistants), it is vital that we can verify the accuracy of perception systems. This will likely require heterogen… view at source ↗
Figure 6
Figure 6. Figure 6: We consolidate the responses into a single roadmap, with colour-coding to distinguish [PITH_FULL_IMAGE:figures/full_fig_p033_6.png] view at source ↗
read the original abstract

Engineering reliable autonomous systems is an important and growing topic in computer science. As autonomous systems become more prevalent, easy-to-use techniques for building them reliably are increasingly important. This workshop report captures and expands on the discussions at the Lorentz Center Workshop "Engineering Reliable Autonomous Systems" (ERAS), held from 10 to 14 June 2024. The workshop was co-organised by the organisers of the Workshop on Formal Methods for Autonomous Systems (FMAS) and the Workshop on Agents and Robots for reliable Engineered Autonomy (AREA). It brought together members of the FMAS and AREA communities, industry practitioners, and representatives from sectors where autonomous systems pose distinctive engineering challenges. The workshop focused on three main research topics: techniques for verification and validation of autonomous systems; engineering real-world autonomous systems; and software architectures for safe autonomous systems. Its main outcome is a catalogue of challenges in these areas and, most importantly, a pathway to solutions. Some challenges can already be tackled by techniques that are well known in academia but have not yet become regularly used in practice. Other challenges remain unresolved and require further research. This roadmap is intended to support future research and industrial collaboration.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 2 minor

Summary. This manuscript is a report on the ERAS 2024 workshop (Lorentz Center, 10–14 June 2024) co-organised by the FMAS and AREA communities. It summarises discussions among academics, industry practitioners, and sector representatives on three topics: verification and validation techniques for autonomous systems, engineering real-world autonomous systems, and software architectures for safe autonomous systems. The central claim is that the workshop produced a catalogue of challenges in these areas together with pathways to solutions, distinguishing challenges addressable by existing but under-adopted academic techniques from those requiring new research; the report positions itself as a roadmap to guide future work and collaboration.

Significance. If the catalogue and pathways accurately reflect the workshop record, the report supplies a structured, cross-community synthesis of open problems and actionable next steps in reliable autonomous systems. Its value lies in bridging formal-methods and agent/robot communities with industrial input and in explicitly separating immediately transferable techniques from longer-term research needs.

minor comments (2)
  1. [Abstract] Abstract: the statement that 'some challenges can already be tackled by techniques that are well known in academia but have not yet become regularly used in practice' is left without concrete examples; adding one or two brief illustrations in the main text would increase the report's utility as a roadmap.
  2. [Main body (organisation of catalogue)] The three research topics are introduced but the subsequent catalogue is not explicitly cross-referenced to them; a short mapping table or subsection headings would improve traceability between topics and listed challenges.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their positive review and recommendation to accept the manuscript. The referee's summary correctly captures the workshop report's scope, outcomes, and intended contribution as a cross-community roadmap.

Circularity Check

0 steps flagged

No significant circularity

full rationale

The manuscript is a workshop report that summarizes and expands on discussions from the ERAS 2024 event. Its central claim is descriptive: the workshop produced a catalogue of challenges in verification/validation, real-world engineering, and safe architectures, plus suggested pathways. No equations, derivations, fitted parameters, theorems, or quantitative models are asserted. The content is a straightforward record of external participant discussions and does not reduce any claim to its own inputs by construction, self-citation chains, or renaming. The report is self-contained against the event record itself.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters, axioms, or invented entities are introduced; the paper is a descriptive workshop summary with no mathematical or modeling content.

pith-pipeline@v0.9.1-grok · 5864 in / 973 out tokens · 17978 ms · 2026-06-26T08:36:59.623588+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

87 extracted references · 61 canonical work pages

  1. [1]

    Autonomous Nuclear Waste Management.IEEE Intelligent Systems, 33(6):47–55, 2018.doi:10

    Jonathan M Aitken, Sandor M Veres, Affan Shaukat, Yang Gao, Elisa Cucco, Louise A Dennis, Michael Fisher, Jeffrey A Kuo, Thomas Robinson, and Paul E Mort. Autonomous Nuclear Waste Management.IEEE Intelligent Systems, 33(6):47–55, 2018.doi:10. 1109/MIS.2018.111144814

    arXiv 2018

  2. [2]

    ROS 2 in a Nutshell: A Survey.ACM Computing Surveys, 2024.doi:10

    Abdulrahman Al-Batati, Anis Koubaa, Khaled Gabr, Mohamed Abdelkader, and Hamad Aloqaily. ROS 2 in a Nutshell: A Survey.ACM Computing Surveys, 2024.doi:10. 1145/3815113

    2024

  3. [3]

    Introduction to Neural Network Verification.Foundations and Trends in Programming Languages, 7(1–2):1–157, 2021.doi:10.1561/2500000051

    Aws Albarghouthi. Introduction to Neural Network Verification.Foundations and Trends in Programming Languages, 7(1–2):1–157, 2021.doi:10.1561/2500000051

    work page doi:10.1561/2500000051 2021

  4. [4]

    Safe Reinforcement Learning via Shielding

    Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe Reinforcement Learning via Shielding. InAAAI Con- ference on Artificial Intelligence, pages 2669–2678. AAAI Press, 2018.doi:10.1609/ aaai.v32i1.11797

    2018

  5. [5]

    Safety Verification of Autonomous Vehicles for Coordinated Evasive Maneuvers

    Matthias Althoff, Daniel Althoff, Dirk Wollherr, and Martin Buss. Safety Verification of Autonomous Vehicles for Coordinated Evasive Maneuvers. InIEEE Intelligent Vehicles Symposium, pages 1078–1083, 2010.doi:10.1109/IVS.2010.5548121

    work page doi:10.1109/ivs.2010.5548121 2010

  6. [6]

    Anderson and Louise A

    Christopher R. Anderson and Louise A. Dennis. Autonomous Systems’ Safety Cases for use in UK Nuclear Environments.arXiv preprint arXiv:2310.02344, 2023.doi:10.48420/ 19512010

    arXiv 2023

  7. [7]

    Autonomous Emergency Triage Support System

    Ol’Tunde Ashaolu, William Lyons, Ioannis Stefanakos, Radu Calinescu, Ibrahim Habli, Victoria Hodge, Chiara Picardi, Katherine Plant, and Beverley Townsend. Autonomous Emergency Triage Support System. InInternational Conference on Computational Science and Computational Intelligence, pages 1332–1337, 2023.doi:10.1109/CSCI62032. 2023.00220

    work page doi:10.1109/csci62032 2023

  8. [8]

    Neuro-Symbolic AI + Agent Systems: A First Reflection on Trends, Opportunities and Challenges

    Vaishak Belle, Michael Fisher, Alessandra Russo, Ekaterina Komendantskaya, and Alistair Nottle. Neuro-Symbolic AI + Agent Systems: A First Reflection on Trends, Opportunities and Challenges. InInternational Conference on Autonomous Agents and Multiagent Systems, pages 180–200. Springer, 2023.doi:10.1007/978-3-031-56255-6_10

    work page doi:10.1007/978-3-031-56255-6_10 2023

  9. [9]

    Benjumea

    Diana C. Benjumea. Formalising Safety Requirements for Robotic Autonomous Systems in Highly Regulated Domains. InInternational Requirements Engineering Conference, pages 512–516. IEEE, 2024.doi:10.1109/RE59067.2024.00066

    work page doi:10.1109/re59067.2024.00066 2024

  10. [10]

    Autonomous Auto- mobilities: The Social Impacts of Driverless Vehicles.Current Sociology, 68(1):116–134, 2020.doi:10.1177/001139211881674

    David Bissell, Thomas Birtchnell, Anthony Elliott, and Eric L Hsu. Autonomous Auto- mobilities: The Social Impacts of Driverless Vehicles.Current Sociology, 68(1):116–134, 2020.doi:10.1177/001139211881674

    work page doi:10.1177/001139211881674 2020

  11. [11]

    Assume-Guarantee Testing.ACM SIGSOFT Software Engineering Notes, 31(2), 2005.doi:10.1145/ 1118537.1123060

    Colin Blundell, Dimitra Giannakopoulou, and Corina S Pˇasˇareanu. Assume-Guarantee Testing.ACM SIGSOFT Software Engineering Notes, 31(2), 2005.doi:10.1145/ 1118537.1123060

    arXiv 2005

  12. [12]

    Robots in the Nuclear Industry: A Review of Technologies and Ap- plications.Industrial Robot: An International Journal, 38(2):113–118, 2011.doi: 10.1108/01439911111106327

    Robert Bogue. Robots in the Nuclear Industry: A Review of Technologies and Ap- plications.Industrial Robot: An International Journal, 38(2):113–118, 2011.doi: 10.1108/01439911111106327

    work page doi:10.1108/01439911111106327 2011

  13. [13]

    Andrea Bombarda, Silvia Bonfanti, Martina De Sanctis, Angelo Gargantini, Patrizio Pel- liccione, Elvinia Riccobene, and Patrizia Scandurra. Evaluation Framework for Autonom- ous Systems: The Case of Programmable Electronic Medical Systems.IEEE Transactions on Software Engineering, 50(4):995–1014, 2024.doi:10.1109/TSE.2024.3374382. 36

    work page doi:10.1109/tse.2024.3374382 2024

  14. [14]

    Dennis, Michael Fisher, and Alan F

    Paul Bremner, Louise A. Dennis, Michael Fisher, and Alan F. Winfield. On Proactive, Transparent, and Verifiable Ethical Reasoning for Robots.Proceedings of the IEEE, 107(3):541–561, 2019.doi:10.1109/JPROC.2019.2898267

    work page doi:10.1109/jproc.2019.2898267 2019

  15. [15]

    BS 8611 Guide to the Ethical Design and Application of Robots and Robotic Devices.http://www.bsigroup.com, 2016

    British Standards Institution (BSI). BS 8611 Guide to the Ethical Design and Application of Robots and Robotic Devices.http://www.bsigroup.com, 2016

    2016

  16. [16]

    Johnson, and Changliu Liu

    Christopher Brix, Mark Niklas Müller, Stanley Bak, Taylor T. Johnson, and Changliu Liu. First Three Years of the International Verification of Neural Networks Competition (VNN- COMP).International Journal on Software Tools for Technology Transfer, 25(3):329–339, 2023.doi:10.1007/s10009-023-00703-4

    work page doi:10.1007/s10009-023-00703-4 2023

  17. [17]

    Simon Burton, Ibrahim Habli, Tom Lawton, John McDermid, Phillip Morgan, and Zoe Porter. Mind the Gaps: Assuring the Safety of Autonomous Systems from an Engineering, Ethical, and Legal Perspective.Artificial Intelligence, 279:103201, 2020.doi:10.1016/ j.artint.2019.103201

    arXiv 2020

  18. [18]

    Cardoso, Louise A

    Rafael C. Cardoso, Louise A. Dennis, Marie Farrell, Michael Fisher, and Matt Luck- cuck. Towards Compositional Verification for Modular Robotic Systems. InInternational Workshop on Formal Methods for Autonomous Systems, volume 329 ofElectronic Pro- ceedings in Theoretical Computer Science, pages 15–22. Open Publishing Association, 2020.doi:10.4204/EPTCS.329.2

    work page doi:10.4204/eptcs.329.2 2020

  19. [19]

    Cardoso, Marie Farrell, Matt Luckcuck, Angelo Ferrando, and Michael Fisher

    Rafael C. Cardoso, Marie Farrell, Matt Luckcuck, Angelo Ferrando, and Michael Fisher. Heterogeneous Verification of an Autonomous Curiosity Rover. In Ritchie Lee, Susmit Jha, Anastasia Mavridou, and Dimitra Giannakopoulou, editors,NASA Formal Methods Symposium, volume 12229 ofLNCS, pages 353–360, Cham, 2020. Springer.doi:10. 1007/978-3-030-55754-6_20

    2020

  20. [20]

    Verified Simulation for Robotics.Science of Computer Programming, 174:1–37, 2019

    Ana Cavalcanti, Augusto Sampaio, Alvaro Miyazawa, Pedro Ribeiro, Madiel Conserva Filho, André Didier, Wei Li, and Jon Timmis. Verified Simulation for Robotics.Science of Computer Programming, 174:1–37, 2019. URL:papers/CSMRCD19.pdf,doi:doi. org/10.1016/j.scico.2019.01.004

    work page doi:10.1016/j.scico.2019.01.004 2019

  21. [21]

    Assume-Guarantee Reasoning for Safe Component Behaviours

    Chris Chilton, Bengt Jonsson, and Marta Kwiatkowska. Assume-Guarantee Reasoning for Safe Component Behaviours. InFormal Aspects of Component Software, volume 7684 of LNCS, pages 92–109. Springer, 2013.doi:10.1007/978-3-642-35861-6_6

    work page doi:10.1007/978-3-642-35861-6_6 2013

  22. [22]

    Cobleigh, Dimitra Giannakopoulou, and Corina S

    Jamieson M. Cobleigh, Dimitra Giannakopoulou, and Corina S. Pasareanu. Learning Assumptions for Compositional Verification. InTools and Algorithms for the Construction and Analysis of Systems, volume 2619 ofLNCS, pages 331–346. Springer, 2003.doi: 10.1007/3-540-36577-X_24

    work page doi:10.1007/3-540-36577-x_24 2003

  23. [23]

    Compositional Verification of Architectural Models

    Darren Cofer, Andrew Gacek, Steven Miller, Michael W Whalen, Brian LaValley, and Lui Sha. Compositional Verification of Architectural Models. InNASA Formal Methods Symposium, volume 7226 ofLNCS, pages 126–140. Springer, 2012.doi:10.1007/ 978-3-642-28891-3_13

    2012

  24. [24]

    Small Size 4WD Mobile Robot.https://global.agilex

    Agilex Robotics Company. Small Size 4WD Mobile Robot.https://global.agilex. ai/products/scout-mini, 2024. Accessed: 2024-06-12

    2024

  25. [25]

    DoModernSystemsRequire New Quality Dimensions? InQuality of Information and Communications Technology, pages 83–90

    MartinaDeSanctis, PaolaInverardi, andPatrizioPelliccione. DoModernSystemsRequire New Quality Dimensions? InQuality of Information and Communications Technology, pages 83–90. Springer, 2024.doi:10.1007/978-3-031-70245-7_6

    work page doi:10.1007/978-3-031-70245-7_6 2024

  26. [26]

    Dennis, Michael Fisher, Jonathan M

    Louise A. Dennis, Michael Fisher, Jonathan M. Aitken, Sandor M. Veres, Yang Gao, Affan Shaukat, and Guy Burroughes. Reconfigurable Autonomy.KI — Künstliche Intelligenz, pages 1–9, 2014.doi:10.1007/s13218-014-0308-1. 37

    work page doi:10.1007/s13218-014-0308-1 2014

  27. [27]

    Dennis, Michael Fisher, Marija Slavkovik, and Matthew P

    Louise A. Dennis, Michael Fisher, Marija Slavkovik, and Matthew P. Webster. Formal Verification of Ethical Choices in Autonomous Systems.Robotics and Autonomous Sys- tems, 77:1–14, 2016.doi:10.1016/j.robot.2015.11.012

    work page doi:10.1016/j.robot.2015.11.012 2016

  28. [28]

    The Fridge Door is Open

    Clare Dixon, Matt Webster, Joe Saunders, Michael Fisher, and Kerstin Dautenhahn. “The Fridge Door is Open”-Temporal Verification of a Robotic Assistant’s Behaviours. InAdvances in Autonomous Robotics Systems, volume 8717 ofLNCS, pages 97–108. Springer, 2014.doi:10.1007/978-3-319-10401-0_9

    work page doi:10.1007/978-3-319-10401-0_9 2014

  29. [29]

    Auto- mated Circular Assume-Guarantee Reasoning.Formal Aspects of Computing, 30(5):571– 595, 2018.doi:10.1007/s00165-017-0436-0

    Karam Abd Elkader, Orna Grumberg, Corina S Păsăreanu, and Sharon Shoham. Auto- mated Circular Assume-Guarantee Reasoning.Formal Aspects of Computing, 30(5):571– 595, 2018.doi:10.1007/s00165-017-0436-0

    work page doi:10.1007/s00165-017-0436-0 2018

  30. [30]

    Cardoso, Louise A

    Marie Farrell, Rafael C. Cardoso, Louise A. Dennis, Clare Dixon, Michael Fisher, Geor- gios Kourtis, Alexei Lisitsa, Matt Luckcuck, and Matt Webster. Modular Verification of Autonomous Space Robotics. InAssurance of Autonomy for Robotic Space Missions Workshop, 2019.doi:10.48550/arXiv.1908.10738

    work page doi:10.48550/arxiv.1908.10738 2019

  31. [31]

    Robotics and Integrated Formal Meth- ods: NecessityMeetsOpportunity

    Marie Farrell, Matt Luckcuck, and Michael Fisher. Robotics and Integrated Formal Meth- ods: NecessityMeetsOpportunity. InIntegratedFormalMethods, volume11023ofLNCS, pages 161–171. Springer, Springer, 2018.doi:10.1007/978-3-319-98938-9_10

    work page doi:10.1007/978-3-319-98938-9_10 2018

  32. [32]

    Formal Modelling and Runtime Verification of Autonomous Grasping for Active Debris Removal

    Marie Farrell, Nikos Mavrakis, Angelo Ferrando, Clare Dixon, and Yang Gao. Formal Modelling and Runtime Verification of Autonomous Grasping for Active Debris Removal. Frontiers in Robotics and AI, 8, 2021.doi:10.3389/frobt.2021.639282

    work page doi:10.3389/frobt.2021.639282 2021

  33. [33]

    Cardoso, Craig Schlenoff, and Michael Fisher

    Angelo Ferrando, Zeid Kootbally, Pavel Piliptchak, Rafael C. Cardoso, Craig Schlenoff, and Michael Fisher. Runtime Verification of the ARIAC Competition: Can a Robot be Agile and Safe at the same time? InItalian Workshop on Artificial Intelligence and Robotics, CEUR Workshop Proceedings, pages 7–11. CEUR-WS.org, 2020. URL: https://ceur-ws.org/Vol-2806/short2.pdf

    2020

  34. [34]

    Conceptual Analysis and Architecture Re- quirements Specification: Towards the CoreSense Architectural Concept

    Alex Gabriel, Forough Zamani, Carlos Hernández, Carlos García, Esther Aguado, and Ricardo Sanz. Conceptual Analysis and Architecture Re- quirements Specification: Towards the CoreSense Architectural Concept. https://coresense.eu/wp-content/uploads/2023/11/CORESENSE_D2. 1-Conceptual-Analysis-and-Architecture-Requirements-Specification.pdf,

    2023

  35. [35]

    Accessed: 2024-06-12

    2024

  36. [36]

    CRutoN: Automatic Verification of a Robotic Assistant’s Beha- viours

    Paul Gainer, Clare Dixon, Kerstin Dautenhahn, Michael Fisher, Ullrich Hustadt, Joe Saun- ders, and Matt Webster. CRutoN: Automatic Verification of a Robotic Assistant’s Beha- viours. InInternational Workshop on Automated Verification of Critical Systems, volume 10471 ofLNCS, pages 119–133. Springer, 2017.doi:10.1007/978-3-319-67113-0\ _8

    work page doi:10.1007/978-3-319-67113-0 2017

  37. [37]

    Deep Learning with Logical Constraints

    Eleonora Giunchiglia, Mihaela Catalina Stoian, and Thomas Lukasiewicz. Deep Learning with Logical Constraints. InInternational Joint Conference on Artificial Intelligence, pages 5478–5485. ijcai.org, 2022.doi:10.24963/ijcai.2022/767

    work page doi:10.24963/ijcai.2022/767 2022

  38. [38]

    Multi-Robot Task Planning to Secure Human Group Progress

    Roland Godet, Charles Lesire, and Arthur Bit-Monnot. Multi-Robot Task Planning to Secure Human Group Progress. InInternational Workshop on Agents and Robots for reliable Engineered Autonomy, EPTCS, pages 113–126, 2023.doi:10.4204/EPTCS. 391.13

    work page doi:10.4204/eptcs 2023

  39. [39]

    Urbano, Patrick Grycz, Katharina Bause, Minakshi Kaushik, Vincenzo Scotti, Akhila Bairy, Maike Schwammberger, Maribel Acosta, Albert Albers, 38 Anne Koziolek, and Tobias Düser

    Nathan Hagel, Johannes Mäkelburg, Claus Hammann, Thomas Weber, Thomas Alex- ander Völk, Francesco P. Urbano, Patrick Grycz, Katharina Bause, Minakshi Kaushik, Vincenzo Scotti, Akhila Bairy, Maike Schwammberger, Maribel Acosta, Albert Albers, 38 Anne Koziolek, and Tobias Düser. Explainability in Automated Cross-Domain Model- Driven Brake System Development...

    work page doi:10.1109/asew67777.2025.00047 2025

  40. [40]

    The Agile Robotics for Indus- trial Automation Competition.AI Magazine, 39(4):77–79, 2018.doi:10.1609/aimag

    William Harrison, Anthony Downs, and Craig Schlenoff. The Agile Robotics for Indus- trial Automation Competition.AI Magazine, 39(4):77–79, 2018.doi:10.1609/aimag. v39i4.2781

    work page doi:10.1609/aimag 2018

  41. [41]

    GuidanceontheAssuranceofMachineLearninginAutonomousSystems(AMLAS)

    Richard Hawkins, Colin Paterson, Chiara Picardi, Yan Jia, Radu Calinescu, and Ibrahim Habli. GuidanceontheAssuranceofMachineLearninginAutonomousSystems(AMLAS). arXiv preprint arXiv:2102.01564, 2021.doi:10.48550/arXiv.2102.01564

    work page doi:10.48550/arxiv.2102.01564 2021

  42. [42]

    PhD thesis, Arizona State University, 2021

    Mohammad Hekmatnejad.Formalizing Safety, Perception, and Mission Requirements for Testing and Planning in Autonomous Vehicles. PhD thesis, Arizona State University, 2021

    2021

  43. [43]

    Deshmukh, Yezhou Yang, and Georgios Fainekos

    Mohammad Hekmatnejad, Bardh Hoxha, Jyotirmoy V. Deshmukh, Yezhou Yang, and Georgios Fainekos. Formalizing and Evaluating Requirements of Perception Systems for Automated Vehicles using Spatio-Temporal Perception Logic.International Journal of Robotics Research, 43(2):203–238, 2024.doi:10.1177/02783649231223546

    work page doi:10.1177/02783649231223546 2024

  44. [44]

    Assessing Public Perception of Self-Driving Xars: The autonomous Vehicle Acceptance Model

    Charlie Hewitt, Ioannis Politis, Theocharis Amanatidis, and Advait Sarkar. Assessing Public Perception of Self-Driving Xars: The autonomous Vehicle Acceptance Model. InInternational Conference on Intelligent User Interfaces, pages 518–527, 2019.doi: 10.1145/3301275.3302268

    work page doi:10.1145/3301275.3302268 2019

  45. [45]

    Ethics guidelines for trustworthy AI, 2019, last update 31 January 2024

    High-Level Expert Group on AI. Ethics guidelines for trustworthy AI, 2019, last update 31 January 2024. URL:https://digital-strategy.ec.europa.eu/en/library/ ethics-guidelines-trustworthy-ai

    2019

  46. [46]

    Standard, IEEE SA, 2024

    IEEE 7009-2024 Standard for Fail-Safe Design of Autonomous and Semi-Autonomous Systems. Standard, IEEE SA, 2024

    2024

  47. [47]

    Standard, IEEE SA, Forth- coming

    IEEE P7009.1 Standard for Safety Management of Autonomous and Semi-Autonomous Systems–Interventions in the Event of Anomalous Behavior. Standard, IEEE SA, Forth- coming

  48. [48]

    Standard, International Organization for Standardization, 2018

    ISO 26262-1:2018 Road vehicles — Functional safety Part 1: Vocabulary. Standard, International Organization for Standardization, 2018

    2018

  49. [49]

    ISO/IEC 25010 - System and Software Quality Models.https://iso25000

    ISO25000. ISO/IEC 25010 - System and Software Quality Models.https://iso25000. com/index.php/en/iso-25000-standards/iso-25010, 2023

    2023

  50. [50]

    The Vision of Autonomic Computing.Computer, 36(1):41–50, 2003.doi:10.1109/MC.2003.1160055

    Jeffrey O Kephart and David M Chess. The Vision of Autonomic Computing.Computer, 36(1):41–50, 2003.doi:10.1109/MC.2003.1160055

    work page doi:10.1109/mc.2003.1160055 2003

  51. [51]

    Kheng Lee Koay, Matt Webster, Clare Dixon, Paul Gainer, Dag Sverre Syrdal, Michael Fisher, and Kerstin Dautenhahn. Use and Usability of Software Verification Methods to Detect Behaviour Interference When Teaching an Assistive Home Companion Robot: A Proof-of-Concept Study.Paladyn, Journal of Behavioral Robotics, 12(1):402–422, 2021. doi:10.1515/PJBR-2021-0028

    work page doi:10.1515/pjbr-2021-0028 2021

  52. [52]

    Queipo, Raphael T

    Arda Kurt and Ümit Özgüner. Hierarchical Finite State Machines for Autonomous Mo- bile Systems.Control Engineering Practice, 21(2):184–194, 2013.doi:10.1016/j. conengprac.2012.09.020

    work page doi:10.1016/j 2013

  53. [53]

    Algorithms for Verifying Deep Neural Networks.Foundations and Trends in Optimization, 4(3-4):244–404, 2021.doi:10.1561/2400000035

    ChangliuLiu, TomerArnon, ChrisLazarus, ChristopherStrong, ClarkBarrett, andMykelJ Kochenderfer. Algorithms for Verifying Deep Neural Networks.Foundations and Trends in Optimization, 4(3-4):244–404, 2021.doi:10.1561/2400000035. 39

    work page doi:10.1561/2400000035 2021

  54. [54]

    ARCH-COMP23 Category Report: Artificial Intelligence and Neural Network Control Systems (AINNCS) for Continuous and Hybrid Systems Plants

    Diego Manzanas Lopez, Matthias Althoff, Marcelo Forets, Taylor T Johnson, Tobias Ladner, and Christian Schilling. ARCH-COMP23 Category Report: Artificial Intelligence and Neural Network Control Systems (AINNCS) for Continuous and Hybrid Systems Plants. InInternational Workshop on Applied Verification of Continuous and Hybrid Systems, volume 96 ofEPiC Seri...

    work page doi:10.29007/x38n 2023

  55. [55]

    Pearson, 2023

    Qinghua Lu, Liming Zhu, Jon Whittle, and Xiwei Xu.Responsible AI: Best Practices for Creating Trustworthy AI Systems. Pearson, 2023. URL:https://books.google.it/ books?id=yFelzwEACAAJ

    2023

  56. [56]

    Surveying Neuro- Symbolic Approaches for Reliable Artificial Intelligence of Things.Journal of Reliable Intelligent Environments, 10(3):257–279, 2024.doi:10.1007/s40860-024-00231-1

    Zhen Lu, Imran Afridi, Hong Jin Kang, Ivan Ruchkin, and Xi Zheng. Surveying Neuro- Symbolic Approaches for Reliable Artificial Intelligence of Things.Journal of Reliable Intelligent Environments, 10(3):257–279, 2024.doi:10.1007/s40860-024-00231-1

    work page doi:10.1007/s40860-024-00231-1 2024

  57. [57]

    Dennis, Clare Dixon, and Michael Fisher

    Matt Luckcuck, Marie Farrell, Louise A Dennis, Clare Dixon, and Michael Fisher. Formal Specification and Verification of Autonomous Robotic Systems: A Survey.ACM Com- puting Surveys, 52(5):1–41, 2019.doi:10.1145/3342355

    work page doi:10.1145/3342355 2019

  58. [58]

    Principles for the Development and Assurance of Autonomous Systems for Safe Use in Hazardous Environments, June 2021.doi:10.5281/zenodo.5012322

    MattLuckcuck, MichaelFisher, LouiseDennis, SteveFrost, AndyWhite, andDougStyles. Principles for the Development and Assurance of Autonomous Systems for Safe Use in Hazardous Environments, June 2021.doi:10.5281/zenodo.5012322

    work page doi:10.5281/zenodo.5012322 2021

  59. [59]

    Artificial Intelligence Act, 2024

    Madiega Tambiama André. Artificial Intelligence Act, 2024. URL:https: //www.europarl.europa.eu/RegData/etudes/BRIE/2021/698792/EPRS_ BRI(2021)698792_EN.pdf

    2024

  60. [60]

    A Classification Framework of Uncertainty in Architecture-Based Self-Adaptive Systems with Multiple Quality Re- quirements

    Sara Mahdavi-Hezavehi, Paris Avgeriou, and Danny Weyns. A Classification Framework of Uncertainty in Architecture-Based Self-Adaptive Systems with Multiple Quality Re- quirements. InManaging Trade-Offs in Adaptable Software Architectures, pages 45–77. Elsevier, 2017.doi:10.1016/B978-0-12-802855-1.00003-4

    work page doi:10.1016/b978-0-12-802855-1.00003-4 2017

  61. [61]

    RoboChart: Modelling and Verification of the Functional Behaviour of Ro- botic Applications.Software and Systems Modeling., 18(5):3097–3149, 2019.doi: 10.1007/s10270-018-00710-z

    Alvaro Miyazawa, Pedro Ribeiro, Wei Li, Ana Cavalcanti, Jon Timmis, and Jim Wood- cock. RoboChart: Modelling and Verification of the Functional Behaviour of Ro- botic Applications.Software and Systems Modeling., 18(5):3097–3149, 2019.doi: 10.1007/s10270-018-00710-z

    work page doi:10.1007/s10270-018-00710-z 2019

  62. [62]

    Bradley, Marilyn Wolf, and Taylor T

    Krishna Muvva, Justin M. Bradley, Marilyn Wolf, and Taylor T. Johnson. Assuring Learning-Enabled Components in Small Unmanned Aircraft Systems. InAIAA Scitech 2021 Forum. AIAA, 2021.doi:10.2514/6.2021-0994

    work page doi:10.2514/6.2021-0994 2021

  63. [63]

    Safety Assessment Principles for Nuclear Facilities, 2014 Edition Revision 0

    ONR. Safety Assessment Principles for Nuclear Facilities, 2014 Edition Revision 0. 2014. URL:http://www.onr.org.uk/saps/index.htm

    2014

  64. [64]

    From Automation to Autonomous Systems: A Legal Phenomenology with Problems of Accountability

    Ugo Pagallo. From Automation to Autonomous Systems: A Legal Phenomenology with Problems of Accountability. InInternational Joint Conference on artificial intelligence, pages 17–23, 2017

    2017

  65. [65]

    Simone Pettigrew, Lin Fritschi, and Richard Norman. The Potential Implications of Autonomous Vehicles In and Around the Workplace.International Journal of Environ- mental Research and Public Health, 15(9):1876, 2018.doi:10.3390/ijerph15091876

    work page doi:10.3390/ijerph15091876 2018

  66. [66]

    Leveraging Compositional Methods for Modeling and Verification of an Autonomous Taxi System, 2023.doi: 10.1109/ICAA58325.2023.00013

    Alessandro Pinto, Anthony Corso, and Edward Schmerling. Leveraging Compositional Methods for Modeling and Verification of an Autonomous Taxi System, 2023.doi: 10.1109/ICAA58325.2023.00013

    work page doi:10.1109/icaa58325.2023.00013 2023

  67. [67]

    Rao and Michael P

    Anand S. Rao and Michael P. Georgeff. BDI Agents: From Theory to Practice. In International Conference on Multi-Agent Systems, pages 312–319. AAAI, 1995. 40

    1995

  68. [68]

    About the Importance of Autonomy and Digital Twins for the Future of Manufacturing.IFAC- PapersOnLine, 48(3):567–572, 2015.doi:10.1016/j.ifacol.2015.06.141

    Roland Rosen, Georg Von Wichert, George Lo, and Kurt D Bettenhausen. About the Importance of Autonomy and Digital Twins for the Future of Manufacturing.IFAC- PapersOnLine, 48(3):567–572, 2015.doi:10.1016/j.ifacol.2015.06.141

    work page doi:10.1016/j.ifacol.2015.06.141 2015

  69. [69]

    Safety Assurance Challenges for Autonomous Drones in Underground Mining Environments

    Philippa Ryan, Arjun Badyal, Samuel Sze, Benjamin Hardin, Hasan Bin Firoz, Paulina Lewinska, and Victoria Hodge. Safety Assurance Challenges for Autonomous Drones in Underground Mining Environments. InTowards Autonomous Robotic Systems, page 169–181, Berlin, Heidelberg, 2025. Springer.doi:10.1007/978-3-031-72059-8_15

    work page doi:10.1007/978-3-031-72059-8_15 2025

  70. [70]

    Neuro-Symbolic Artificial Intelligence: Current Trends.AI Communications, 34(3):197–209, 2021.doi: 10.3233/AIC-210084

    Md Kamruzzaman Sarker, Lu Zhou, Aaron Eberhart, and Pascal Hitzler. Neuro-Symbolic Artificial Intelligence: Current Trends.AI Communications, 34(3):197–209, 2021.doi: 10.3233/AIC-210084

    work page doi:10.3233/aic-210084 2021

  71. [71]

    Seshia, Dorsa Sadigh, and S

    Sanjit A. Seshia, Dorsa Sadigh, and S. Shankar Sastry. Toward Verified Artificial Intelli- gence.Communications of the ACM, 65(7):46–55, 2022.doi:10.1145/3503914

    work page doi:10.1145/3503914 2022

  72. [72]

    Kaddouh, Andy Blight, Lenka Mudrich, Pedro Ribeiro, Hugo Araújo, Rob Richardson, Louise A

    Nabil Shaukat, Shival Dubey, Bilal Y. Kaddouh, Andy Blight, Lenka Mudrich, Pedro Ribeiro, Hugo Araújo, Rob Richardson, Louise A. Dennis, Ana Cavalcanti, and Mo- hammad Reza Mousavi. Trustworthy ROS Software Architecture for Autonomous Drones Missions: From RoboChart Modelling to ROS Implementation. InInternational Confer- ence on Mechatronic and Embedded ...

    work page doi:10.1109/mesa61532.2024.10704818 2024

  73. [73]

    SUAVE: An Exemplar for Self-Adaptive Underwater Vehicles

    Gustavo Rezende Silva, Juliane Päßler, Jeroen Zwanepol, Elvin Alberts, Silvia Lizeth TapiaTarifa, IliasGerostathopoulos, EinarBrochJohnsen, andCarlosHernándezCorbato. SUAVE: An Exemplar for Self-Adaptive Underwater Vehicles. InIEEE/ACM Symposium on Software Engineering for Adaptive and Self-Managing Systems, pages 181–187. IEEE, 2023.doi:10.1109/SEAMS5907...

    work page doi:10.1109/seams59076.2023.00031 2023

  74. [74]

    A Survey of Sustainability in Large Language Models: Applications, Economics, and Challenges

    Aditi Singh, Nirmal Prakashbhai Patel, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. A Survey of Sustainability in Large Language Models: Applications, Economics, and Challenges. InAnnual Computing and Communication Workshop and Conference, pages 8–14. IEEE, 2025.doi:10.1109/CCWC62904.2025.10903774

    work page doi:10.1109/ccwc62904.2025.10903774 2025

  75. [75]

    Composi- tional Design of Multi-Robot Systems Control Software on ROS.ACM Transactions on Embedded Computing Systems, 18(5s), 2019.doi:10.1145/3358197

    Stefano Spellini, Michele Lora, Franco Fummi, and Sudipta Chattopadhyay. Composi- tional Design of Multi-Robot Systems Control Software on ROS.ACM Transactions on Embedded Computing Systems, 18(5s), 2019.doi:10.1145/3358197

    work page doi:10.1145/3358197 2019

  76. [76]

    Tadewos, Laya Shamgah, and Ali Karimoddini

    Tadewos G. Tadewos, Laya Shamgah, and Ali Karimoddini. Automatic Safe Behaviour Tree Synthesis for Autonomous Agents. InConference on Decision and Control, pages 2776–2781. IEEE, 2019.doi:10.1109/CDC40024.2019.9030183

    work page doi:10.1109/cdc40024.2019.9030183 2019

  77. [77]

    Specification-Based Testing of the Image-Recognition Performance of Automated Driving Systems.IEEE Access, 13:6321–6349, 2025.doi:10.1109/ACCESS.2024.3516082

    Kento Tanaka, Toshiaki Aoki, Takashi Tomita, Daisuke Kawakami, and Nobuo Chida. Specification-Based Testing of the Image-Recognition Performance of Automated Driving Systems.IEEE Access, 13:6321–6349, 2025.doi:10.1109/ACCESS.2024.3516082

    work page doi:10.1109/access.2024.3516082 2025

  78. [78]

    Townsend, Colin Paterson, T

    Beverley A. Townsend, Colin Paterson, T. T. Arvind, Gabriel Nemirovsky, Radu Calinescu, Ana Cavalcanti, Ibrahim Habli, and Alan Thomas. From Pluralistic Normative Principles to Autonomous-Agent Rules.Minds and Machines, 32(4):683–715, 2022. URL:https: //doi.org/10.1007/s11023-022-09614-w,doi:10.1007/S11023-022-09614-W

    work page doi:10.1007/s11023-022-09614-w 2022

  79. [79]

    Hoang-Dung Tran, Weiming Xiang, and Taylor T. Johnson. Verification Approaches for Learning-Enabled Autonomous Cyber–Physical Systems.IEEE Design & Test, 39(1):24– 34, 2022.doi:10.1109/MDAT.2020.3015712

    work page doi:10.1109/mdat.2020.3015712 2022

  80. [80]

    Verhagen, Mark A

    Ruben S. Verhagen, Mark A. Neerincx, and Myrthe L. Tielman. A Two-Dimensional Ex- planation Framework to Classify AI as Incomprehensible, Interpretable, or Understandable. InInternational Workshop on Explainable and Transparent AI and Multi-Agent Systems, page 119–138. Springer, 2021.doi:10.1007/978-3-030-82017-6_8. 41

    work page doi:10.1007/978-3-030-82017-6_8 2021

Showing first 80 references.