Scenario Execution for Robotics: A generic, backend-agnostic library for running reproducible robotics experiments and tests

Testing and evaluation of robotics systems is a difficult and oftentimes tedious task due to the systems' complexity and a lack of tools to conduct reproducible robotics experiments. Additionally, almost all available tools are either tailored towards a specific application domain, simulator or middleware. Particularly scenario-based testing, a common practice in the domain of automated driving, is not sufficiently covered in the robotics domain. In this paper, we propose a novel backend- and middleware-agnostic approach for conducting systematic, reproducible and automatable robotics experiments called Scenario Execution for Robotics. Our approach is implemented as a Python library built on top of the generic scenario description language OpenSCENARIO 2 and Behavior Trees and is made publicly available on GitHub. In extensive experiments, we demonstrate that our approach supports multiple simulators as backend and can be used as a standalone Python-library or as part of the ROS2 ecosystem. Furthermore, we demonstrate how our approach enables testing over ranges of varying values. Finally, we show how Scenario Execution for Robotics allows to move from simulation-based to real-world experiments with minimal adaptations to the scenario description file.