Framework and Methodology for Verification of a Complex Scientific Simulation Software, Flash-X

Computational science relies on scientific software as its primary instrument for scientific discovery. Therefore, similar to the use of other types of scientific instruments, correct software and the correct operation of the software is necessary for executing rigorous scientific investigations. Scientific software verification can be especially difficult, as users typically need to modify the software as part of a scientific study. Systematic methodologies for building test suites for scientific software are rare in the literature. Here, we describe a methodology that we have developed for Flash-X, a community simulation software for multiple scientific domains, that has composable components that can be permuted and combined in a multitude of ways to generate a wide range of applications. Ensuring sufficient code coverage by a test suite is particularly challenging due to this composability. Our methodology includes a consideration of trade-offs between meeting software quality goals, developer productivity, and meeting the scientific goals of the Flash-X user community.