T-Wise Presence Condition Coverage and Sampling for Configurable Systems

Sampling techniques, such as t-wise interaction sampling are used to enable efficient testing for configurable systems. This is achieved by generating a small yet representative sample of configurations for a system, which circumvents testing the entire solution space. However, by design, most recent approaches for t-wise interaction sampling only consider combinations of configuration options from a configurable system's variability model and do not take into account their mapping onto the solution space, thus potentially leaving critical implementation artifacts untested. Tartler et al. address this problem by considering presence conditions of implementation artifacts rather than pure configuration options, but do not consider the possible interactions between these artifacts. In this paper, we introduce t-wise presence condition coverage, which extends the approach of Tartler et al. by using presence conditions extracted from the code as basis to cover t-wise interactions. This ensures that all t-wise interactions of implementation artifacts are included in the sample and that the chance of detecting combinations of faulty configuration options is increased. We evaluate our approach in terms of testing efficiency and testing effectiveness by comparing the approach to existing t-wise interaction sampling techniques. We show that t-wise presence condition sampling is able to produce mostly smaller samples compared to t-wise interaction sampling, while guaranteeing a t-wise presence condition coverage of 100%.