Base-Rate Fallacy Redux and a Deep Dive Review in Cybersecurity

This paper examines the current state of the science underlying cybersecurity research with an emphasis on the non-signature-based intrusion detection domain. First, the paper re-examines the base-rate fallacy originally published by Axelsson, putting the impact of false positives into context. Given the relative high numbers of false positives, the paper argues for deeper analysis of false positives, akin to the analysis that true positives are treated to. The second section of the paper examines the metrics being used to analyze non-signature intrusion detection techniques, the current status quo of employed metrics, and the impact of the status quo on scientific advancement. Finally, the paper analyzes the use of online attack graphs and their applicability, especially in scenarios of constrained environments, such as Internet of Things devices. The use of offline attack graphs in such constrained environments is also examined. In essence, a deep dive review identified multiple areas throughout the field in which the effectiveness and validity of the scientific method can be greatly improved, e.g., through removal of logical fallacies.