Pith. sign in

REVIEW 5 major objections 5 minor 45 references

On the Effect of Ruleset Tuning and Data Imbalance on Explainable Network Security Alert Classifications: a Case-Study on DeepCASE

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Imbalanced alert streams degrade both the performance and the explanation quality of DeepCASE, and tuning detection rules to reduce imbalance restores both.

desk verdict An ambitious case study with real SOC data whose central imbalance claim is entangled with per-dataset hyperparameter tuning; worth review, but the main regression doesn't isolate label imbalance. read the letter →

arxiv 2507.01571 v1 pith:F5OOY5VZ submitted 2025-07-02 cs.CR cs.LGcs.NI

Koen T. W. Teuwen , Sam Baggen , Emmanuele Zambon , Luca Allodi This is my paper · ORCID
classification cs.CRcs.LGcs.NI
keywords SecurityOperationsCenternetworkintrusiondetectionlabelimbalanceRatiorulesettuningDeepCASEalertexplainabilityrelaxedF1-score
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that label imbalance is what hurts automated alert post-processing in security operations centers, and that the ordinary practice of tuning detection rules can fix it. The evidence is a case study of DeepCASE, a semi-supervised alert-context classifier, run on 24.5 million alerts from a real SOC with 616 replayed successful attacks. The paper builds datasets that vary imbalance by simulating the SOC's rule-filtering process while controlling for dataset size, heterogeneity, and dimensionality. Its main quantitative result is a regression on a workload-aware relaxed F1-score in which the imbalance ratio has a coefficient around $-1.03$, roughly an order of magnitude larger than any other tested factor. A second analysis finds that DeepCASE's attention-based explanations become more similar to an expert's judgment as imbalance falls, though the paper explicitly qualifies this result because the two experts did not agree strongly with each other.

What carries the argument

The central object is the Imbalance Ratio, defined in the paper as $\mathrm{IR}=1-\frac{1}{n_c-1}\sum_{i=1}^{n_c}\frac{n_{c_i}}{n-n_{c_i}}$, which approaches 1 for highly imbalanced class distributions. The argument is carried by a robust linear regression on the relaxed F1-score, a workload-aware metric that treats alerts rejected by DeepCASE as analyst workload, over datasets that separately vary label imbalance, dataset size, context heterogeneity, and dimensionality. For explanations, the machinery is the cosine similarity between DeepCASE's vectors of total attention per event and a manually constructed expert ground truth. The proposed mechanism for improvement is that disabling noisy detection rules removes many majority-class false-positive alerts, lowering the imbalance without introducing new data.

What would settle it

Feed DeepCASE a version of the Low-IR dataset in which the same imbalance ratio is reached by randomly under-sampling Non-Incident sequences instead of by retroactive rule filtering, while keeping the unfiltered event types; if the relaxed F1 gain disappears, the effect is caused by rule-based complexity reduction, not by label imbalance itself.

Watch

Extended reading notes

Core claim

The paper's central claim, stated on its own terms, is that label imbalance is the dominant determinant of DeepCASE's operational classification performance and that reducing imbalance through ruleset tuning improves both classification and explanation quality. The regression on the relaxed F1-score gives the label imbalance ratio a coefficient of $-1.03$ ($p < 0.01$), while dataset size, heterogeneity, and unique events have coefficients between $-0.04$ and $0.01$, leading the authors to conclude that imbalance, not these other data characteristics, drives the loss. On the explanation side, a substantial share of DeepCASE's vectors of total attention per event are far from the expert-labeled relevance vectors on the unfiltered data, and the whole similarity distribution shifts upward as the imbalance ratio falls through the tuned datasets. The paper concludes that ruleset tuning, by suppressing noisy majority-class alerts, is an effective way to improve both the accuracy and the explainability of alert post-processing.

Load-bearing premise

The quality of an explanation is measured by how well it matches a single security expert's opinion, yet the paper's own validation shows that two experts agree with each other about as much as DeepCASE agrees with one expert, so the measured improvement in explanation correctness has no settled meaning without stronger expert consensus.

Editorial extensions

If this is right

  • SOCs that already tune their rulesets can expect DeepCASE-style post-processing to retrieve more incidents and reject fewer non-incident alerts.
  • Label imbalance, rather than dataset size or context heterogeneity, should be reported and controlled in any evaluation of alert post-processing methods.
  • Micro-averaged F1 masks minority-class failures in imbalanced SOC data, so workload-aware metrics such as the relaxed F1-score should be preferred.
  • Reducing imbalance through input-data quality offers a complementary path to improving automation alongside algorithmic fixes such as focal loss or cost-sensitive learning.
  • Ruleset tuning alone reduced irrelevant alerts by 99.6 percent in the study, meaning traditional detection hygiene and ML-based post-processing are not competing approaches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the causal story is right, data-level imbalance remedies such as under-sampling the majority class or oversampling incidents should reproduce most of the tuning gain; this is a testable hypothesis the paper does not run.
  • The explanation-correctness result is only as strong as expert consensus; with expert agreement comparable to DeepCASE-to-expert agreement, a panel-based ground truth would be needed to certify the improvement.
  • Applying the same tuning simulation to NoDoze or AlertPro would test whether the imbalance effect generalizes beyond DeepCASE to other context-based alert triage methods.
  • The relaxed F1-score could be adopted more widely as an operational performance metric for alert triage, but its usefulness depends on SOCs making the analyst workload cost of rejected alerts explicit.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper uses a real SOC alert dataset with replayed attacks to study whether ruleset tuning, by reducing label imbalance, improves the classification performance and explanation quality of DeepCASE. The authors construct unfiltered, high-IR, medium-IR, and low-IR datasets, plus control datasets varying dataset size, event imbalance, heterogeneity, and dimensionality, and run DeepCASE on each with per-dataset hyperparameter optimization. A regression over 80 runs reports a label-IR coefficient of -1.03 on a relaxed F1-score (Table 1), and a comparison of attention vectors against one expert's labels suggests that tuned datasets produce more similar explanations (Figure 2). The paper concludes that ruleset tuning can benefit both classification performance and explainability of alert post-processing methods.

Significance. The practical question addressed here is important: if simple ruleset tuning improves the input quality for ML-based alert post-processing, then SOCs have a low-cost lever for improving automation. The paper makes a useful contribution by working with real SOC data, by proposing and using a relaxed F1 metric that reflects analyst workload, and by attempting to control for dataset size, heterogeneity, and dimensionality. It also explicitly discusses several known pitfalls in ML-for-security evaluations. However, the central attribution of the performance gain to label imbalance is currently confounded by per-dataset hyperparameter optimization and by collinearity with event IR, and the explanation-correctness claim is weakened by the paper's own admission that expert agreement is no better than DeepCASE-to-expert agreement. These are load-bearing issues that require additional experimental and statistical work.

major comments (5)
  1. [§3.2, Table 1, Appendix C.1] The regression that supports the paper's main quantitative claim conflates label imbalance with per-dataset hyperparameter optimization. Section 3.2 states that each run uses the hyperparameters obtained from the optimization in Appendix A, and Table 4 shows materially different optima across datasets (e.g., minimum cluster size 50 for unfiltered vs. 5 for all tuned sets; epsilon 0.8 vs. 1.0). The hyperparameter control experiment in Appendix C.1 shows that under the original default hyperparameters the low-IR dataset's macro F1 falls from 0.854 to 0.447, whereas the other datasets change far less. The coefficient of -1.03 on label IR in Table 1 therefore absorbs the joint effect of imbalance and hyperparameter choice, not the effect of imbalance alone. Because Section 4.2's explanation comparison also uses the per-dataset optimized models, the same confound applies to the explanation results. A fixed-hyperparameter comparison, or a design that treats hyperparameters as a factor and estimates their interaction, is required to support the attribution.
  2. [Table 1] The standard errors and confidence intervals in Table 1 are computed on 80 runs that are not independent: the runs consist of five repetitions of each of 16 dataset configurations (four IR-experiment datasets plus twelve control datasets). A robust linear model applied to the run-level data treats the five runs from one configuration as independent observations, which is pseudoreplication and will produce overconfident intervals. The reported interval [-1.04, -1.02] for the label-IR coefficient is implausibly narrow given the dispersion visible in Figure 1. The regression should be estimated with standard errors clustered by dataset configuration, or using configuration-level means.
  3. [Table 1, §5] The paper cannot separate label IR from event IR. The event-IR variable was dropped from the regression because its VIF exceeded 10, and Section 5 acknowledges that part of the label-IR effect may be due to event IR. Because ruleset tuning changes both quantities together, the evidence supports a claim about imbalance generally, not specifically about label imbalance as the title and research questions assert. The authors should either obtain data or experiments that vary label IR while holding event IR fixed, or reframe the central claim accordingly.
  4. [§4.2, §5, Appendix D] The explanation-correctness conclusion is stronger than the paper's own validity caveat allows. Section 5 states that expert 1 and expert 2 agreed with each other about as much as DeepCASE agreed with expert 1, and that 'we cannot draw strong conclusions from the correctness of DeepCASE's explanations.' Given that statement, Figure 2's comparison of DeepCASE to expert 1's vectors cannot support the Section 4.2 conclusion that tuning 'effectively assists DeepCASE in providing correct explanations.' At most it shows that tuning moves DeepCASE closer to one expert's judgments. The conclusion should be reworded, or additional validation of the ground truth should be provided.
  5. [§3.2, §5] The acknowledged temporal data snooping is not shown to be neutral across treatments. The procedure copies test-set Incident sequences into the training set until the train and test Incident percentages are similar, so the amount of copying is a function of the imbalance manipulation itself. The statement in Section 5 that the pitfall is 'present equally in all performed experiments' needs quantitative support: the authors should report, per dataset, how many test sequences were copied and how the resulting training-set composition differs, and ideally run a sensitivity analysis without copying.
minor comments (5)
  1. [§3.2] The text uses 'Variance Influence Factor'; the standard term is 'Variance Inflation Factor'.
  2. [§3.2] There is a typo in 'context hetetogeneity'; it should read 'heterogeneity'.
  3. [§4.1, Table 1] The KS test p-value of 0.15 only indicates a failure to reject a distributional hypothesis; it is not a measure of variance explained. Reporting R-squared or adjusted R-squared would make the fit assessment more informative.
  4. [§6] The conclusion states that 'label imbalance' is the key driver, without the hedge that Section 5 applies ('be it event or label'). Align the conclusion with the acknowledged ambiguity.
  5. [Data Availability] The data and code are described as archived at the institution and shared upon request; a public artifact would materially strengthen reproducibility of the regression and control experiments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirical comparisons against measured labels and expert ratings, not derivations from fitted inputs.

full rationale

The paper's central claims are empirical: it varies label imbalance through ruleset filtering, runs DeepCASE with per-dataset optimized hyperparameters plus a default-hyperparameter control, and regresses measured relaxed F1 on measured dataset characteristics (Table 1). The label-IR coefficient (-1.03) is an estimated effect, not a parameter fitted to the outcome and then renamed a prediction; no equation in the paper defines IR in terms of F1 or vice versa. DeepCASE [8] is not self-cited, since none of its authors overlap with the present author list. Self-citations [6] and [31] are contextual (rule-design prior work, attack-injection dataset) and are not load-bearing for the imbalance claim. The explanation analysis compares DeepCASE attention vectors to external expert labels, and the paper explicitly flags the limit of this ground truth in Section 5: 'Since expert 1 and expert 2 agreed with each other about as much as DeepCASE agreed with expert 1, we cannot draw strong conclusions from the correctness of DeepCASE's explanations.' That is an honest limitation, not a circular step. The hyperparameter-optimization effect (Appendix C.1, Figure 3) is a validity confound: the low-IR advantage largely disappears under default hyperparameters, but the paper reports this control, and a confound is not a definitional reduction; it is a correctness risk, not a circularity risk.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a synthetic minority class, an expert-labeled explanation ground truth that shows only moderate inter-rater agreement, and two methodological choices (test-to-train copying and regression with collinear predictors) that the authors themselves flag as potentially biasing. These are not equations fitted to the target outcome, so they are listed as axioms rather than free parameters, except for the per-dataset hyperparameters that are genuinely tuned.

free parameters (1)
  • DeepCASE hyperparameters = Vary per dataset, e.g., context length 15/10/20/15 for unfiltered/high/medium/low IR (Table 4)
    Selected by random search on macro F1 per dataset; different hyperparameters per dataset could confound the comparison, though a control experiment with default values was run.
assumptions (4)
  • domain assumption The replayed attack traffic produces alerts indistinguishable from real incidents in the monitored environment.
    Section 3.1 describes injection of 10 recorded attacks into the SOC SIEM and manual inspection of the 616 resulting alerts; the entire Incident class is synthetic, so if these alerts differ systematically from real incidents the imbalance effect may not generalize.
  • domain assumption Expert labels for explanation relevance are a valid ground truth.
    Appendix D shows experts agreed on cosine similarity >0.8 for about 60% of vectors, but Section 5 concedes agreement was comparable to DeepCASE's agreement with an expert, weakening the ground truth.
  • ad hoc to paper Copying test-set incidents into the training set does not bias the comparison across datasets.
    Section 3.2 explicitly acknowledges this is temporal data snooping but argues it is present equally across all experiments; this is an unvalidated assumption about the comparability of the snooping effect.
  • ad hoc to paper The regression attribution to label IR is valid despite collinearity with event IR.
    Section 5 notes that event IR was dropped from the regression due to high VIF and that part of the label IR effect may actually be event IR; this weakens the specific attribution to label imbalance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Effect of Ruleset Tuning and Data Imbalance on Explainable Network Security Alert Classifications: a Case-Study on DeepCASE." pith.science (2026). https://pith.science/paper/F5OOY5VZ

@misc{pith2026250701571,
  author       = {Pith},
  title        = {Pith review of: On the Effect of Ruleset Tuning and Data Imbalance on Explainable Network Security Alert Classifications: a Case-Study on DeepCASE},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F5OOY5VZ}},
  note         = {Machine review of arXiv:2507.01571}
}
read the original abstract

Automation in Security Operations Centers (SOCs) plays a prominent role in alert classification and incident escalation. However, automated methods must be robust in the presence of imbalanced input data, which can negatively affect performance. Additionally, automated methods should make explainable decisions. In this work, we evaluate the effect of label imbalance on the classification of network intrusion alerts. As our use-case we employ DeepCASE, the state-of-the-art method for automated alert classification. We show that label imbalance impacts both classification performance and correctness of the classification explanations offered by DeepCASE. We conclude tuning the detection rules used in SOCs can significantly reduce imbalance and may benefit the performance and explainability offered by alert post-processing methods such as DeepCASE. Therefore, our findings suggest that traditional methods to improve the quality of input data can benefit automation.

Figures

Figures reproduced from arXiv: 2507.01571 by the authors.

Figure 2
Figure 2. Cumulative distribution plots of the cosine similarity comparing [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. Cumulative Distribution of the Cosine Similarity of Expert [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 31 canonical work pages

  1. [1]

    Knerler, I

    K. Knerler, I. Parker, and C. Zimmerman, 11 Strategies of a World-Class Cybersecurity Operations Center , 2022. [On- line]. Available: https://www.mitre.org/news-insights/publication/ 11-strategies-world-class-cybersecurity-operations-center

    work page 2022

  2. [2]

    Security operations center: A systematic study and open challenges,

    M. Vielberth, F. Bohm, I. Fichtinger, and G. Pernul, “Security operations center: A systematic study and open challenges,” IEEE Access, vol. 8, p. 227756–227779, 2020. [Online]. Available: https://doi.org/10.1109/ACCESS.2020.3045514

    work page arXiv 2020

  3. [3]

    Suricata,

    Open Information Security Foundation, “Suricata,” Novem- ber 2024. [Online]. Available: https://web.archive.org/web/ 20241001122450/https://suricata.io/

    work page 2024

  4. [4]

    ’give me structure’: Synthesis and evaluation of a (network) threat analysis process supporting tier 1 investigations in a security operation center,

    L. Kersten, T. Mulders, E. Zambon, C. Snijders, and L. Allodi, “’give me structure’: Synthesis and evaluation of a (network) threat analysis process supporting tier 1 investigations in a security operation center,” in Nineteenth Symposium on Usable Privacy and Security (SOUPS 2023) . Anaheim, CA: USENIX Association, Aug. 2023, pp. 97–111. [Online]. Availa...

    work page 2023

  5. [5]

    True attacks, attack attempts, or benign triggers? an empirical measurement of network alerts in a security operations center,

    L. Yang, Z. Chen, C. Wang, Z. Zhang, S. Booma, P. Cao, C. Adam, A. Withers, Z. Kalbarczyk, R. K. Iyer, and G. Wang, “True attacks, attack attempts, or benign triggers? an empirical measurement of network alerts in a security operations center,” in 33rd USENIX Security Symposium (USENIX Security 24) . Philadelphia, PA: USENIX Association, Aug. 2024, pp. 15...

    work page 2024

  6. [6]

    Ruling the unruly: Designing effective, low-noise network intrusion detection rules for security operations centers,

    K. Teuwen, T. Mulders, E. Zambon, and L. Allodi, “Ruling the unruly: Designing effective, low-noise network intrusion detection rules for security operations centers,” in Proceedings of the 2025 ACM on Asia Conference on Computer and Communications Security . New York, NY , USA: Association for Computing Machinery, Inc, 2024. [Online]. Available: https://...

    work page arXiv 2025

  7. [7]

    Alert alchemy: Soc workflows and decisions in the management of nids rules,

    M. Vermeer, N. Kadenko, M. van Eeten, C. Ga ˜n´an, and S. Parkin, “Alert alchemy: Soc workflows and decisions in the management of nids rules,” in Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security . New York, NY , USA: Association for Computing Machinery, 2023, p. 2770–2784. [Online]. Available: https://doi.org/10.1145/...

    arXiv 2023

  8. [8]

    DeepCASE: Semi-Supervised Contextual Analysis of Security Events,

    T. van Ede, H. Aghakhani, N. Spahn, R. Bortolameotti, M. Cova, A. Continella, M. van Steen, A. Peter, C. Kruegel, and G. Vigna, “DeepCASE: Semi-Supervised Contextual Analysis of Security Events,” in Proceedings of the IEEE Symposium on Security and Privacy (S&P). IEEE, 2022

    work page 2022

Show all 45 references
  1. [9]

    Omega- log: High-fidelity attack investigation via transparent multi-layer log analysis,

    W. U. Hassan, M. A. Noureddine, P. Datta, and A. Bates, “Omega- log: High-fidelity attack investigation via transparent multi-layer log analysis,” in Network and distributed system security sympo- sium, 2020

    2020

  2. [10]

    Combating alert fatigue with alertpro: Context-aware alert prioritization using reinforcement learning for multi-step attack detection,

    X. Wang, X. Yang, X. Liang, X. Zhang, W. Zhang, and X. Gong, “Combating alert fatigue with alertpro: Context-aware alert prioritization using reinforcement learning for multi-step attack detection,” Computers & Security , vol. 137, p. 103583, 2024. [Online]. Available: https:/...

    2024

  3. [11]

    Insomnia: Towards concept-drift robustness in network intrusion detection,

    G. Andresini, F. Pendlebury, F. Pierazzi, C. Loglisci, A. Appice, and L. Cavallaro, “Insomnia: Towards concept-drift robustness in network intrusion detection,” in Proceedings of the 14th ACM Workshop on Artificial Intelligence and Security . New York, NY , USA: Association fo...

    2021

  4. [12]

    Transcending transcend: Revisiting malware classification in the presence of concept drift,

    F. Barbero, F. Pendlebury, F. Pierazzi, and L. Cavallaro, “Transcending transcend: Revisiting malware classification in the presence of concept drift,” in 2022 IEEE Symposium on Security and Privacy (SP) , 2022, pp. 805–823. [Online]. Available: https://doi.org/10.1109/SP46214...

    2022

  5. [13]

    Anomaly detection in the open world: Normality shift detection, explanation, and adap- tation,

    D. Han, Z. Wang, W. Chen, K. Wang, R. Yu, S. Wang, H. Zhang, Z. Wang, M. Jin, J. Yang, X. Shi, and X. Yin, “Anomaly detection in the open world: Normality shift detection, explanation, and adap- tation,” in 30th Annual Network and Distributed System Security Symposium, NDSS 20...

    2023

  6. [14]

    xNIDS: Explaining deep learning-based network intrusion detection systems for active intrusion responses,

    F. Wei, H. Li, Z. Zhao, and H. Hu, “xNIDS: Explaining deep learning-based network intrusion detection systems for active intrusion responses,” in 32nd USENIX Security Symposium (USENIX Security 23) . Anaheim, CA: USENIX Association, Aug. 2023, pp. 4337–4354. [Online]. Availabl...

    2023

  7. [15]

    Ruling the rules: Quantifying the evolution of rulesets, alerts and incidents in network intrusion detection,

    M. Vermeer, M. van Eeten, and C. Ga ˜n´an, “Ruling the rules: Quantifying the evolution of rulesets, alerts and incidents in network intrusion detection,” in Proceedings of the 2022 ACM on Asia Conference on Computer and Communications Security . New York, NY , USA: Associatio...

    2022

  8. [16]

    Detecting credential spearphishing attacks in enterprise settings,

    G. Ho, A. Sharma, M. Javed, V . Paxson, and D. Wagner, “Detecting credential spearphishing attacks in enterprise settings,” Proc. of 26th USENIX Security , 2017

    2017

  9. [17]

    A systematic review on imbalanced data challenges in machine learning: Applications and solutions,

    H. Kaur, H. S. Pannu, and A. K. Malhi, “A systematic review on imbalanced data challenges in machine learning: Applications and solutions,” ACM Comput. Surv. , vol. 52, no. 4, Aug. 2019. [Online]. Available: https://doi.org/10.1145/3343440

    2019 doi

  10. [18]

    An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics,

    V . L ´opez, A. Fern ´andez, S. Garc ´ıa, V . Palade, and F. Herrera, “An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics,” Information Sciences , vol. 250, pp. 113–141, 2013. [Online]. Available: h...

    2013 doi

  11. [19]

    The base-rate fallacy and the difficulty of intrusion detection,

    S. Axelsson, “The base-rate fallacy and the difficulty of intrusion detection,” ACM Trans. Inf. Syst. Secur., vol. 3, no. 3, p. 186–205, 8

  12. [20]

    TESSERACT: Eliminating experimental bias in malware classification across space and time,

    F. Pendlebury, F. Pierazzi, R. Jordaney, J. Kinder, and L. Cavallaro, “TESSERACT: Eliminating experimental bias in malware classification across space and time,” in 28th USENIX Security Symposium (USENIX Security 19) . Santa Clara, CA: USENIX Association, Aug. 2019, pp. 729–

    2019

  13. [21]

    Dos and don’ts of machine learning in computer security,

    D. Arp, E. Quiring, F. Pendlebury, A. Warnecke, F. Pierazzi, C. Wressnegger, L. Cavallaro, and K. Rieck, “Dos and don’ts of machine learning in computer security,” in 31st USENIX Security Symposium (USENIX Security 22) . Boston, MA: USENIX Association, Aug. 2022, pp. 3971–3988...

    2022

  14. [22]

    The curse of dimensionality in data mining and time series prediction,

    M. Verleysen and D. Franc ¸ois, “The curse of dimensionality in data mining and time series prediction,” in Computational Intelligence and Bioinspired Systems, J. Cabestany, A. Prieto, and F. Sandoval, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2005, pp. 758–770

    2005

  15. [23]

    Error prevalence in nids datasets: A case study on cic-ids-2017 and cse-cic-ids-2018,

    L. Liu, G. Engelen, T. Lynar, D. Essam, and W. Joosen, “Error prevalence in nids datasets: A case study on cic-ids-2017 and cse-cic-ids-2018,” in 2022 IEEE Conference on Communications and Network Security (CNS) , 2022, pp. 254–262. [Online]. Available: https://doi.org/10.1109...

    2017

  16. [24]

    Sok: Explainable machine learning for computer security applications,

    A. Nadeem, D. V os, C. Cao, L. Pajola, S. Dieck, R. Baumgartner, and S. Verwer, “Sok: Explainable machine learning for computer security applications,” in 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P) , 2023, pp. 221–240. [Online]. Available: https://doi.o...

    2023

  17. [25]

    A review on the attention mechanism of deep learning,

    Z. Niu, G. Zhong, and H. Yu, “A review on the attention mechanism of deep learning,” Neurocomputing, vol. 452, pp. 48–62, 2021. [Online]. Available: https://doi.org/10.1016/j.neucom.2021.03.091

    2021 doi

  18. [26]

    A density-based algorithm for discovering clusters in large spatial databases with noise,

    M. Ester, H.-P. Kriegel, J. Sander, X. Xu et al., “A density-based algorithm for discovering clusters in large spatial databases with noise,” in kdd, vol. 96, no. 34, 1996, pp. 226–231

    1996

  19. [27]

    How complex is your classification problem? A survey on measuring classification complexity,

    A. C. Lorena, L. P. F. Garcia, J. Lehmann, M. C. P. de Souto, and T. K. Ho, “How complex is your classification problem? A survey on measuring classification complexity,” CoRR, vol. abs/1808.03591, 2018. [Online]. Available: http: //arxiv.org/abs/1808.03591

    2018 arXiv

  20. [28]

    The effect of dataset size on training tweet sentiment classifiers,

    J. Prusa, T. M. Khoshgoftaar, and N. Seliya, “The effect of dataset size on training tweet sentiment classifiers,” in 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA) , 2015, pp. 96–102. [Online]. Available: https://doi.org/10.1109/ICMLA.2015.22

    2015 doi

  21. [29]

    Nodoze: Combatting threat alert fatigue with automated provenance triage,

    W. U. Hassan, S. Guo, D. Li, Z. Chen, K. Jee, Z. Li, and A. Bates, “Nodoze: Combatting threat alert fatigue with automated provenance triage,” in network and distributed systems security symposium, 2019

    2019

  22. [30]

    Bad design smells in benchmark nids datasets,

    R. Flood, G. Engelen, D. Aspinall, and L. Desmet, “Bad design smells in benchmark nids datasets,” in 2024 IEEE 9th European Symposium on Security and Privacy (EuroS&P) , 2024, pp. 658–675. [Online]. Available: https://doi.org/10.1109/ EuroSP60621.2024.00042

    2024

  23. [31]

    ’give me structure’: Synthesis and evaluation of a (network) threat analysis process supporting tier 1 investigations in a security operation center,

    L. Kersten, T. Mulders, E. Zambon-Mazzocato, C. Snijders, and L. Allodi, “’give me structure’: Synthesis and evaluation of a (network) threat analysis process supporting tier 1 investigations in a security operation center,” in Proceedings of the 19th Sym- posium on Usable Pri...

    2023

  24. [32]

    Troubleshooting an intrusion detection dataset: the cicids2017 case study,

    G. Engelen, V . Rimmer, and W. Joosen, “Troubleshooting an intrusion detection dataset: the cicids2017 case study,” in 2021 IEEE Security and Privacy Workshops (SPW) . IEEE, 2021, pp. 7–12

    2021

  25. [33]

    B. K. Slinker and S. A. Glantz, Primer of applied regression and analysis of variance . McGraw-Hill, 1990

    1990

  26. [34]

    G. S. Maddala and K. Lahiri, Introduction to econometrics . Macmillan New York, 1992, vol. 2

    1992

  27. [35]

    Regression with graphics: A second course in applied statistics,

    J. B. Gray, “Regression with graphics: A second course in applied statistics,” 1994

    1994

  28. [36]

    The kolmogorov-smirnov test for goodness of fit,

    F. J. Massey Jr, “The kolmogorov-smirnov test for goodness of fit,” Journal of the American statistical Association , vol. 46, no. 253, pp. 68–78, 1951

    1951

  29. [37]

    Capturing tacit knowledge in security operation centers,

    S. Y . Cho, J. Happa, and S. Creese, “Capturing tacit knowledge in security operation centers,” IEEE Access, vol. 8, pp. 42 021–42 041, 2020

    2020

  30. [38]

    A tale of four metrics,

    R. Connor, “A tale of four metrics,” in Similarity Search and Applications, L. Amsaleg, M. E. Houle, and E. Schubert, Eds. Cham: Springer International Publishing, 2016, pp. 210–217

    2016

  31. [39]

    A unifying view of class overlap and imbalance: Key concepts, multi-view panorama, and open avenues for research,

    M. S. Santos, P. H. Abreu, N. Japkowicz, A. Fern ´andez, and J. Santos, “A unifying view of class overlap and imbalance: Key concepts, multi-view panorama, and open avenues for research,” Information Fusion , vol. 89, p. 228–253, Jan. 2023. [Online]. Available: https://doi.org...

    2023 doi

  32. [40]

    Focal loss for dense object detection,

    T.-Y . Lin, P. Goyal, R. Girshick, K. He, and P. Doll ´ar, “Focal loss for dense object detection,” in 2017 IEEE International Conference on Computer Vision (ICCV) , 2017, pp. 2999–3007. [Online]. Available: https://doi.org/10.1109/ICCV .2017.324

    2017 doi

  33. [41]

    Deepdive: Evaluating deepcase on dataset imbalance & validity of explanations,

    S. Baggen, “Deepdive: Evaluating deepcase on dataset imbalance & validity of explanations,” Master’s thesis, Mathematics and Computer Science, 2024. [Online]. Available: https://research.tue.nl/en/studentTheses/ deepdive-evaluating-deepcase-on-dataset-imbalance-validity-of-exp

    2024

  34. [42]

    Random search for hyper-parameter optimization

    J. Bergstra and Y . Bengio, “Random search for hyper-parameter optimization.” Journal of machine learning research, vol. 13, no. 2, 2012. Appendix A. Hyperparameter optimization DeepCASE has several hyperparameters that can be optimized as discussed in Section 2.1 and detailed...

    2012

  35. [45]

    O PTIMAL HYPERPARAMETERS FOR THE DIFFERENT DATASETS

    https://github.com/Thijsvanede/DeepCASE/blob/sp/experiments/ baseline/results/deepcase after automatic.txt#L58 11 TABLE 4. O PTIMAL HYPERPARAMETERS FOR THE DIFFERENT DATASETS . Hyperparameter Unfiltered High Label IR Medium Label IR Low Label IR Context Length 15 10 20 15 Cont...

  36. [746]

    Available: https://www.usenix.org/conference/ usenixsecurity19/presentation/pendlebury 9

    [Online]. Available: https://www.usenix.org/conference/ usenixsecurity19/presentation/pendlebury 9

  37. [2000]

    Available: https://doi.org/10.1145/357830.357849

    [Online]. Available: https://doi.org/10.1145/357830.357849

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.