Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Robust Anomaly Detection in Network Traffic: Evaluating Machine Learning Models on CICIDS2017

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A one-class SVM trained only on benign network traffic detects previously unseen attack types more reliably than supervised deep-learning models on CICIDS2017.

desk verdict Useful per-class benchmark on CICIDS2017, but the OCSVM 'most practical' conclusion is not supported by the paper's own numbers—Bot accuracy is 4.4%. read the letter →

arxiv 2506.19877 v2 pith:HMVSVVWE submitted 2025-06-23 cs.CR cs.AIcs.LG

Zhaoyang Xu , Yunbo Liu This is my paper
classification cs.CRcs.AIcs.LG
keywords IntrusionDetectionSystem(IDS)AnomalyUnsupervisedLearningCICIDS2017NetworkSecurityMachineOne-ClassSVMLocalOutlierFactor
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper compares four machine-learning models—two supervised deep networks, a one-class support vector machine, and a density-based outlier detector—on the CICIDS2017 network-traffic dataset. The authors hold out three attack types as "unknown" and find that supervised models (MLP, CNN) excel on familiar attacks but largely fail on novel ones, with recall dropping to 0.175 and 0.195. The one-class SVM, trained only on benign traffic, achieves the best balance on the unknown-attack test set (F1 0.758) and is argued to be the most practical model for real-world intrusion detection. The paper aims to establish that learning a boundary around normal behavior generalizes better to unseen threats than learning signatures of known attacks.

What carries the argument

The load-bearing mechanism is the One-Class SVM, a boundary-based anomaly detector that learns a compact decision surface around normal (benign) traffic in a kernel feature space and flags anything outside that surface as an attack. The experimental setup reinforces this by holding out three attack types—DoS slowloris, DoS Slowhttptest, and Bot—from all training, so the unknown-attack test measures genuine generalization.

What would settle it

Run the same comparison on a deduplicated CICIDS2017 split plus a second benchmark such as UNSW-NB15, holding out a wider and more varied set of unseen attack types (including low-volume, stealthy ones); if OCSVM's F1 on unknown attacks no longer exceeds MLP/CNN or falls below them, the paper's central generalization claim is falsified.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a One-Class SVM (OCSVM) that trains exclusively on benign traffic detects novel attack types more reliably than supervised deep models: on the Unknown Attack Test Set it reaches Accuracy 0.7919 and F1 0.7575, while MLP and CNN collapse to F1 below 0.33. The authors interpret this as evidence that "training exclusively on benign data supports better generalization to unseen malicious behaviors," and they conclude that boundary-based anomaly detection is the most practical approach for real-world intrusion detection.

Load-bearing premise

The paper assumes the three held-out attack types (DoS slowloris, DoS Slowhttptest, Bot) stand in for novel attacks in general, and that CICIDS2017—used without deduplication—gives a valid test distribution for measuring that generalization.

Editorial extensions

If this is right

  • Deployments facing frequent novel attacks should consider benign-only boundary models like OCSVM over supervised classifiers.
  • Supervised deep models should not be relied on for zero-day detection unless retrained or augmented with anomaly-based heads.
  • The F1 gap on unknown attacks quantifies the cost of supervised overfitting: more than 0.4 F1 points on this test.
  • Tuning OCSVM parameters such as ν and the kernel could further reduce false alarms below the reported 882 benign misclassifications on the unknown-attack set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The three held-out attacks are all relatively high-volume, volumetric or botnet-like, so the generalization advantage may not extend to stealthy, low-rate, or highly evasive attacks; testing on such types would sharpen the claim.
  • CICIDS2017 is known to contain duplicate flows; deduplicating before splitting could change the reported accuracies, so the numbers should be read as upper-bound estimates.
  • OCSVM's boundary could be combined with feature selection or ensemble scoring to reduce the precision drop seen on the Overall Test Set (Precision 0.6525), a trade-off the paper does not explore.
  • A hybrid pipeline that uses supervised models for known attacks and OCSVM for residuals could yield higher combined F1 than either paradigm alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper compares four classifiers—MLP, 1D-CNN, OCSVM, and LOF—on the CICIDS2017 dataset in two scenarios: detecting known attack types and detecting three held-out 'unknown' attack types (DoS slowloris, DoS Slowhttptest, Bot). It reports that supervised MLP and CNN achieve high accuracy on known attacks but collapse on unknown ones, LOF gives moderate accuracy with high recall on unknown attacks, and OCSVM best balances precision and recall, leading the authors to recommend OCSVM as the most practical approach for real-world intrusion detection.

Significance. If the central claim were supported, a controlled comparison of supervised, boundary-based, and density-based detectors on a standard benchmark would be a useful practical reference for IDS design. The paper has some strengths: a fixed random seed for reproducible splits, binary classification metrics, and per-attack-type accuracy breakdowns. However, the main conclusion is not supported by the paper's own tables, and the experimental design has unresolved methodological issues. The contribution is therefore largely a negative or cautionary result unless the analysis is substantially reworked.

major comments (5)
  1. [§IV, Tables II–IV and §V] The central recommendation that OCSVM is the most practical model is contradicted by the paper's own results. On the Overall Test Set, OCSVM achieves F1=0.5520 with recall 0.4784, while MLP and CNN achieve F1=0.9446 and 0.9160, respectively; this is not 'robust detection across both scenarios' as claimed in the Summary of Observed Trends. On the Unknown Attack Test Set, OCSVM's pooled F1=0.7575 is a micro-average dominated by DoS Slowhttptest (5,499 of 13,261 attack samples), yet Table IV shows OCSVM's per-class accuracy on Bot is only 0.0443, compared with LOF's 0.4680. The claim that OCSVM 'maintains strong detection rates... even when facing unknown attacks' is therefore unsupported; the authors should report macro-averaged per-attack-type metrics and a leave-one-out analysis before drawing this conclusion.
  2. [§III-A] The construction of the Unknown Attack Test Set is a convenience sample. The three held-out attack types are chosen because their sample sizes are 'moderate,' but no argument is given that they are representative of unseen attacks in general, and all three may share distributional properties (low-rate or botnet-like traffic). Without leave-one-out evaluation or additional held-out attack combinations, the paper cannot support the broad claim that OCSVM generalizes to 'previously unseen threats.'
  3. [§III-A] No deduplication of CICIDS2017 is reported. CICIDS2017 is known to contain duplicate flows, and leaving duplicates in the training and test splits can inflate accuracy and mask true generalization performance. The authors should state whether duplicates were removed and, if not, rerun the experiments on a deduplicated version of the dataset.
  4. [§III-A] The description of StandardScaler usage is ambiguous for the unsupervised models. The text says the scaler is fit on 'the training set' and then applied to both test splits; for OCSVM and LOF the training set is benign-only, while for MLP and CNN it includes known attack classes. Since the test splits contain attack features far outside the benign training range, the choice of scaling distribution can materially change anomaly scores. Please specify exactly which samples were used to fit the scaler for each model and justify that choice.
  5. [§IV] No variance or significance information is provided. All experiments use a single split with random_state=42, with no repeated runs, bootstrap confidence intervals, or statistical tests. Given the class imbalance and the small per-class counts for rare attacks (e.g., Bot, Web Attack – SQL Injection), the reported differences between models, and especially the OCSVM-versus-LOF comparison on unknown attacks, may not be stable.
minor comments (5)
  1. [§III-C2] The CNN subsection opens with an incomplete phrase, 'benign tr,' which appears to be a fragment of a sentence; it should be completed.
  2. [§IV] The confusion-matrix numbers for MLP on the Overall Test Set (TN=453,433, FP=1,187, FN=11,771, TP=110,367) sum to 576,758, which does not match the reported test-set size; please verify the arithmetic and the split counts.
  3. [§V] The 'Conclusions from Model Behavior' paragraph repeats the unsupported claim that OCSVM is 'especially well-suited for real-world intrusion detection'; this should be revised to match the evidence presented in Tables II–IV.
  4. [References] Reference [24] lists an arXiv preprint but the 'Available' field is empty; the URL should be provided.
  5. [§II] There is a punctuation and grammar issue in the sentence 'this capability to detect novel threats is well-supported in the anomaly-detection literature [4]. Any significant deviation...'—the period before 'Any' should likely be a comma or semicolon.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported metrics are measured on held-out test data, and the self-citations appear only in discussion and future work, not in the load-bearing empirical argument.

full rationale

The paper's central claims are empirical comparisons of MLP, CNN, OCSVM, and LOF on held-out portions of CICIDS2017. The reported accuracies, recalls, F1-scores, and confusion-matrix counts are computed directly from test-set predictions; no parameter is fitted to the target conclusion, and no test-set result is fed back into model construction. The OCSVM 'best for unknown attacks' claim is an interpretation of Table III and Figure 2, not a consequence of the model definitions or of any cited prior work. The self-citations that do appear, such as [12], [15], and [26], are confined to the discussion and future-work paragraphs and are not used to justify the experimental design or the correctness of the measurements. There is no imported uniqueness theorem, no ansatz smuggled in via citation, and no renaming of a known result. A legitimate scientific weakness exists: the pooled Unknown Attack Test Set metric is dominated by DoS Slowhttptest, and Table IV shows OCSVM's per-class accuracy on Bot is only 0.0443, so the generalization claim is stronger than the per-class evidence supports. However, that is a validity or robustness concern, not circularity: the conclusion does not reduce to its inputs by construction. The derivation chain from experiment to conclusion is therefore self-contained with respect to circularity, and the appropriate score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on dataset and split assumptions rather than on derived equations. The primary load-bearing choices are the selection of the three unknown-attack types, the lack of deduplication, and the ambiguous scaling procedure.

free parameters (5)
  • OCSVM nu = 0.05
    Hand-set boundary tightness parameter; directly controls the false alarm versus recall tradeoff that underlies the paper's recommendation of OCSVM.
  • LOF n_neighbors = 80
    Hand-set number of neighbors for density estimation; controls which flows are flagged as outliers.
  • MLP hidden units = (100, 50)
    Architectural choice; no tuning procedure reported.
  • CNN focal loss gamma and alpha = 2.0 / 0.25
    Hand-set loss parameters together with class weights (benign=1.0, malicious=5.0) that shape the CNN training objective.
  • Random seed = 42
    Single seed for all splits and model initialization; no multi-seed analysis is performed.
assumptions (4)
  • domain assumption CICIDS2017 labels and features are accurate and representative of real network traffic.
    Invoked in Section III-A as the evaluation testbed; any label noise or non-representative sampling would bias all model comparisons.
  • domain assumption The three held-out attack types are a fair proxy for unknown attacks.
    Section III-A states they were selected for moderate sample size, not for coverage of the attack taxonomy.
  • domain assumption Applying the StandardScaler fitted on the training set does not leak information into the test splits.
    Section III-A; for OCSVM and LOF the training set is benign-only, yet the text does not specify which training set is used to fit the scaler.
  • domain assumption Duplicate records in CICIDS2017 do not affect the results.
    No deduplication is described; duplicates are a known artifact of this dataset and can inflate accuracy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Robust Anomaly Detection in Network Traffic: Evaluating Machine Learning Models on CICIDS2017." pith.science (2026). https://pith.science/paper/HMVSVVWE

@misc{pith2026250619877,
  author       = {Pith},
  title        = {Pith review of: Robust Anomaly Detection in Network Traffic: Evaluating Machine Learning Models on CICIDS2017},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HMVSVVWE}},
  note         = {Machine review of arXiv:2506.19877}
}
read the original abstract

Identifying suitable machine learning paradigms for intrusion detection remains critical for building effective and generalizable security solutions. In this study, we present a controlled comparison of four representative models - Multi-Layer Perceptron (MLP), 1D Convolutional Neural Network (CNN), One-Class Support Vector Machine (OCSVM) and Local Outlier Factor (LOF) - on the CICIDS2017 dataset under two scenarios: detecting known attack types and generalizing to previously unseen threats. Our results show that supervised MLP and CNN achieve near-perfect accuracy on familiar attacks but suffer drastic recall drops on novel attacks. Unsupervised LOF attains moderate overall accuracy and high recall on unknown threats at the cost of elevated false alarms, while boundary-based OCSVM balances precision and recall best, demonstrating robust detection across both scenarios. These findings offer practical guidance for selecting IDS models in dynamic network environments.

Figures

Figures reproduced from arXiv: 2506.19877 by the authors.

Figure 2
Figure 2. Confusion matrices on the Unknown Attack Test [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Latent Sculpting for Zero-Shot Generalization: A Manifold Learning Approach to Out-of-Distribution Anomaly Detection

    cs.LG 2025-12 conditional novelty 4.0 of 10

    Two-stage DCCL+MAF detector reports 0.87 zero-shot OOD F1 on CIC-IDS-2017, driven by DoS classes while bot recall stays at 4%.

Reference graph

Works this paper leans on

27 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [5]

    Evaluation of CI- CIDS2017 with qualitative comparison of machine learning algorithm,

    T. Elmasri, N. Samir, M. Mashaly, and Y . Atef, “Evaluation of CI- CIDS2017 with qualitative comparison of machine learning algorithm,” in 2020 IEEE Cloud Summit , 2020, pp. 46–51

    work page 2020

  2. [8]

    Anomaly based intrusion detection system: A deep learning approach,

    S. Tossou, M. Qorib, and T. Kacem, “Anomaly based intrusion detection system: A deep learning approach,” in 2023 International Symposium on Networks, Computers and Communications (ISNCC) , 2023, pp. 1–6

    work page 2023

  3. [1]

    Year-over-Year Developments in Financial Fraud Detection via Deep Learning: A Systematic Literature Review

    Y . Chen, C. Zhao, Y . Xu, and C. Nie, “Year-over-year developments in financial fraud detection via deep learning: A systematic literature review,” 2025. [Online]. Available: https://arxiv.org/abs/2502.00201

    work page Pith review arXiv 2025

  4. [2]

    The evaluation of network anomaly detection systems: Statistical analysis of the UNSW-NB15 data set and the comparison with the KDD99 data set,

    N. Moustafa and J. Slay, “The evaluation of network anomaly detection systems: Statistical analysis of the UNSW-NB15 data set and the comparison with the KDD99 data set,” Information Security Journal: A Global Perspective , vol. 25, no. 1-3, pp. 18–31, 2016. [Online]. Available: https://doi.org/10.1080/19393555.2015.1125974

    arXiv 2016

  5. [3]

    Outside the closed world: On using machine learning for network intrusion detection,

    R. Sommer and V . Paxson, “Outside the closed world: On using machine learning for network intrusion detection,” in Proceedings of the IEEE Symposium on Security and Privacy , 2010, pp. 305–316

    work page 2010

  6. [4]

    Anomaly detection: A survey,

    V . Chandola, A. Banerjee, and V . Kumar, “Anomaly detection: A survey,” ACM Computing Surveys , vol. 41, no. 3, p. Article 15, 2009. [Online]. Available: https://doi.org/10.1145/1541880.1541882

    arXiv 2009

  7. [6]

    Research on enhancing cloud computing network security using artificial intelligence algorithms,

    Y . Wang and X. Yang, “Research on enhancing cloud computing network security using artificial intelligence algorithms,” 2025. [Online]. Available: https://arxiv.org/abs/2502.17801

    work page arXiv 2025

  8. [7]

    A hybrid deep learning anomaly detection framework for intrusion detection,

    R. Kale, Z. Lu, K. W. Fok, and V . L. L. Thing, “A hybrid deep learning anomaly detection framework for intrusion detection,” in 2022 IEEE 8th Intl Conference on Big Data Security on Cloud (BigDataSecurity), IEEE Intl Conference on High Performance and Smart Computing (HPSC), and IEEE Intl Conference on Intelligent Data and Security (IDS) , 2022, pp. 137–142

    work page 2022

Show all 27 references
  1. [9]

    A machine learning approach for intrusion detection system on NSL-KDD dataset,

    I. Abrar, Z. Ayub, F. Masoodi, and A. M. Bamhdi, “A machine learning approach for intrusion detection system on NSL-KDD dataset,” in 2020 International Conference on Smart Electronics and Communication (ICOSEC), 2020, pp. 919–924

    2020

  2. [10]

    A detailed analysis of the KDD Cup 99 data set,

    M. Tavallaee, E. Bagheri, W. Lu, and A. A. Ghorbani, “A detailed analysis of the KDD Cup 99 data set,” in 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications, 2009, pp. 1–6

    2009

  3. [11]

    A new deep learning based intrusion detection system for cloud security,

    S. Hizal, Ü. Çavu¸ so ˘glu, and D. Akgün, “A new deep learning based intrusion detection system for cloud security,” in 2021 3rd International Congress on Human-Computer Interaction, Optimization and Robotic Applications (HORA), 2021, pp. 1–4

    2021

  4. [12]

    Evaluating supervised learning models for fraud detection: A comparative study of classical and deep architectures on imbalanced transaction data,

    C. Wang, C. Nie, and Y . Liu, “Evaluating supervised learning models for fraud detection: A comparative study of classical and deep architectures on imbalanced transaction data,” 2025. [Online]. Available: https://arxiv. org/abs/2505.22521

    2025

  5. [13]

    Estimating the support of a high-dimensional distribution,

    B. Schölkopf, J. C. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. Williamson, “Estimating the support of a high-dimensional distribution,” Neural Computation, vol. 13, no. 7, pp. 1443–1471, 2001

    2001

  6. [14]

    UltraRE: Enhancing RecEraser for recommendation unlearning via error decomposition,

    Y . Li, C. Chen, Y . Zhang, W. Liu, L. Lyu, X. Zheng, D. Meng, and J. Wang, “UltraRE: Enhancing RecEraser for recommendation unlearning via error decomposition,” in Advances in Neural Information Processing Systems, vol. 36, A. Oh et al., Eds. Curran Associates, Inc., 2023, pp...

    2023

  7. [15]

    SETransformer: A hy- brid attention-based architecture for robust human activity recognition,

    Y . Liu, X. Qin, Y . Gao, X. Li, and C. Feng, “SETransformer: A hy- brid attention-based architecture for robust human activity recognition,”

  8. [16]

    Pruning visual concepts for efficient and interpretable transfer learning,

    Z. Li and Z. Ke, “Pruning visual concepts for efficient and interpretable transfer learning,” in Second Workshop on Visual Concepts , 2025. [Online]. Available: https://openreview.net/forum?id=ADXwXtOlFt

    2025

  9. [17]

    Making users indistinguishable: Attribute-wise unlearning in recommender systems,

    Y . Li, C. Chen, X. Zheng, Y . Zhang, Z. Han, D. Meng, and J. Wang, “Making users indistinguishable: Attribute-wise unlearning in recommender systems,” in Proceedings of the 31st ACM International Conference on Multimedia (MM ’23) , ACM, Oct. 2023, pp. 984–994. [Online]. Avail...

    2023

  10. [18]

    Post-training attribute unlearning in recommender systems,

    C. Chen, Y . Zhang, Y . Li, J. Wang, L. Qi, X. Xu, X. Zheng, and J. Yin, “Post-training attribute unlearning in recommender systems,” 2024. [Online]. Available: https://arxiv.org/abs/2403.06737

    2024 arXiv

  11. [19]

    Robust detection of distributed targets based on Rao test and Wald test,

    S. Sun, J. Liu, W. Liu, and T. Jian, “Robust detection of distributed targets based on Rao test and Wald test,” Signal Processing, vol. 180, pp. 107801, 2021. [Online]. Available: https://doi.org/10.1016/j.sigpro. 2020.107801

    2021

  12. [20]

    MEDIAN: Adaptive Intermediate-grained Aggregation Network for Composed Image Retrieval,

    Q. Huang, Z. Chen, Z. Li, C. Wang, X. Song, Y . Hu, and L. Nie, “MEDIAN: Adaptive Intermediate-grained Aggregation Network for Composed Image Retrieval,” in Proc. IEEE ICASSP , 2025, pp. 1–5

    2025

  13. [21]

    Credit risk analysis for SMEs using graph neural networks in supply chain,

    Z. Zhang, Q. Shen, Z. Hu, Q. Liu, and H. Shen, “Credit risk analysis for SMEs using graph neural networks in supply chain,” arXiv preprint arXiv:2507.07854, 2025. [Online]. Available: https://arxiv.org/abs/2507. 07854

    2025 arXiv

  14. [22]

    DDPM-MoCo: Advancing industrial surface defect generation and detection with generative and contrastive learning,

    Y . He, X. Wang, and T. Shi, “DDPM-MoCo: Advancing industrial surface defect generation and detection with generative and contrastive learning,” in International Joint Conference on Artificial Intelligence (IJCAI), 2024, pp. 34–49

    2024

  15. [23]

    Guarding the Gate: ConceptGuard Battles Concept-Level Backdoors in Concept Bottleneck Models,

    S. Lai, Y . Huang, J. Yang, G. Huang, W. Chen, and Y . Yue, “Guarding the Gate: ConceptGuard Battles Concept-Level Backdoors in Concept Bottleneck Models,” arXiv preprint arXiv:2411.16512 , 2024. [Online]. Available: https://arxiv.org/abs/2411.16512

    2024 arXiv

  16. [24]

    Drive: Dependable Robust Interpretable Visionary Ensemble Framework in Autonomous Driving,

    S. Lai, T. Xue, H. Xiao, L. Hu, J. Wu, N. Feng, R. Guan, H. Liao, Z. Li, and Y . Yue, “Drive: Dependable Robust Interpretable Visionary Ensemble Framework in Autonomous Driving,”arXiv preprint arXiv:2409.10330, 2024. [Online]. Available:

    2024 arXiv

  17. [25]

    Large language models as topological structure enhancers for text-attributed graphs,

    S. Sun, Y . Ren, C. Ma, and X. Zhang, “Large language models as topological structure enhancers for text-attributed graphs,”arXiv preprint arXiv:2311.14324, 2023. [Online]. Available: https://arxiv.org/abs/2311. 14324

    2023 arXiv

  18. [26]

    Research on brand strategy of hotel enterprises—taking Hyatt Hotel Group as an example,

    J. Song, K. Ding, R. Cheng, X. Zhao, X. Luo, Y . Liu, Y . Tian, and Z. Duan, “Research on brand strategy of hotel enterprises—taking Hyatt Hotel Group as an example,” Open Journal of Business and Management, vol. 13, no. 2, pp. 861–869, 2025

    2025

  19. [2025]

    Available: https://arxiv.org/abs/2505.19369

    [Online]. Available: https://arxiv.org/abs/2505.19369

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.