Pith. sign in

REVIEW 4 major objections 5 minor 28 references

An AutoML-based approach for Network Intrusion Detection

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read An AutoML-generated stacked ensemble reaches 90% accuracy and 89% F1 on binary NSL-KDD intrusion detection, outperforming individual Random Forest, XGBoost, CatBoost, and LightGBM models.

desk verdict Routine MLJAR AutoML run on NSL-KDD with a central evaluation flaw: weighted binary metrics can hide near-zero detection of rare attack classes. read the letter →

arxiv 2411.15920 v1 pith:SFI72VSG submitted 2024-11-24 cs.LG

classification cs.LG
keywords networkintrusiondetectionAutoMLstackedensembleMLJARNSL-KDDdatasetcybersecuritygradientboostingbinaryclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that AutoML can replace the manual pipeline of feature engineering, model selection, and hyperparameter tuning in network intrusion detection. Using the MLJAR AutoML framework on the NSL-KDD dataset, it builds a stacked ensemble of LightGBM, CatBoost, and XGBoost that reaches 90% accuracy and 89% F1 on a binary normal-versus-attack test set. This beats the same authors' Random Forest, XGBoost, CatBoost, and LightGBM models, which score between 78% and 80% accuracy. If the result holds, automated model stacking would offer a practical route to more accurate and lower-maintenance intrusion detection systems.

What carries the argument

The central mechanism is the stacked ensemble generated by the MLJAR AutoML framework, an automated machine learning tool for tabular data that handles feature engineering, model selection, and hyperparameter tuning. The stack combines LightGBM, CatBoost, and XGBoost variants with strategically calibrated weights, and the stacking layer is what carries the claimed improvement: it balances the individual models' strengths and offsets their limitations, yielding higher accuracy and fewer false positives than any single model.

What would settle it

Run the stacked ensemble on NSL-KDDTest+, keep the five original classes, and inspect per-class precision and recall for the rare U2R and R2L classes, which have only 200 and 2,754 test examples respectively. If recall on these classes is near zero while aggregate accuracy stays at 90%, the claim that the model minimizes false positives and works well for the full intrusion detection task would not hold.

Watch

Extended reading notes

Core claim

The central discovery is that a Stacked Ensemble model produced automatically by the MLJAR AutoML framework outperforms every individual classifier it is built from. Trained and tested on NSL-KDD collapsed to binary Normal/Attack, the stacked model achieves 90% accuracy, 90% precision, 89% recall, and 89% F1-score, compared with 78% accuracy for Random Forest and LightGBM and 80% for XGBoost and CatBoost. The authors attribute the gain to ensemble stacking, which combines the strengths of the constituent algorithms, together with AutoML's automated feature engineering and hyperparameter optimization. The paper presents this as evidence that AutoML-based approaches can deliver both higher detection accuracy and reduced false positives while removing much of the manual overhead of traditional machine learning workflows.

Load-bearing premise

The central assumption is that the binary NSL-KDD benchmark, evaluated with aggregate weighted metrics, faithfully represents how well the model would detect intrusions in practice, including rare attack types.

Editorial extensions

If this is right

  • An AutoML-driven stacked ensemble can be produced without manual feature selection, lowering the barrier to deploying machine-learned intrusion detection.
  • The 90% accuracy and 89% F1 reported for the stacked ensemble make it a stronger candidate than any single gradient-boosting model tested on the same binary NSL-KDD task.
  • Automated retraining and hyperparameter tuning, which the paper cites as properties of the AutoML workflow, are what make the model adaptable to changing traffic patterns over time.
  • Reducing false positives at the aggregate level means security operators would see fewer false alarms, which the paper argues is essential in practice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper reports only aggregate weighted metrics; a per-class breakdown on the five original NSL-KDD classes would test whether the 89% recall is spread across all attack types or concentrated in the dominant Normal and DoS classes.
  • Because NSL-KDD is a 1999-era benchmark, the 10-point gap over single models may not transfer to modern traffic; re-running the same MLJAR pipeline on a recent dataset such as CICIDS2017 would show whether the stacking advantage persists.
  • The same AutoML stacking recipe could plausibly be applied to other tabular security tasks like phishing or malware detection, but the paper provides no evidence on transferability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes an AutoML-based approach to network intrusion detection using the MLJAR framework on the NSL-KDD dataset. The authors train Random Forest, XGBoost, CatBoost, LightGBM, and a stacked ensemble produced by MLJAR, after converting the five-class problem into a binary Normal/Attack classification task. The main reported result, in Table III, is that the stacked ensemble achieves 90% accuracy, 90% precision, 89% recall, and 89% F1 score, outperforming the four individual models. The paper argues that the AutoML workflow reduces manual overhead in feature engineering and model selection, and concludes that the stacked model is more robust and effective for intrusion detection.

Significance. If the reported results are reliable, the paper provides a useful practical demonstration that an off-the-shelf AutoML stack can improve aggregate predictive performance on a standard benchmark compared with a small set of hand-configured baseline models. The paper is transparent about the hyperparameters used and it makes a clear, falsifiable claim about the stacked model's aggregate metrics. However, the evaluation in its current form is not sufficient to support the intrusion-detection-specific conclusions: the binary collapse and support-weighted metrics can conceal poor performance on rare attack classes, no variability or significance information is reported, and the false-positive minimization claim is not directly measured. The significance of the contribution is therefore conditional on additional per-class and statistical evaluation.

major comments (4)
  1. [Section IV-D and Table III] The binary collapse into Normal/Attack and the use of support-weighted metrics can hide poor performance on rare classes. With only 52 U2R and 995 R2L training examples, and 200 U2R and 2,754 R2L test examples (Table I), a model that labels all such instances as Normal can still achieve high weighted accuracy. No per-class confusion matrix or per-class recall for U2R/R2L is provided, so the reported 90% accuracy does not by itself establish that the stacked model is effective for the rare attack types that are central to intrusion detection. Please report a confusion matrix and per-class precision, recall, and F1 for all models, with explicit discussion of U2R and R2L.
  2. [Section V] The statement that the Stacked Ensemble is "effective in minimizing false positives" is not directly supported by the reported metrics. Weighted precision of 90% (Table III) is not a false-positive rate, and with class imbalance it can be dominated by the large Normal class. A low false-positive rate on the rare attack classes is not demonstrated. Please report false-positive rates per class, or a confusion matrix, and ideally ROC/PR curves or a detection-versus-false-alarm trade-off.
  3. [Section IV-D and Table III] All comparisons rest on a single train/test split with no variance estimate. Because the text reports only point estimates, the 10-point gap between the Stacked Ensemble and XGBoost/CatBoost cannot be distinguished from split-to-split variability. Please provide cross-validated metrics with standard deviations, or repeated independent runs with a significance test such as McNemar's test or a paired bootstrap.
  4. [Sections II and V] The abstract and conclusions state that the approach outperforms "traditional ML-based approaches," but the only baselines are the authors' own RF, XGBoost, CatBoost, and LightGBM implementations. No comparison with published NSL-KDD results is given, so the broader claim of outperforming conventional approaches is not established. Please either add literature baselines evaluated under the same protocol or soften the claim to "outperforms the individual models trained in this study."
minor comments (5)
  1. [Section IV-A, Table I] The text says the test set has 22,544 records, but Table I sums to 22,543; please correct the inconsistency.
  2. [Section IV-C, Eq. (12)] Equation (12) uses |yi| while Eqs. (10) and (11) use |Yi| for class support; the notation should be unified.
  3. [Sections I and IV-D] There are several typographical errors: "rely heavily depend" in Section I, "NV IDIA" in Section IV-D, and "Fg. 2" in Section V. These should be corrected.
  4. [Section IV-B] The hyperparameter search budget (number of Bayesian optimization iterations) and the exact MLJAR version are not reported; adding these details would improve reproducibility.
  5. [Section III] The stacking procedure is described only as "strategically calibrated weights" with no detail on the meta-learner or the exact set of base models used by MLJAR; please state the MLJAR configuration and any relevant defaults.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all claimed results are measured benchmark scores on a held-out test split.

full rationale

The paper reports an empirical comparison of AutoML-driven stacked ensemble and individual models on the NSL-KDD dataset. The central claim is that the Stacked Ensemble achieves 90% accuracy and 89% F1 score on the test set, outperforming Random Forest, XGBoost, CatBoost, and LightGBM. This is a measured outcome, not a derived or predicted quantity that reduces to its inputs by construction. Hyperparameters are tuned via Bayesian optimization with 10-fold cross-validation on the training portion (Section IV-B), and the reported metrics are evaluated on the separate NSL-KDDTest+ set (Section IV-D). No fitted parameter is renamed as a prediction, and no equation defines the claimed result in terms of itself. The definitions of Accuracy, Precision, Recall, and F1-score in Section IV-C are standard evaluation formulas applied to actual classifications, not tautological inputs. The only self-referential element is the citation of the MLJAR AutoML framework as the tool used, which is external software, not a prior result by the same authors invoked to forbid alternatives. Concerns about binary class collapse, weighted metrics, and lack of per-class results are legitimate evaluation-design and correctness risks, but they are not circularity: the reported numbers would still be empirical measurements even if they were misleading for the intrusion-detection task. Therefore, the derivation chain is self-contained and the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper's central numerical claim rests on the NSL-KDD benchmark, a binary normal/attack reformulation, tuned hyperparameters, and the MLJAR AutoML pipeline. None of these are introduced by the paper; all are borrowed from the prior literature or tool defaults. The paper introduces no new entities or derived parameters.

free parameters (5)
  • Random Forest hyperparameters = max_features=0.5; min_samples_split=20; max_depth=4
    Chosen by Bayesian optimization with 10-fold cross-validation on NSL-KDD training data; the reported Random Forest accuracy (78%) depends on these choices.
  • XGBoost hyperparameters = learning_rate=0.075; max_depth=8; min_child_weight=5; subsample=1.0; colsample_bytree=1.0
    Tuned on the NSL-KDD training set; the reported XGBoost accuracy (80%) depends on these choices.
  • CatBoost hyperparameters = learning_rate=0.05; depth=8; rsm=0.8
    Tuned on the NSL-KDD training set; the reported CatBoost accuracy (80%) depends on these choices.
  • LightGBM hyperparameters = learning_rate=0.05; num_leaves=63; feature_fraction=0.9; bagging_fraction=0.9; min_data_in_leaf=5
    Tuned on the NSL-KDD training set; the reported LightGBM accuracy (78%) depends on these choices.
  • Stacked ensemble combination weights = not reported
    MLJAR calibrates the weights of LightGBM, CatBoost, and XGBoost in the stacked ensemble (Section III); the actual weights are not given, so the ensemble result cannot be reconstructed from the paper.
assumptions (4)
  • domain assumption NSL-KDD is a representative benchmark for network intrusion detection
    Section IV-A treats NSL-KDD as a popular benchmark; if it is unrepresentative of modern network traffic, the measured performance does not transfer to real deployments.
  • domain assumption The binary normal/attack collapse preserves the intrusion detection task
    Section IV-D converts the five-class task into binary; no per-class metrics are reported for rare U2R and R2L classes, so the aggregate may hide failures on minority attacks.
  • domain assumption The test split was not used in AutoML model selection or hyperparameter tuning
    Section IV-B describes tuning with 10-fold cross-validation on the training set and Section IV-D uses NSL-KDDTest+ for evaluation, but no leakage checks or random seeds are reported; violation would invalidate the results.
  • domain assumption MLJAR's automated preprocessing and stacking behave as documented
    Section III relies on MLJAR for feature engineering and ensemble construction; no code or version pinning is provided, so the pipeline cannot be independently checked.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An AutoML-based approach for Network Intrusion Detection." pith.science (2026). https://pith.science/paper/SFI72VSG

@misc{pith2026241115920,
  author       = {Pith},
  title        = {Pith review of: An AutoML-based approach for Network Intrusion Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SFI72VSG}},
  note         = {Machine review of arXiv:2411.15920}
}
read the original abstract

In this paper, we present an automated machine learning (AutoML) approach for network intrusion detection, leveraging a stacked ensemble model developed using the MLJAR AutoML framework. Our methodology combines multiple machine learning algorithms, including LightGBM, CatBoost, and XGBoost, to enhance detection accuracy and robustness. By automating model selection, feature engineering, and hyperparameter tuning, our approach reduces the manual overhead typically associated with traditional machine learning methods. Extensive experimentation on the NSL-KDD dataset demonstrates that the stacked ensemble model outperforms individual models, achieving high accuracy and minimizing false positives. Our findings underscore the benefits of using AutoML for network intrusion detection, as the AutoML-driven stacked ensemble achieved the highest performance with 90\% accuracy and an 89\% F1 score, outperforming individual models like Random Forest (78\% accuracy, 78\% F1 score), XGBoost and CatBoost (both 80\% accuracy, 80\% F1 score), and LightGBM (78\% accuracy, 78\% F1 score), providing a more adaptable and efficient solution for network security applications.

Figures

Figures reproduced from arXiv: 2411.15920 by the authors.

Figure 1
Figure 1. A comparative overview of traditional ML and AutoML approaches. Traditional ML requires manual steps like data preprocessing, feature [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Comparison of accuracy and F1 score across individual models [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 2
Figure 2. Comparison of performance metrics (Accuracy, Precision, Recall, [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 27 canonical work pages

  1. [1]

    Network intrusion detection for iot security based on learning tech- niques,

    N. Chaabouni, M. Mosbah, A. Zemmari, C. Sauvignac, and P. Faruki, “Network intrusion detection for iot security based on learning tech- niques,” IEEE Communications Surveys & Tutorials , vol. 21, no. 3, pp. 2671–2701, 2019

  2. [2]

    Cybercrime To Cost The World $10.5 Trillion Annually By 2025 — cybersecurityventures.com,

    cybercrimemag, “Cybercrime To Cost The World $10.5 Trillion Annually By 2025 — cybersecurityventures.com,” https:// cybersecurityventures.com/hackerpocalypse-cybercrime-report-2016/, [Accessed 12-10-2024]

  3. [3]

    Cost benefits of using machine learning features in nids for cyber security in uk small medium enterprises (sme),

    N. Rawindaran, A. Jayal, E. Prakash, and C. Hewage, “Cost benefits of using machine learning features in nids for cyber security in uk small medium enterprises (sme),” Future Internet , vol. 13, no. 8, p. 186, 2021

  4. [4]

    Cost of a data breach 2024 | IBM — ibm.com,

    “Cost of a data breach 2024 | IBM — ibm.com,” https://www.ibm. com/reports/data-breach, [Accessed 12-10-2024]

  5. [5]

    Com- parative research on network intrusion detection methods based on machine learning,

    C. Zhang, D. Jia, L. Wang, W. Wang, F. Liu, and A. Yang, “Com- parative research on network intrusion detection methods based on machine learning,” Computers & Security , vol. 121, p. 102861, 2022

  6. [6]

    A signature-based intrusion detection system for the internet of things,

    P. Ioulianou, V . Vasilakis, I. Moscholios, and M. Logothetis, “A signature-based intrusion detection system for the internet of things,” Information and Communication Technology F orm , 2018

  7. [7]

    Network intrusion detection system: A machine learning approach,

    M. Panda, A. Abraham, S. Das, and M. R. Patra, “Network intrusion detection system: A machine learning approach,” Intelligent Decision Technologies, vol. 5, no. 4, pp. 347–356, 2011

  8. [8]

    A new intrusion detection sys- tem based on knn classification algorithm in wireless sensor network,

    W. Li, P. Yi, Y . Wu, L. Pan, and J. Li, “A new intrusion detection sys- tem based on knn classification algorithm in wireless sensor network,” Journal of Electrical and Computer Engineering , vol. 2014, no. 1, p. 240217, 2014

Show all 28 references
  1. [9]

    Network intrusion detection system: a survey on artificial intelligence-based techniques,

    M. S. Habeeb and T. R. Babu, “Network intrusion detection system: a survey on artificial intelligence-based techniques,” Expert Systems , vol. 39, no. 9, p. e13066, 2022

  2. [10]

    The Untold Story of NotPetya, the Most Devastating Cyberattack in History — wired.com,

    A. Greenberg, “The Untold Story of NotPetya, the Most Devastating Cyberattack in History — wired.com,” https://www.wired.com/ story/notpetya-cyberattack-ukraine-russia-code-crashed-the-world/, [Accessed 12-10-2024]

  3. [11]

    Ad- vancing cybersecurity: a comprehensive review of ai-driven detection techniques,

    A. H. Salem, S. M. Azzam, O. Emam, and A. A. Abohany, “Ad- vancing cybersecurity: a comprehensive review of ai-driven detection techniques,” Journal of Big Data , vol. 11, no. 1, p. 105, 2024

  4. [12]

    Survey on intrusion detection system using machine learning techniques,

    S. K. Wagh, V . K. Pachghare, and S. R. Kolhe, “Survey on intrusion detection system using machine learning techniques,” International Journal of Computer Applications , vol. 78, no. 16, pp. 30–37, 2013

  5. [13]

    A novel ensembled technique for anomaly detection,

    S. Garg and S. Batra, “A novel ensembled technique for anomaly detection,” International Journal of Communication Systems , vol. 30, no. 11, p. e3248, 2017

  6. [14]

    Automl accurately predicts endovascular mechanical thrombectomy in acute large vessel ischemic stroke,

    R. Raj, S. K. Kannath, J. Mathew, and P. Sylaja, “Automl accurately predicts endovascular mechanical thrombectomy in acute large vessel ischemic stroke,” Frontiers in Neurology , vol. 14, p. 1259958, 2023

  7. [15]

    Automl: A systematic review on automated machine learning with neural architecture search,

    I. Salehin, M. S. Islam, P. Saha, S. Noman, A. Tuni, M. M. Hasan, and M. A. Baten, “Automl: A systematic review on automated machine learning with neural architecture search,” Journal of Information and Intelligence, vol. 2, no. 1, pp. 52–81, 2024

  8. [16]

    Hutter, L

    F. Hutter, L. Kotthoff, and J. Vanschoren, Automated machine learn- ing: methods, systems, challenges . Springer Nature, 2019

  9. [17]

    Mljar: State-of-the-art automated machine learning framework for tabular data. version 0.10.3,

    A. Pło ´nska and P. Pło´nski, “Mljar: State-of-the-art automated machine learning framework for tabular data. version 0.10.3,” Łapy, Poland,

  10. [18]

    A machine learning-based intrusion de- tection system for securing remote desktop connections to electronic flight bag servers,

    R. Bitton and A. Shabtai, “A machine learning-based intrusion de- tection system for securing remote desktop connections to electronic flight bag servers,” IEEE Transactions on Dependable and Secure Computing, vol. 18, no. 3, pp. 1164–1181, 2019

  11. [19]

    Network anomaly detection using exponential random graph models and autoregressive moving average,

    M. Tsikerdekis, S. Waldron, and A. Emanuelson, “Network anomaly detection using exponential random graph models and autoregressive moving average,” IEEE Access , vol. 9, pp. 134 530–134 542, 2021

  12. [20]

    A detection method for anomaly flow in software defined network,

    H. Peng, Z. Sun, X. Zhao, S. Tan, and Z. Sun, “A detection method for anomaly flow in software defined network,” IEEE Access , vol. 6, pp. 27 809–27 817, 2018

  13. [21]

    A novel model for anomaly detection in network traffic based on kernel support vector machine,

    Q. Ma, C. Sun, B. Cui, and X. Jin, “A novel model for anomaly detection in network traffic based on kernel support vector machine,” Computers & Security , vol. 104, p. 102215, 2021

  14. [22]

    Group-wise principal component analysis for exploratory intrusion detection,

    J. Camacho, R. Therón, J. M. García-Giménez, G. Maciá-Fernández, and P. García-Teodoro, “Group-wise principal component analysis for exploratory intrusion detection,” IEEE Access , vol. 7, pp. 113 081– 113 093, 2019

  15. [23]

    A lightweight supervised intrusion detection mechanism for iot networks,

    S. Roy, J. Li, B.-J. Choi, and Y . Bai, “A lightweight supervised intrusion detection mechanism for iot networks,” Future Generation Computer Systems , vol. 127, pp. 276–285, 2022

  16. [24]

    Dfaid: Density-aware and feature-deviated active intrusion detection over network traffic streams,

    B. Li, Y . Wang, K. Xu, L. Cheng, and Z. Qin, “Dfaid: Density-aware and feature-deviated active intrusion detection over network traffic streams,” Computers & Security , vol. 118, p. 102719, 2022

  17. [25]

    A study on nsl-kdd dataset for intrusion detection system based on classification algorithms,

    L. Dhanabal and S. Shantharajah, “A study on nsl-kdd dataset for intrusion detection system based on classification algorithms,” Inter- national journal of advanced research in computer and communication engineering, vol. 4, no. 6, pp. 446–452, 2015

  18. [26]

    Exploring discrepancies in findings obtained with the kdd cup’99 data set,

    V . Engen, J. Vincent, and K. Phalp, “Exploring discrepancies in findings obtained with the kdd cup’99 data set,” Intelligent Data Analysis, vol. 15, no. 2, pp. 251–276, 2011

  19. [27]

    Machine learning for network intrusion detection—a comparative study,

    M. Al Lail, A. Garcia, and S. Olivo, “Machine learning for network intrusion detection—a comparative study,” Future Internet , vol. 15, no. 7, p. 243, 2023

  20. [2021]

    Available: https://github.com/mljar/mljar-supervised

    [Online]. Available: https://github.com/mljar/mljar-supervised

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.