{"id":"adf7491e-1cd0-4c8d-8e9f-98b9b79197aa","arxiv_id":"2411.15920","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An MLJAR AutoML stacked ensemble achieves 90% accuracy and 89% F1 on binary NSL-KDD intrusion detection, beating four single models in the authors' comparison.","lead":"The authors use an automated machine learning tool, MLJAR, to build a stacked model that classifies network traffic as normal or malicious, reporting 90% accuracy on a standard benchmark. A smart generalist might read this to see whether AutoML can replace manual machine learning workflows for cybersecurity tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 90% accuracy claim rests on weighted aggregate metrics after binary collapse, which can mask near-zero detection of the rare U2R/R2L attack classes; without per-class results, the intrusion-detection claim is unverified.","rationale":"The reader's weakest_assumption correctly identifies the absence of per-class metrics as the most load-bearing gap. Since the task is intrusion detection, and U2R and R2L are exactly the rare attacks that an NID must catch, reporting only support-weighted aggregates after collapsing to binary can hide a complete failure on those classes. The internal comparison among the authors' own models may be correct as far as it goes, but the central claim is framed as a contribution to intrusion detection, not just a ranking on aggregate accuracy. The reader's CONDITIONAL verdict already demands per-class results and additional evidence, so my concern does not change the verdict; it reinforces it. I considered whether the fairness of the baseline comparison (whether the individual models received the same feature engineering and tuning budget as the AutoML stack) is a stronger concern, but the text is ambiguous on that point and the rare-class issue is directly quantified by the class counts in Table I, making it the more concrete and decisive test.","tokens_in":5,"tokens_out":9850,"duration_ms":248077,"concrete_test":"Compute the confusion matrix of the Stacked Ensemble (and each baseline) on NSL-KDD Test+ using the original five class labels before collapsing to binary, and report per-class precision, recall, and F1 for the U2R and R2L classes. If the per-class recall for U2R plus R2L is below 30% while the aggregate accuracy remains 90%, the headline result is an artifact of class imbalance and the claim that the model is effective for intrusion detection should be re-stated accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the reported 90% accuracy, 90% precision, 89% recall, and 89% F1 score are weighted averages computed after collapsing the five NSL-KDD classes into a binary Normal/Attack task (Sections IV-A and IV-D). The training set contains only 52 U2R and 995 R2L examples, and the test set has 200 U2R and 2,754 R2L examples (Table I). Because the reported metrics weight each class by its support, a model that classifies nearly all R2L and U2R instances as Normal can still achieve roughly 87-90% accuracy if it is correct on the much larger Normal, DoS, and Probe classes. The paper provides no per-class confusion matrix, no recall for U2R/R2L, and no evidence that the Stacked Ensemble's aggregate gain reflects genuine intrusion-detection capability rather than dominance on a few easy classes. The claim that the model is 'effective in minimizing false positives' is likewise unsupported: high weighted precision does not establish a low false-positive rate on the rare attack types that are most critical for intrusion detection. This is not merely a comparison gap relative to the literature; it is a missing property of the very result being reported, and it directly affects whether the central claim supports the paper's stated purpose.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an AutoML-based approach to network intrusion detection using the MLJAR framework on the NSL-KDD dataset. The authors train Random Forest, XGBoost, CatBoost, LightGBM, and a stacked ensemble produced by MLJAR, after converting the five-class problem into a binary Normal/Attack classification task. The main reported result, in Table III, is that the stacked ensemble achieves 90% accuracy, 90% precision, 89% recall, and 89% F1 score, outperforming the four individual models. The paper argues that the AutoML workflow reduces manual overhead in feature engineering and model selection, and concludes that the stacked model is more robust and effective for intrusion detection.","tokens_in":8208,"tokens_out":3447,"duration_ms":32759,"significance":"If the reported results are reliable, the paper provides a useful practical demonstration that an off-the-shelf AutoML stack can improve aggregate predictive performance on a standard benchmark compared with a small set of hand-configured baseline models. The paper is transparent about the hyperparameters used and it makes a clear, falsifiable claim about the stacked model's aggregate metrics. However, the evaluation in its current form is not sufficient to support the intrusion-detection-specific conclusions: the binary collapse and support-weighted metrics can conceal poor performance on rare attack classes, no variability or significance information is reported, and the false-positive minimization claim is not directly measured. The significance of the contribution is therefore conditional on additional per-class and statistical evaluation.","major_comments":[{"comment":"The binary collapse into Normal/Attack and the use of support-weighted metrics can hide poor performance on rare classes. With only 52 U2R and 995 R2L training examples, and 200 U2R and 2,754 R2L test examples (Table I), a model that labels all such instances as Normal can still achieve high weighted accuracy. No per-class confusion matrix or per-class recall for U2R/R2L is provided, so the reported 90% accuracy does not by itself establish that the stacked model is effective for the rare attack types that are central to intrusion detection. Please report a confusion matrix and per-class precision, recall, and F1 for all models, with explicit discussion of U2R and R2L.","section":"Section IV-D and Table III"},{"comment":"The statement that the Stacked Ensemble is \"effective in minimizing false positives\" is not directly supported by the reported metrics. Weighted precision of 90% (Table III) is not a false-positive rate, and with class imbalance it can be dominated by the large Normal class. A low false-positive rate on the rare attack classes is not demonstrated. Please report false-positive rates per class, or a confusion matrix, and ideally ROC/PR curves or a detection-versus-false-alarm trade-off.","section":"Section V"},{"comment":"All comparisons rest on a single train/test split with no variance estimate. Because the text reports only point estimates, the 10-point gap between the Stacked Ensemble and XGBoost/CatBoost cannot be distinguished from split-to-split variability. Please provide cross-validated metrics with standard deviations, or repeated independent runs with a significance test such as McNemar's test or a paired bootstrap.","section":"Section IV-D and Table III"},{"comment":"The abstract and conclusions state that the approach outperforms \"traditional ML-based approaches,\" but the only baselines are the authors' own RF, XGBoost, CatBoost, and LightGBM implementations. No comparison with published NSL-KDD results is given, so the broader claim of outperforming conventional approaches is not established. Please either add literature baselines evaluated under the same protocol or soften the claim to \"outperforms the individual models trained in this study.\"","section":"Sections II and V"}],"minor_comments":[{"comment":"The text says the test set has 22,544 records, but Table I sums to 22,543; please correct the inconsistency.","section":"Section IV-A, Table I"},{"comment":"Equation (12) uses |yi| while Eqs. (10) and (11) use |Yi| for class support; the notation should be unified.","section":"Section IV-C, Eq. (12)"},{"comment":"There are several typographical errors: \"rely heavily depend\" in Section I, \"NV IDIA\" in Section IV-D, and \"Fg. 2\" in Section V. These should be corrected.","section":"Sections I and IV-D"},{"comment":"The hyperparameter search budget (number of Bayesian optimization iterations) and the exact MLJAR version are not reported; adding these details would improve reproducibility.","section":"Section IV-B"},{"comment":"The stacking procedure is described only as \"strategically calibrated weights\" with no detail on the meta-learner or the exact set of base models used by MLJAR; please state the MLJAR configuration and any relevant defaults.","section":"Section III"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of an applied machine learning venue and the central empirical claim is not inherently implausible. The main barrier to acceptance is evaluation rigor: the paper needs per-class results, false-positive analysis, and statistical evidence before the intrusion-detection claims can be supported. I would not reject the paper, because the missing elements are obtainable through additional experiments and reporting rather than through a fundamental redesign."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is an applied AutoML paper, not a methods paper. The authors run MLJAR's stacked ensemble on NSL-KDD, report 90% accuracy and 89% F1, and compare against four individual models they trained themselves. That internal comparison looks honest and the numbers in Table III are consistent with each other. The write-up is clear about the binary Normal/Attack collapse and the weighted-average metrics, which is more than many papers do.\n\nWhat is actually new is thin: MLJAR is an existing library, NSL-KDD is a fossils benchmark, and the result is a data point, not a concept. Still, for a practitioner wondering whether AutoML saves time on tabular security data, the comparison between the stacked ensemble and the individual models is a useful existence proof. The paper does not ship code or configuration artifacts, so the exact workflow is not reproducible as-is, but the MLJAR setup is standard enough that a motivated reader could approximate it.\n\nThe soft spot is not the internal ranking; it is the claim that the stacked model is \"effective\" for intrusion detection. Accuracy, precision, recall, and F1 are all reported as weighted averages over the binary collapsed classes. The training set has 52 U2R and 995 R2L examples; the test set has 200 and 2,754. Without a per-class breakdown, a model that misses every rare attack can still score around 89-90% weighted accuracy simply by getting Normal, DoS, and Probe right. The paper gives no confusion matrix, no recall for U2R/R2L, and no evidence that the ensemble's aggregate gain is real detection rather than dominance on easy classes. The \"minimizing false positives\" claim is likewise unsupported: high weighted precision does not tell you the false-positive rate on rare attacks, which is where false positives matter most in NIDS. That is not a missing literature comparison; it is a missing property of the reported result.\n\nThere are also smaller issues: no error bars or repeated runs, no baselines from published NSL-KDD work, and a few typos (\"NV IDIA\", \"the the\"). None of those are fatal; the rare-class problem is the load-bearing one.\n\nMy bottom line: the paper is a decent student-level applied study, but the central claim overreaches the evidence. It deserves a serious referee, because the fix is within reach: add a per-class confusion matrix, report recall on U2R and R2L, add repeated runs with variance, and compare against existing NSL-KDD results. If the authors supply those, the paper becomes a modest but honest reproducibility case study. I would not cite it for the 90% figure, but I might cite it as an AutoML-on-NSL-KDD data point if the revision lands.\n\nRecommendation: send to peer review, but with the per-class requirement made explicit.","headline":"Routine MLJAR AutoML run on NSL-KDD with a central evaluation flaw: weighted binary metrics can hide near-zero detection of rare attack classes.","tokens_in":8783,"tokens_out":2135,"would_cite":false,"duration_ms":20543,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An AutoML-generated stacked ensemble reaches 90% accuracy and 89% F1 on binary NSL-KDD intrusion detection, outperforming individual Random Forest, XGBoost, CatBoost, and LightGBM models.","keywords":["network intrusion detection","AutoML","stacked ensemble","MLJAR","NSL-KDD dataset","cybersecurity","gradient boosting","binary classification"],"falsifier":"Run the stacked ensemble on NSL-KDDTest+, keep the five original classes, and inspect per-class precision and recall for the rare U2R and R2L classes, which have only 200 and 2,754 test examples respectively. If recall on these classes is near zero while aggregate accuracy stays at 90%, the claim that the model minimizes false positives and works well for the full intrusion detection task would not hold.","tokens_in":7759,"feed_emoji":"🛡️","tokens_out":5392,"duration_ms":45148,"temperature":0.7,"pith_summary":"The paper argues that AutoML can replace the manual pipeline of feature engineering, model selection, and hyperparameter tuning in network intrusion detection. Using the MLJAR AutoML framework on the NSL-KDD dataset, it builds a stacked ensemble of LightGBM, CatBoost, and XGBoost that reaches 90% accuracy and 89% F1 on a binary normal-versus-attack test set. This beats the same authors' Random Forest, XGBoost, CatBoost, and LightGBM models, which score between 78% and 80% accuracy. If the result holds, automated model stacking would offer a practical route to more accurate and lower-maintenance intrusion detection systems.","feed_headline":"AutoML stacked ensemble hits 90% accuracy on NSL-KDD","feed_subtitle":"Automated stacking of three boosting models beats each single model on intrusion detection.","key_machinery":"The central mechanism is the stacked ensemble generated by the MLJAR AutoML framework, an automated machine learning tool for tabular data that handles feature engineering, model selection, and hyperparameter tuning. The stack combines LightGBM, CatBoost, and XGBoost variants with strategically calibrated weights, and the stacking layer is what carries the claimed improvement: it balances the individual models' strengths and offsets their limitations, yielding higher accuracy and fewer false positives than any single model.","core_discovery":"The central discovery is that a Stacked Ensemble model produced automatically by the MLJAR AutoML framework outperforms every individual classifier it is built from. Trained and tested on NSL-KDD collapsed to binary Normal/Attack, the stacked model achieves 90% accuracy, 90% precision, 89% recall, and 89% F1-score, compared with 78% accuracy for Random Forest and LightGBM and 80% for XGBoost and CatBoost. The authors attribute the gain to ensemble stacking, which combines the strengths of the constituent algorithms, together with AutoML's automated feature engineering and hyperparameter optimization. The paper presents this as evidence that AutoML-based approaches can deliver both higher detection accuracy and reduced false positives while removing much of the manual overhead of traditional machine learning workflows.","pith_inferences":["The paper reports only aggregate weighted metrics; a per-class breakdown on the five original NSL-KDD classes would test whether the 89% recall is spread across all attack types or concentrated in the dominant Normal and DoS classes.","Because NSL-KDD is a 1999-era benchmark, the 10-point gap over single models may not transfer to modern traffic; re-running the same MLJAR pipeline on a recent dataset such as CICIDS2017 would show whether the stacking advantage persists.","The same AutoML stacking recipe could plausibly be applied to other tabular security tasks like phishing or malware detection, but the paper provides no evidence on transferability."],"forward_implications":["An AutoML-driven stacked ensemble can be produced without manual feature selection, lowering the barrier to deploying machine-learned intrusion detection.","The 90% accuracy and 89% F1 reported for the stacked ensemble make it a stronger candidate than any single gradient-boosting model tested on the same binary NSL-KDD task.","Automated retraining and hyperparameter tuning, which the paper cites as properties of the AutoML workflow, are what make the model adaptable to changing traffic patterns over time.","Reducing false positives at the aggregate level means security operators would see fewer false alarms, which the paper argues is essential in practice."],"supporting_citations":[{"why":"Supplies the NSL-KDD training and test sets used for all experiments.","marker":"[25]"},{"why":"Provides the MLJAR AutoML framework that generates the stacked ensemble.","marker":"[17]"},{"why":"Supplies the evaluation metrics, including weighted precision, recall, and F1-score.","marker":"[27]"}],"fun_headline_variants":["AutoML stack hits 90% accuracy on NSL-KDD, beats single models","Stacked AutoML outperforms individual models on intrusion detection","90% accuracy: AutoML ensemble tops Random Forest, XGBoost, CatBoost","Automated stacking wins on NSL-KDD with 90% accuracy over solo models","AutoML stacked ensemble outdoes alone classifiers on network ID"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that the binary NSL-KDD benchmark, evaluated with aggregate weighted metrics, faithfully represents how well the model would detect intrusions in practice, including rare attack types.","fun_headline_variants_meta":{"raw":{"variants":["AutoML stack hits 90% accuracy on NSL-KDD, beats single models","Stacked AutoML outperforms individual models on intrusion detection","90% accuracy: AutoML ensemble tops Random Forest, XGBoost, CatBoost","Automated stacking wins on NSL-KDD with 90% accuracy over solo models","AutoML stacked ensemble outdoes alone classifiers on network ID"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1369,"prompt_tokens":893,"completion_tokens":476,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":376}},"tokens_in":509,"tokens_out":476,"duration_ms":4385,"temperature":1.0,"reasoning_tokens":376,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:43:52.014401+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the stacked ensemble on NSL-KDDTest+, keep the five original classes, and inspect per-class precision and recall for the rare U2R and R2L classes, which have only 200 and 2,754 test examples respectively. If recall on these classes is near zero while aggregate accuracy stays at 90%, the claim that the model minimizes false positives and works well for the full intrusion detection task would not hold.","supporting_citations":[{"cited_title":"Mljar: State-of-the-art automated machine learning framework for tabular data. version 0.10.3,","cited_arxiv_id":null,"evidence_quote":"Provides the MLJAR AutoML framework that generates the stacked ensemble."},{"cited_title":"Machine learning for network intrusion detection—a comparative study,","cited_arxiv_id":null,"evidence_quote":"Supplies the evaluation metrics, including weighted precision, recall, and F1-score."}],"review_version":1}