{"id":"76c04ca7-0830-4628-933a-765954f2a46a","arxiv_id":"2412.02845","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Tuning Random Forest with GridSearchCV on the BoTNeTIoT-L01 dataset produces 99.39% test accuracy, with no new detection method or reproducible pipeline.","lead":"This paper applies five standard machine learning classifiers with grid-search tuning to a public IoT intrusion detection dataset and reports that Random Forest reaches 99.39% accuracy. The result is a routine benchmark, not a new method, and the paper does not provide code or a clear feature-selection strategy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 99.39% accuracy rests on an 80/20 random split of network-flow data (Section III-A); without a temporal or session-aware split, the headline number may reflect memorized flow/session patterns rather than generalization to unseen attacks.","rationale":"I read the paper as a benchmark claim: RF with the listed hyperparameters reaches 99.39% on BoTNeTIoT-L01, and this is better than prior IoT IDS results. The most load-bearing assumption is that the 20% test split represents independent, unseen traffic. Network-flow datasets are dominated by temporally and spatially correlated flows; an unstated random split can leak session information. Section III-A gives only '80-20 ratio', and Section IV-A's confusion-matrix total (123,126) does not match 20% of the stated dataset size, so the exact test split cannot be reconstructed. The companion novelty claim ('no other study has reached this level') is independently unsupported because Table III compares different datasets, but that is secondary to the accuracy figure itself. I agree with the reader's weakest assumption. A single, decisive check is to re-run with a time- or session-aware split; the outcome determines whether the central claim survives. Until then, CONDITIONAL is the right verdict, so I recommend no change.","tokens_in":6892,"tokens_out":7098,"duration_ms":71499,"concrete_test":"Obtain BoTNeTIoT-L01 (with its timestamps/flow metadata) and rerun the RF model with the reported hyperparameters (criterion=gini, max_depth=8, max_features=sqrt, n_estimators=200) under three splits: original random 80/20, time-ordered (first 80% of timestamps train, last 20% test), and session-aware group split (all flows sharing a source IP or attack session confined to one side). Compare accuracy and F1. If either non-random split drops accuracy by more than roughly one percentage point, or F1 drops materially, the reported 99.39% is inflated by leakage; if the numbers are stable, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the test split being an honest sample of unseen traffic. Section III-A specifies only an '80-20 ratio' split. BoTNeTIoT-L01 is a network-flow dataset; attack sessions generate many correlated flows, and random splitting can place flows from the same session or same 5-tuple on both sides of the train/test boundary. That would let the classifier memorize addresses/ports rather than detect novel attacks. No timestamp-ordered split, session-aware grouping, or code is provided to rule this out. The test-set size is also ambiguous: the confusion matrix in Section IV-A sums to 123,126 rows, whereas 20% of the stated 820,834 rows is ~164,167. This reinforces that the provenance of the test set is not established. If leakage is present, the 99.39% accuracy is not a valid estimate of performance on unseen traffic, so the paper's central claim fails.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical comparison of five machine-learning classifiers (Random Forest, Decision Tree, K-Nearest Neighbors, Gradient Boosting, and AdaBoost) on the BoTNeTIoT-L01 IoT intrusion-detection dataset. The authors state that the data were split 80/20 for training/testing, that GridSearchCV with five-fold cross-validation was used for hyperparameter tuning, and that Random Forest achieved the highest accuracy of 99.39% with an F1 score of 0.994. The paper includes confusion matrices, per-model precision/recall/F1 tables, ROC AUC values, and a comparison table against four prior studies. The central claim is that this accuracy is higher than all previous IoT intrusion-detection results.","tokens_in":7086,"tokens_out":6370,"duration_ms":62193,"significance":"If the empirical result were properly established, it would be a modest benchmark result: a tuned Random Forest can reach very high accuracy on this particular IoT flow dataset. The paper has some strengths: the hyperparameters of all five models are reported, the standard evaluation metrics are defined, and the confusion-matrix totals are internally consistent (each sums to 123,126 rows). However, the evaluation protocol is not documented to a level that supports the central generalization claim. There is no temporal or session-aware split, no repeated-run variance or error bars, an unexplained discrepancy between the reported test-set size and the stated 20% of the dataset, and the 'no other study has reached this level' claim is not supported by the evidence presented. The paper does not define the 'novel component selection and extraction strategy' promised in the abstract. As presented, the contribution is a benchmark with unresolved correctness risks rather than a citable research result.","major_comments":[{"comment":"The evaluation protocol does not establish that the test set represents unseen traffic. The paper states only that the BoTNeTIoT-L01 dataset has 820,834 rows and was split into an 80-20 ratio for training and testing, with no mention of temporal ordering, session-aware grouping, or stratification. Because BoTNeTIoT-L01 is a network-flow dataset, a random split can place flows from the same attack session or the same 5-tuple on both sides of the train/test boundary, allowing the classifier to memorize addresses and ports rather than detect novel attacks. The test-set size also does not match the stated split: the confusion matrices in Section IV sum to 123,126 rows, whereas 20% of 820,834 is approximately 164,167. The paper does not report any preprocessing, deduplication, or filtering that would explain this discrepancy. This directly affects the validity of the headline 99.39% accuracy figure.","section":"Section III-A and Section IV-A"},{"comment":"Table I is inconsistent with the confusion matrices for Random Forest and Decision Tree. For Random Forest, the confusion matrix in Section IV-A gives precision = 59,268/(59,268+610) ≈ 0.990 and recall = 59,268/(59,268+143) ≈ 0.998, but Table I reports precision 0.997 and recall 0.989. For Decision Tree, Section IV-B gives precision ≈ 0.993 and recall ≈ 0.991, but Table I reports precision 0.991 and recall 0.993. The two rows appear to have precision and recall swapped. This error changes the per-metric ranking of the models and must be corrected.","section":"Section IV-D versus Section IV-A/B"},{"comment":"The claim that 'no other study has been able to reach this level of accuracy for IoT intrusion detection' is not supported by the evidence. Table III compares only four prior studies, and most of them use different datasets (for example, [4] uses NF-UNSW-NB15-v2). The table provides no confidence intervals, no paired statistical tests, and no repeated-run variance. The comparison therefore cannot justify the statement that the proposed model outperforms all existing work. This claim should be removed or replaced with a dataset-matched comparison with appropriate uncertainty quantification.","section":"Section II and Section IV-E"},{"comment":"The abstract states that the paper 'proposes and implements a novel component selection and extraction strategy (our strategy)', but no such feature-selection or feature-extraction strategy is described anywhere in the manuscript. Section III describes the dataset and the five classifiers but does not present a feature-selection method. The claimed novelty is therefore unsubstantiated. The authors should either describe the strategy in detail or remove the claim.","section":"Abstract and Section I"},{"comment":"The report of AUC = 1.00 for Random Forest is inconsistent with the confusion matrix. The confusion matrix in Section IV-A contains 753 misclassifications (610 false positives and 143 false negatives). With any false positive or false negative on the test set, the ROC curve cannot reach TPR = 1 at FPR = 0, so the AUC must be strictly less than 1. This suggests the AUC values in Table II were computed on a different set (perhaps the training set) or are incorrectly reported. All AUC values should be recomputed on the same test set used for the accuracy numbers.","section":"Section IV-A and Table II"}],"minor_comments":[{"comment":"In the Random Forest confusion-matrix paragraph, '610 false negatives' should read '610 false positives'; the text already lists 143 false negatives immediately before.","section":"Section IV-A"},{"comment":"In the Decision Tree confusion-matrix paragraph, '430 false negatives' should read '430 false positives'; the text already lists 520 false negatives.","section":"Section IV-B"},{"comment":"The model name 'Decision Free' appears in both tables and should be 'Decision Tree'.","section":"Tables I and II"},{"comment":"There are repeated spelling errors ('hyperparamter', 'tuinted', 'tuined') and grammatical issues that should be corrected in a final revision.","section":"Throughout"},{"comment":"The paper states that five-fold cross-validation was applied but does not clarify whether it was used for hyperparameter selection, for model evaluation, or both. This should be stated explicitly, together with the exact role of the 80/20 split.","section":"Section III-B"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has the character of a student project report. The core empirical claim is not verifiable as reported because the test-set provenance is unclear and the evaluation protocol lacks a session-aware split. If the authors can provide code and rerun the experiments with a temporal or session-aware split, the accuracy claim could be rehabilitated, but the 'novel strategy' and 'no other study' claims should be removed regardless. The fit with a security journal is marginal; a workshop or short-paper venue with reproducibility requirements would be more appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a routine application of five off-the-shelf classifiers to the BoTNeTIoT-L01 dataset, with hyperparameter tuning. The one thing that would make it interesting—a novel component selection and extraction strategy—is named in the abstract and never described anywhere in the paper. What this leaves is a standard benchmark result whose main number, 99.39% RF accuracy, may be real but is not verifiable from the manuscript.\n\nCredit where due: the paper uses a real public dataset, gives tuned hyperparameters for each model, and reports confusion-matrix totals that are internally consistent with the claimed accuracy (122,373 correct, 753 wrong out of 123,126). That is more than some papers at this level do.\n\nThe soft spots are not minor. First, the 80/20 split is not described as temporal, session-aware, or stratified. For network-flow data, random splitting can put highly correlated flows from the same attack session on both sides of the train/test boundary, so the 99.39% could reflect memorized 5-tuples rather than detection of unseen attacks. The paper gives no timestamp-ordered split and no code to rule this out. Second, the test set size is ambiguous: the confusion matrix sums to 123,126 rows, while 20% of the stated 820,834 rows is about 164,167. That discrepancy needs an explanation. Third, Table III compares this paper's accuracy on BoTNeTIoT-L01 with numbers from other datasets, so it does not support the claim that no other study reached this level. Fourth, the \"novel strategy\" is never defined, so the novelty claim is empty. There are also smaller issues: two numbers in the RF confusion matrix are both labeled false negatives (one is presumably false positives), and the related-work paragraph simply asserts that no other study reached this accuracy.\n\nOn balance, I think the reader's conditional verdict is fair but slightly generous. The accuracy number itself is not contradicted, but the load-bearing assumption about the split is unaddressed and the provenance of the test set is unclear. If the authors release code and re-run with a session-aware split, this could become a usable data point; as it stands, it is not publishable in its current form.\n\nWho is this for? People teaching a machine-learning-for-security course might use it as an example of common pitfalls, but I would not cite it. Recommendation: desk reject, with an invitation to resubmit after fixing the split, releasing code, correcting the confusion-matrix labels, and comparing on the same dataset.","headline":"Routine ML benchmark on a public IoT dataset with an unverifiable 99.39% accuracy headline and a load-bearing data-split problem; the novelty claim is never actually described.","tokens_in":7600,"tokens_out":4087,"would_cite":false,"duration_ms":40841,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A tuned Random Forest classifier achieves 99.39% test accuracy on the BoTNeTIoT-L01 IoT intrusion detection dataset.","keywords":["IoT intrusion detection","Random Forest","BoTNeTIoT-L01","machine learning","hyperparameter tuning","GridSearchCV","network security","classifier comparison"],"falsifier":"Re-run the same Random Forest configuration on BoTNeTIoT-L01 with a time-ordered or session-aware split, keeping the first 80% of flows for training and the last 20% for testing, and compare the resulting accuracy to 99.39%. If it falls materially, the random split was leaking information and the headline result does not measure detection of unseen attacks.","tokens_in":6702,"feed_emoji":"🛡️","tokens_out":5477,"duration_ms":51786,"temperature":0.7,"pith_summary":"This paper sets out to show that a Random Forest classifier, tuned with GridSearchCV and tested on the BoTNeTIoT-L01 IoT intrusion dataset, can identify network attacks with 99.39% accuracy and an F1 score of 0.994. It compares five machine-learning classifiers and reports that Random Forest beats Decision Tree, Gradient Boosting, AdaBoost, and K-Nearest Neighbor, and also beats the prior results it lists. The authors claim no earlier study has reached this accuracy for IoT intrusion detection. A reader should care because a fast, tuned ensemble that reliably flags malicious IoT traffic would be directly useful in guarding networks of connected devices.","feed_headline":"Tuned Random Forest hits 99.39% accuracy on IoT intrusion detection","feed_subtitle":"A tuned Random Forest beats four classifiers and prior models on the BoTNeTIoT-L01 dataset.","key_machinery":"The load-bearing object is a Random Forest ensemble: many decision trees trained on bootstrap samples, each voting on whether a flow is an attack, with the majority vote deciding. The specific configuration—gini splitting, depth 8, sqrt feature sampling, 200 trees—is located by GridSearchCV, and five-fold cross-validation is used to stabilize the estimate. The claim also depends on the BoTNeTIoT-L01 dataset's class balance and its 820,834 flows, over which a random 80/20 split produces 122,373 correct and 753 wrong test predictions for Random Forest.","core_discovery":"The paper's central claim is that the Random Forest classifier with criterion 'gini', max depth 8, sqrt max features, and 200 estimators—obtained through GridSearchCV and assessed with five-fold cross-validation—achieves the highest test accuracy of 99.39% on a balanced 80/20 split of the BoTNeTIoT-L01 dataset, with precision 0.997, recall 0.989, F1 0.994, and AUC 1.00. The same pipeline yields 99.23% for Decision Tree, 97.89% for Gradient Boosting, 95.26% for AdaBoost, and 94.84% for K-Nearest Neighbor. The paper further claims that this Random Forest result surpasses the previously published models it compares against, including a 99.30% Random Forest on a different industrial-IoT dataset.","pith_inferences":["The paper relies on a random row-wise split; a temporal or session-aware split would likely lower the accuracy because attack flows from the same burst would no longer appear on both sides of the training/test boundary. Testing that split is the quickest way to see how much of 99.39% is genuine generalization.","The 'novel component selection and extraction strategy' is not described in the methodology, so an ablation study removing it would clarify whether the accuracy comes from feature engineering or from the tuned Random Forest alone.","Applying the same tuned Random Forest to other public IoT intrusion datasets (for instance newer or less balanced ones) would show whether the 99.39% level transfers beyond BoTNeTIoT-L01."],"forward_implications":["An IDS built on this tuned Random Forest would flag roughly 99 of every 100 attack flows in this dataset while keeping false alarms near zero, an operating point strong enough for deployment screening.","Because GridSearchCV and five-fold cross-validation are standard, the same tuning recipe can be applied directly to other IoT traffic datasets.","The comparison suggests that a tuned classical ensemble can match or beat deep-learning IDS on tabular flow features, which matters for resource-constrained IoT hardware.","The specific 99.39% figure is tied to this dataset and split; reproducing it requires the same hyperparameters, dataset, and test protocol."],"supporting_citations":[{"why":"Supplies the BoTNeTIoT-L01 dataset, the 820,834-flow balanced corpus on which all reported accuracies are measured.","marker":"[11]"},{"why":"Provides the closest prior Random Forest baseline (99.30%) that the paper must beat to support its accuracy claim.","marker":"[4]"},{"why":"Provides a stacked VGG-16 ensemble baseline (98.3%) used in the comparison table.","marker":"[9]"},{"why":"Provides an LSTM baseline (97.7%) used in the comparison table.","marker":"[6]"},{"why":"Provides a hybrid deep-learning baseline (97%) used in the comparison table.","marker":"[10]"}],"fun_headline_variants":["RF hits 99.39% on IoT IDS, beats four rivals","99.39% accuracy: tuned RF tops IoT intrusion detection","Best ML IDS: RF 99.39% on IoT","Tuned RF outperforms four classifiers on IoT IDS","99.39% accuracy: Random Forest wins on IoT IDS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 80/20 random split treats each network flow as independent, so training and test sets are assumed not to contain flows from the same attack session; if they do, the reported 99.39% overstates performance on unseen attacks.","fun_headline_variants_meta":{"raw":{"variants":["RF hits 99.39% on IoT IDS, beats four rivals","99.39% accuracy: tuned RF tops IoT intrusion detection","Best ML IDS: RF 99.39% on IoT","Tuned RF outperforms four classifiers on IoT IDS","99.39% accuracy: Random Forest wins on IoT IDS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000426,"raw_usage":{"total_tokens":2213,"prompt_tokens":1006,"completion_tokens":1207,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":1117}},"tokens_in":622,"tokens_out":1207,"duration_ms":8955,"temperature":1.0,"reasoning_tokens":1117,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:01:31.809922+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same Random Forest configuration on BoTNeTIoT-L01 with a time-ordered or session-aware split, keeping the first 80% of flows for training and the last 20% for testing, and compare the resulting accuracy to 99.39%. If it falls materially, the random split was leaking information and the headline result does not measure detection of unseen attacks.","supporting_citations":[{"cited_title":"IoT Dataset for Intrusion Detection Systems (IDS),","cited_arxiv_id":null,"evidence_quote":"Supplies the BoTNeTIoT-L01 dataset, the 820,834-flow balanced corpus on which all reported accuracies are measured."},{"cited_title":"An ensemble learning based intrusion detection model for industrial IoT security,","cited_arxiv_id":null,"evidence_quote":"Provides the closest prior Random Forest baseline (99.30%) that the paper must beat to support its accuracy claim."},{"cited_title":"Intrusion detection system using feature extraction with machine learning algorithms in IoT,","cited_arxiv_id":null,"evidence_quote":"Provides a stacked VGG-16 ensemble baseline (98.3%) used in the comparison table."},{"cited_title":"Deep learning approach for SDN-enabled intrusion detection system in IoT networks,","cited_arxiv_id":null,"evidence_quote":"Provides an LSTM baseline (97.7%) used in the comparison table."},{"cited_title":"A hybrid deep learning- based intrusion detection system for IoT networks,","cited_arxiv_id":null,"evidence_quote":"Provides a hybrid deep-learning baseline (97%) used in the comparison table."}],"review_version":1}