{"id":"17acf9fb-7cd0-4359-bde5-6c8cbc2b7661","arxiv_id":"2412.05688","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A random forest model with genetic-algorithm hyperparameter tuning reports high accuracy on three public botnet datasets, but the evaluation relies on imbalanced accuracy and lacks independent validation.","lead":"This paper applies machine learning classifiers, tuned with a genetic algorithm, to detect botnet traffic in three public network datasets. The authors report 99.85% average accuracy for a random forest model and describe a prototype detection software.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 0% FPR against CTU-Normal-7/12/22/31 is only meaningful if those legitimate-flow files are disjoint from the CTU-13 flows used to train the RF-GA models; the paper never establishes this, so the central performance claim is not supported.","rationale":"The reader's REJECT verdict is justified, and the weakest point is exactly the FPR evidence. The abstract's accuracy/F1 numbers are internal cross-validation estimates and are themselves debatable because they are unweighted averages over 15 scenario/dataset models and lack error bars. But the '0% FPR' sentence in the Conclusion is a separate, stronger claim that can only be true for genuinely unseen normal traffic. Since the paper provides no code, no data release, and no provenance check, the disjointness of the CTU-Normal files from the CTU-13 training scenarios is an unverified assumption on which the deployment claim rests. If that assumption fails, the central 'near-perfect and no false alarms' result is not an external-validation result at all. The concrete test I propose settles this by de-duplicating flows and retraining. This does not change the reader's verdict; it strengthens it.","tokens_in":26191,"tokens_out":11773,"duration_ms":116524,"concrete_test":"Obtain the CTU-13 scenario .binetflow files and the CTU-Normal-7/12/22/31 files. Compute a canonical flow key (rounded StartTime, Proto, SrcAddr, DstAddr, Sport, Dport, TotPkts, TotBytes) for every normal/background flow in the 13 CTU-13 scenarios and for every flow in the four CTU-Normal files; test whether any CTU-Normal flow key appears in any training scenario. Independently, compare capture time ranges and source IP sets to detect host/session overlap. Then retrain the RF-GA models on the CTU-13 scenarios with any overlapping flows removed and rerun the Table XXVIII FPR measurement. If the FPR is still 0, the concern fails to land; if it becomes positive, the published 0% FPR is an artefact of test-train overlap. The same rerun should be reported per model and under the OR-of-models alert rule used in the product.","verdict_should_be":"REJECT","load_bearing_attack":"The headline result depends on three quantities: 99.85% average accuracy, 97.74% average F1, and 0% FPR on known legitimate traffic. The first two are internal cross-validation numbers; the last is the only out-of-training-distribution result and is the main support for the paper's deployment claim. Section XIII, Table XXVIII computes FPR for the RF-GA CTU-13 models against CTU-Normal-7/12/22/31. For that comparison to be valid, those normal captures must not contain flows (same capture sessions, hosts, or flow records) that were used as normal/background training data in the 13 CTU-13 scenarios. The paper does not give capture timestamps, host lists, or a de-duplication check for the CTU-Normal files, and it does not state whether the normal/background class of any CTU-13 scenario includes traffic from those files. The naming 'CTU-Normal-7'/'CTU-Normal-12' is particularly problematic because CTU-13 has scenario IDs 7 and 12, so an overlapping origin is plausible on its face. If overlap exists, the zero FPR is a leakage artefact: the model has already seen those normal flows, so Table XXVIII documents memorisation, not false-positive behaviour on new legitimate traffic. The concern is compounded by the product rule in Section XI (F-R-4), where a flow is flagged if any of the 13 models calls it botnet, making a true zero-FPR across more than four million normal flows surprising unless the test flows are in-distribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a flow-based botnet detection system that trains supervised machine learning classifiers on the CTU-13, ISOT 2010, and ISCX 2014 datasets. After reducing each dataset to the top 15 features by Random Forest importance, the authors tune hyperparameters with a genetic algorithm (GA) and report that Random Forest with GA achieves an average accuracy of 99.85% and an average F1-score of 97.74% across the three datasets. They further report a 0% false positive rate against the CTU-Normal-7/12/22/31 legitimate traffic datasets when using the 13 CTU-13-trained models, and they present a software prototype implementing the detection engine.","tokens_in":26523,"tokens_out":5325,"duration_ms":48655,"significance":"If the reported performance were sound, the paper would make a meaningful contribution: it would demonstrate that flow-based features, without IP addresses, ports, or payload, can detect botnet traffic with near-perfect accuracy and zero false alarms on known legitimate traffic, using a bio-inspired hyperparameter optimization. The use of multiple public datasets and the comparison against Snort are also strengths. However, the central claims rest on unverified assumptions about the disjointness of the legitimate test data from the training data and about the correct use of cross-validation. Because these assumptions are not met or not demonstrated, the headline numbers cannot be taken at face value.","major_comments":[{"comment":"The 0% FPR claim against known legitimate datasets is not supported because the manuscript does not establish that CTU-Normal-7/12/22/31 are disjoint from the CTU-13 training scenarios. The file names directly echo CTU-13 scenario IDs 7 and 12, making it plausible that their normal traffic was used as the normal/background class during training. The paper provides no capture timestamps, host lists, or de-duplication check, and no statement of whether any CTU-13 scenario's normal class includes those flows. Under the product rule F-R-4 in Section XI, a flow is flagged if any of the 13 models calls it botnet; a true 0% FPR over the reported millions of normal flows is therefore surprising unless those flows are in-distribution with respect to the trained models. This concern directly undermines the abstract's and conclusion's central performance claim.","section":"Section XIII, Table XXVIII"},{"comment":"Feature selection is performed on the full dataset before the stratified 10-fold cross-validation, which leaks information from the test folds into the training folds. The paper states that \"the feature ranking was performed using RF in sklearn, fitting the dataset\" and that \"only the top 15 features of each dataset were retained\" before training and testing; it does not say that feature selection was nested inside each cross-validation fold. This can inflate all reported cross-validation scores, including the headline 99.85% accuracy and 97.74% F1 in Table XXIX. The feature-selection procedure must be re-run within each training fold for the cross-validation numbers to be valid.","section":"Sections VI.B and VII.A"},{"comment":"The headline \"average accuracy of 99.85%\" is an unweighted average over three datasets with highly disparate sizes and class balances. CTU-13 scenarios contain as little as 0.06% botnet flows (Table I), so accuracy is dominated by majority-class correct classification. The paper reports F1-score as a secondary metric, but the abstract's framing of accuracy overstates the detection capability on botnet flows; reporting a macro-averaged metric or a metric such as Matthews correlation coefficient would be necessary to support the claim that the model is highly accurate at detecting botnets.","section":"Abstract and Section XIII, Table XXIX"},{"comment":"The FPR comparison to Snort is based on different numbers of normal flows per dataset, and the paper reports only FPR for the RF-GA models without confidence intervals or any statistical test, so the 0% value cannot be distinguished from a lucky threshold on a small sample. Additionally, the internal consistency of the reported metrics is weak: Table XXX states an F1-score for CTU-13 of 97.5% while Table XXIX's average F1 is 97.74%, and Table XXXI's ISCX F1 of 99.5% does not match the 99.47% in Table XXIX. These numbers should be reconciled before the comparisons can be assessed.","section":"Section XIII, Tables XXVIII and XXIX"}],"minor_comments":[{"comment":"The text around equation (1) is garbled: the sentence \"The FPR can be calculated with the following formula in equation-1, where FP is false positive and TN is true negative: FP FPR = FP + TN (1) and False Positive Rate metrics\" is incomplete and the equation is not cleanly typeset.","section":"Section II.A.3, Eq. (1)"},{"comment":"The extended feature table has a duplicate \"State\" row and misaligned checkmarks, and the caption cites \"[Bullard]\" without a year, which should be corrected for readability.","section":"Table IV"},{"comment":"The K-Neighbors gene pool lists \"Algorithm ['bal tree', 'kd tree']\" with typos; these should be \"ball_tree\" and \"kd_tree\".","section":"Table XXIII"},{"comment":"References [59] and [60] appear out of numerical order in the reference list, and reference [6] has an incomplete author list; both should be corrected.","section":"References"},{"comment":"The GA fitness chromosome for scenario 9 contains the typo \"Noe\" instead of \"None\".","section":"Table XXIX"},{"comment":"The paper would benefit from a data availability statement and a link to the code or scripts, since no code is provided despite the use of public datasets; the absence of code limits reproducibility.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The reader's concern about the CTU-Normal files overlapping with CTU-13 training data is well-founded and is, in my view, the decisive issue. The feature-selection leakage in the cross-validation procedure is a second, independent flaw that affects the internal validity of all reported accuracy figures. Both are load-bearing for the central claims, and the latter would require re-running the entire experimental pipeline to fix. I therefore recommend rejection, despite the paper's use of public datasets and the authors' effort to build a complete software prototype."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent but incremental application of random forest plus genetic-algorithm hyperparameter tuning to three public botnet datasets. The headline numbers—99.85% average accuracy and 0% FPR—do not hold up under scrutiny, and the 0% FPR in particular looks like a leakage artifact.\n\nWhat the paper does well: it runs a systematic comparison across seven classifiers, uses stratified 10-fold cross-validation, does per-dataset feature selection, and documents the GA search space and the product implementation. The related work is honest—it cites several prior papers doing GA-optimized botnet detection (Moubayed et al. 2020, Resende & Drummond 2018) and RF on these datasets, which undercuts the novelty but shows the authors know the landscape. The datasets are public and the preprocessing is described in enough detail to be repeatable in principle.\n\nThe soft spots are load-bearing. First, the average accuracy is an unweighted mean over datasets where the botnet class ranges from 0.06% to 36%. An accuracy of 99.85% mostly reflects majority-class dominance, not detection skill. The F1 numbers are more informative, and they are lower (97.74%). Second, the 0% FPR claim against CTU-Normal-7/12/22/31 is unsupported. The paper never shows that those normal captures are disjoint from the CTU-13 training scenarios, and the naming overlap (CTU-Normal-7 and CTU-Normal-12 matching scenario IDs 7 and 12) makes overlap plausible. The paper also uses a product rule (F-R-4): alert if any of the 13 models flags a flow. With that rule, a true 0% FPR across millions of normal flows is surprising unless the test flows are in-distribution. Third, there are no error bars, no code or data release, and some internal inconsistencies in the comparison tables (e.g., accuracy 99.9% in Table XXX vs 99.85% average). Typos like 'squaredhingel' and 'Noe' suggest light copyediting.\n\nWho is this for? It reads like a solid final-year project or a conference poster. The product demo is nice, and the comparative evaluation could be useful to a practitioner wanting a quick baseline. But as a research contribution it does not clear the bar: the central performance claim is not supported as stated, and the methodology is not new.\n\nRecommendation: desk reject. If the authors can release the code and data, and demonstrate that the legitimate test files are truly disjoint from training, the FPR result might become meaningful—but that would be a different paper.","headline":"A careful, incremental ML comparison undone by a likely leakage in the headline 0% FPR claim; the rest is standard engineering.","tokens_in":27062,"tokens_out":2825,"would_cite":false,"duration_ms":24751,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Random Forest tuned by a Genetic Algorithm can identify botnet traffic from network flow statistics alone, reaching 99.85% average accuracy across three public datasets.","keywords":["botnet detection","flow-based detection","network flows","Random Forest","Genetic Algorithm","hyperparameter optimisation","CTU-13 dataset","ISCX Botnet 2014"],"falsifier":"Open the CTU-13 and CTU-Normal datasets and test whether any flow in the CTU-Normal-7/12/22/31 files matches, by source and destination IP pair, timestamps, and flow features, a flow used in the stratified 10-fold training of the thirteen CTU-13 RF models; any non-trivial overlap would reduce the reported 0% FPR, and rerunning the models on a strictly held-out normal-traffic capture taken at a different time would settle the external claim.","tokens_in":25977,"feed_emoji":"🛡️","tokens_out":7876,"duration_ms":68951,"temperature":0.7,"pith_summary":"Botnets routinely hide their command traffic by encrypting payloads or varying ports and protocols, which breaks signature-matching detectors. This paper argues that a purely flow-based behavioural model—one that reads aggregate connection statistics and drops IPs, ports, protocols and payload data—can separate botnet traffic from normal traffic across three public datasets. It compares seven machine-learning families and uses a Genetic Algorithm to tune hyperparameters, reporting that Random Forest optimised this way reaches an average accuracy of 99.85% and an average F1 of 97.74% over CTU-13, ISOT 2010 and ISCX Botnet 2014. The paper further reports 0% false positives on four known-legitimate CTU normal-traffic files, compared with Snort's average 1.36% on the same files, and packages the result as an intrusion-detection software prototype. If these numbers hold on genuinely unseen traffic, the result matters because a detection method that needs no payload access is robust to the evasion tactics that make signature systems blind.","feed_headline":"Flow-based botnet detector hits 99.85% accuracy","feed_subtitle":"A flow-only model ignores encryption and port-fluxing, so it catches traffic that signature systems miss.","key_machinery":"The load-bearing mechanism is the combination of a flow-based feature representation and GA-driven hyperparameter search. Network flows aggregate packets from one connection into records; the extended CTU-13 feature set supplies 33 fields such as total bytes, packet counts, TCP window sizes, round-trip time, TTL, and byte rates, and the pipeline manually removes IP addresses, ports, protocol identity, and timestamps, leaving 24 structural features, then keeps the top 15 per dataset by Random-Forest feature importance. The Genetic Algorithm treats each classifier hyperparameter as a gene, seeds the population with the known default hyperparameter chromosome, evaluates each chromosome's fitness by stratified 10-fold cross-validated F1, selects the two fittest parents, performs single-point crossover, fills the population with offspring and random chromosomes, and mutates one random gene per chromosome over ten generations. What this machinery does is replace exhaustive grid search, which the paper estimates at years for neural networks, with a guided search whose main methodological payoff is the claim that the final RF models are consistently strong across very different botnet behaviours.","core_discovery":"On the paper's own terms, the central discovery is that a Random Forest classifier with hyperparameters selected by a Genetic Algorithm generalises across three flow-based botnet datasets well enough to approach perfect detection without ever inspecting packet contents. The pipeline removes addressing and port features deliberately, retains the 15 highest-importance flow features per dataset, evaluates via stratified 10-fold cross-validation, then uses GA to maximise F1 with population and generation limits of 10 for classical classifiers and 15 for neural networks. The optimised RF achieves an average accuracy of 99.85% across CTU-13, ISOT and ISCX, an average F1 of 97.74%, and, when thirteen CTU-13-derived models are tested on the CTU-Normal-7/12/22/31 legitimate flow files, a false-positive rate of 0%, compared with Snort's 1.36% average on the same files. The paper interprets this as evidence that flow-level behavioural modelling, stripped of payload-dependent signals, is sufficient for botnet command-and-control detection and can be deployed as a lightweight network intrusion detection system.","pith_inferences":["The 0% false-positive result is only externally meaningful if the CTU-Normal-7/12/22/31 files are disjoint from the CTU-13 scenario traffic used for training; the paper does not establish that disjointness, so a natural next experiment is to check for flow-level overlap or temporal leakage before trusting the comparison.","The same pipeline could be tested on a time-ordered split of one continuous capture, rather than per-scenario stratified folds, to separate the claim 'can classify known botnet behaviour' from 'can detect novel botnet behaviour in production'.","Because GA maximises F1 and the CTU scenarios are highly imbalanced, the reported 99.85% accuracy is dominated by true negatives; a deployment-oriented metric such as detection latency on the minority class or cost per alert would tell operators more than accuracy alone.","Replacing GA with an online or streaming optimisation method could test whether the high F1 depends on expensive per-dataset retraining, which matters if the software product must adapt to new bot families without expert intervention."],"forward_implications":["A detector built on flow statistics alone can identify botnet command-and-control traffic even when the bot encrypts payloads, fragments packets, or fast-fluxes ports and protocols, because none of those signals are used by the model.","The 0% FPR on CTU-Normal-7/12/22/31, if valid, means a flow-based RF-GA model can run on a network mirror and generate far fewer false alerts than signature-based Snort in normal web and P2P environments.","The GA tuning procedure transfers across classifiers: even the weakest default classifier (linear SVM, F1 0.656) improves to 0.741 after optimisation, showing the hyperparameter search space may matter as much as the model family.","Consistent near-99% accuracy across CTU-13, ISOT and ISCX suggests the top flow features (TTL, source bytes, source window, total bytes) capture botnet behaviour that survives dataset shifts.","A serialised RF-GA model can be embedded in a sniffer-based IDS that reads live SPAN-port traffic, so deployment does not require per-network signature updates."],"supporting_citations":[{"why":"Supplies the CTU-13 dataset, its thirteen scenarios, and the extended 33-field flow feature set that the pipeline reuses as the feature schema for all three datasets.","marker":"Garcia et al. (2014)"},{"why":"Supplies the ISOT 2010 dataset of merged Storm/Waledac/Zeus bot traffic plus normal Ericsson and LBNL traffic, used as the second evaluation dataset.","marker":"Saad et al. (2011)"},{"why":"Defines the ISCX Botnet 2014 dataset's merging methodology and motivates port-independent feature selection, used as the third dataset.","marker":"Beigi et al. (2014)"},{"why":"Established RF-based feature ranking on CTU-13 conversational features, the mechanism adopted for selecting the top-15 features.","marker":"Chen et al. (2017)"},{"why":"Reports RF-based tree feature selection with 10-fold cross-validation on ISOT, justifying the cross-validation and feature-selection design.","marker":"Pektas & Acarman (2017)"},{"why":"Reports that a GA-optimised Random Forest outperforms a standalone classifier for botnet detection on DNS data, the direct motivation for the RF-GA pairing.","marker":"Moubayed et al. (2020)"},{"why":"Supports GA plus profiling for adaptive anomaly-based intrusion detection with high detection rate and low false-positive rate, cited as evidence for GA's suitability.","marker":"Resende & Drummond (2018)"},{"why":"Demonstrates GA's utility in reducing feature space and training time for malware detection, cited to justify the bio-inspired optimisation approach.","marker":"Fatima et al. (2019)"}],"fun_headline_variants":["Bio-inspired ML nabs botnets at 99.85% accuracy","Flow-only botnet detector hits 99.85%","Random Forest + GA: 99.85% botnet detection","Botnet detection at 99.85% using flow, not payloads","Genetic-algorithm-tuned RF: 99.85% botnet accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire case that the model has zero false positives on legitimate traffic rests on the assumption that the CTU-Normal-7/12/22/31 traffic files are separate from the CTU-13 scenario data used to train the models; if they overlap, the FPR result is a trivial artifact.","fun_headline_variants_meta":{"raw":{"variants":["Bio-inspired ML nabs botnets at 99.85% accuracy","Flow-only botnet detector hits 99.85%","Random Forest + GA: 99.85% botnet detection","Botnet detection at 99.85% using flow, not payloads","Genetic-algorithm-tuned RF: 99.85% botnet accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000351,"raw_usage":{"total_tokens":1938,"prompt_tokens":990,"completion_tokens":948,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":854}},"tokens_in":606,"tokens_out":948,"duration_ms":8547,"temperature":1.0,"reasoning_tokens":854,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:26:39.311105+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Open the CTU-13 and CTU-Normal datasets and test whether any flow in the CTU-Normal-7/12/22/31 files matches, by source and destination IP pair, timestamps, and flow features, a flow used in the stratified 10-fold training of the thirteen CTU-13 RF models; any non-trivial overlap would reduce the reported 0% FPR, and rerunning the models on a strictly held-out normal-traffic capture taken at a different time would settle the external claim.","supporting_citations":[{"cited_title":"Detecting p2p botnets through network behavior analysis and machine learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the ISOT 2010 dataset of merged Storm/Waledac/Zeus bot traffic plus normal Ericsson and LBNL traffic, used as the second evaluation dataset."},{"cited_title":"Towards effective feature selection in machine learning- based botnet detection approaches,","cited_arxiv_id":null,"evidence_quote":"Defines the ISCX Botnet 2014 dataset's merging methodology and motivates port-independent feature selection, used as the third dataset."},{"cited_title":"An effective conversation -based botnet detection method,","cited_arxiv_id":null,"evidence_quote":"Established RF-based feature ranking on CTU-13 conversational features, the mechanism adopted for selecting the top-15 features."},{"cited_title":"Effective feature selection for botnet detection based on network flow analysis,","cited_arxiv_id":null,"evidence_quote":"Reports RF-based tree feature selection with 10-fold cross-validation on ISOT, justifying the cross-validation and feature-selection design."},{"cited_title":"Optimized Random Forest Model for Botnet Detection Based on DNS Queries","cited_arxiv_id":"2012.11326","evidence_quote":"Reports that a GA-optimised Random Forest outperforms a standalone classifier for botnet detection on DNS data, the direct motivation for the RF-GA pairing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports GA plus profiling for adaptive anomaly-based intrusion detection with high detection rate and low false-positive rate, cited as evidence for GA's suitability."},{"cited_title":"Android malware detection using genetic algo - rithm based optimized feature selection and machine learn - ing,","cited_arxiv_id":null,"evidence_quote":"Demonstrates GA's utility in reducing feature space and training time for malware detection, cited to justify the bio-inspired optimisation approach."}],"review_version":1}