Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Tracking Temporal Evolution of Network Activity for Botnet Detection

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Tracking how a host's communication graph changes over time—via an LSTM over per-window graph features—detects botnet-infected hosts across botnet types at 96.2% accuracy, without reading packet contents.

desk verdict The cross-scenario AUROC is the real evidence here; the 96.2% accuracy is likely inflated by overlapping-window leakage in the train/test split. read the letter →

arxiv 1908.03443 v1 pith:ODNIIPUZ submitted 2019-08-09 cs.CR cs.LG

Kapil Sinha , Arun Viswanathan , Julian Bunn This is my paper · ORCID
classification cs.CRcs.LG
keywords botnetdetectionLSTMgraphfeaturesnetworksecurityCTU-13timeseriesclassificationcontent-agnostic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most botnet detectors rely on signatures, payloads, or flow statistics, all of which fail once the botnet encrypts traffic or changes its protocol. This paper tries to establish a different basis for detection: the time-varying shape of a host's communication graph. It claims that an LSTM fed a per-host time series of graph centrality and degree features can flag infected hosts on the CTU-13 benchmark with 96.2% accuracy, 94.6% sensitivity, and 96.3% specificity, while generalizing across IRC, DDoS, P2P, port-scan, and HTTP botnets. The reason to care is that a structural, content-agnostic detector would remain effective against botnets that encrypt their payloads or change protocols, and would complement payload-based detectors.

What carries the argument

The load-bearing object is the per-host time series of graph features, assembled by sliding a 300-second window with 150-second overlap across the packet capture, building a directed graph of IP addresses (packets as edges), and computing ten centrality/degree statistics for every node in each window. Zero vectors fill windows where a host is silent, so every host has a fixed-length sequence, and the sequence is cut into overlapping five-window samples. A Long Short-Term Memory network—a recurrent architecture that carries a hidden state across time steps and can learn periodic patterns, matching the dormancy/activity cycles botnets show—reads each sample and outputs a sigmoid botnet/normal decision. The mechanism works because repeated contact with a command-and-control server, or with peer bots in a P2P topology, changes degree, centrality, and clustering statistics in a way that is independent of payload, port, and protocol.

What would settle it

Re-run the same CTU-13 experiments with a chronological split: train on the first 70% of each scenario's windows and test on the last 30%, so no test window overlaps any training window, or train only on some scenarios and test on a held-out scenario. If accuracy drops far below 96.2% or the cross-scenario AUROC values fall toward 0.5, the independence assumption, not the graph-plus-LSTM mechanism, is carrying the reported performance.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a botnet's coordinated communication leaves a detectable signature in the temporal evolution of graph statistics, not in any single snapshot. The authors construct 300-second, 150-second-overlapping windows of network traffic, build a directed graph of IP addresses with packets as edges, and extract ten per-node features (degree, neighbor counts, PageRank, betweenness, eigenvector, authority, hub, and local clustering coefficient). After zero-padding absent nodes and cutting the series into five-window samples, a 64-unit LSTM classifies each host as botnet or normal. Trained and tested on scenarios 6, 7, 10, 11 and 12 of CTU-13, the model reaches 96.2% accuracy, a true positive rate of 94.6%, and a false positive rate of 3.73%; the cross-scenario AUROC table shows most models scoring 0.90 or above when tested on scenarios they were not trained on. The paper's conclusion is that temporal graph features give a botnet-agnostic, content-agnostic detector that outperforms all surveyed content-agnostic methods and several content-aware ones.

Load-bearing premise

The evaluation assumes that randomly splitting overlapping 300-second windows into 70% train and 30% test produces independent samples, even though windows overlap by 150 seconds and come from the same hosts and scenarios, so the reported accuracy may be inflated by correlated data on both sides of the split.

Editorial extensions

If this is right

  • Encryption, payload spoofing, and protocol changes do not hide a botnet from this detector, because the features come only from connection structure and packet headers.
  • A model trained on one or a few botnet scenarios transfers to unseen botnet types: the cross-scenario AUROC table reports values at or above 0.9 for most pairs involving the longer scenarios (6, 10, 12), including training on a 1-bot port-scan scenario and testing on P2P and IRC botnets.
  • Because classification happens per host and per window, the pipeline can emit a verdict for a host as soon as a new window ends, without waiting for the full capture or for a clustering pass over all nodes.
  • Compared with the surveyed content-agnostic methods on CTU-13, this approach posts the best sensitivity (0.946 TPR) at a false-positive rate of 0.037, making it the strongest structural-only detector in the comparison.
  • Combining this structural detector with a payload-based detector would produce an ensemble that is harder to evade than either component alone, a combination the paper explicitly proposes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported accuracy is likely optimistic in deployment: because windows overlap by 150 seconds and many windows come from the same host and scenario, the random 70/30 split can leak correlated samples into both training and testing. I would want a split by time (earlier windows train, later windows test) or by scenario to see the accuracy the architecture would genuinely achieve on new infection
  • The 'botnet-agnostic' claim is really a claim about communication structure: botnets that hide control traffic inside ordinary Web-like or low-rate flows, or that avoid repeated contact patterns, may not move the ten graph features enough for the LSTM to separate them from benign hosts. Testing on stealthy, non-periodic botnet traffic would map the boundary of the method.
  • The authors' own timing numbers—7000 minutes of feature extraction for scenario 10, longer than the capture itself—show that the real-time claim depends on the weighted-edge compression and parallelization they propose but do not implement. Until those are done, the training-time pipeline is offline, though a trained model could still run per-window in near real time.
  • A natural extension the authors leave implicit: retrain the LSTM on features aggregated per flow (weighted edges) instead of per packet; if accuracy holds, the speedup could make the method deployable at the network edge. This is directly testable with the paper's released code.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a supervised botnet host detection method that builds time-varying communication graphs from packet captures, extracts ten graph-based features per node per time interval, constructs per-host feature time series with zero padding, and trains an LSTM classifier on short overlapping windows of these time series. The prototype is evaluated on five CTU-13 scenarios (6, 7, 10, 11, 12) using a 70%/30% split, a cross-scenario AUROC table, and a combined-scenario accuracy of 96.2% with a TPR of 94.6% and an FPR of 3.73%. The authors claim that the approach is content-agnostic, generalizable across botnet types, amenable to real-time implementation, and competitive with existing methods.

Significance. If the evaluation were sound, the paper would make a useful practical contribution: a content- and payload-agnostic detector based on graph structure over time, with a public code release, targeting a known limitation of flow-based detectors. The cross-scenario AUROC table (Table 2) provides some independent evidence that the learned representations transfer across scenarios, which is the strongest part of the empirical support. However, the headline metrics rest on a split of overlapping windows that can share hosts and temporal context, and the abstract's real-time claim is contradicted by the authors' own timing measurements. The central ideas are defensible, but the reported quantitative claims need substantial re-evaluation before they can be accepted.

major comments (4)
  1. [§3.2, §3.3.2] The 70%/30% split is applied to windows that overlap by construction: the 300-second intervals advance by 150 seconds, and the 5-interval windows overlap by 2 intervals. As a result, windows from the same host and the same temporal neighborhood can appear on both sides of the split, allowing the LSTM to memorize host- or time-specific patterns rather than learn a generalizable botnet signature. The reported accuracy of 96.2%, TPR of 94.6%, and FPR of 3.73% are therefore not valid estimates of performance on unseen hosts or future time periods. The evaluation should be repeated with a split stratified by host and by disjoint time blocks, with per-scenario metrics and confidence intervals reported.
  2. [Abstract, §4.3] The abstract states the method is 'amenable to real-time implementation,' but Section 4.3 reports that feature extraction for scenario 10, which lasts 4.75 hours, took 7000 minutes of single-core computation, more than an order of magnitude longer than the scenario duration. This is a direct contradiction within the manuscript. The real-time claim should be removed or explicitly restricted to the proposed future optimizations (weighted-edge graphs, parallelization, GPU use), which have not been demonstrated.
  3. [§4.1, Table 3] The authors acknowledge that 'performing a clear comparison with our approach is impossible' because prior work uses different scenario subsets and evaluation methodologies, but Table 3 and the text directly compare reported metrics from these incompatible protocols. The claim that the approach 'by far the best in comparison with content-agnostic approaches' is therefore not supported by the evidence presented. The comparison should be reframed as illustrative, or the competing methods should be reimplemented and evaluated under the same train/test protocol.
  4. [§3.3.1, Table 2] The cross-scenario AUROC table contains near-chance off-diagonal values, e.g., 0.56 for a model trained on scenario 11 and tested on scenario 6, and 0.58 for a model trained on scenario 7 and tested on scenario 11. The conclusion that 'our model generalizes strongly' is therefore overstated. The discussion should quantify this variability, investigate which source-target scenario pairs fail, and restrict the generalization claim to the scenario types and data conditions that actually transfer.
minor comments (6)
  1. [§1, Evaluation paragraph] The sentence 'We develop a prototype implementation of our algorithm, and evaluate it over the over the benchmark CTU-13 dataset' contains a duplicated phrase 'over the over the'; it should be 'over the benchmark CTU-13 dataset'.
  2. [§2.4.2] The statement that 'the only two hyperparameters in our model are window size and step size' is inaccurate: hidden layer size, number of epochs, the loss weighting factor of six, and the 10:1 undersampling ratio are also choices made by the authors. The text should either justify why these are not considered hyperparameters or list them explicitly.
  3. [§2.4.1] The use of a weighted MSE loss with a six-times weight on malicious samples, in combination with a 10:1 undersampling ratio, is not explained or justified. A brief note on how these two mechanisms interact would improve reproducibility.
  4. [Table 3] The row for reference [22] reads 'Yu at al.' and should be 'Yu et al.'.
  5. [§4.2] The phrase 'memory access optimization's' should be 'memory access optimizations'.
  6. [§2.3.1] The text uses the misspelling 'betweeness centrality'; the standard spelling is 'betweenness centrality'.

Circularity Check

1 steps flagged · score 6.0 of 10

The headline 96.2% accuracy is computed from a random split of overlapping windows that share intervals and hosts, so it is a leaked evaluation metric; the cross-scenario AUROC table still gives independent generalization evidence.

  1. fitted input called prediction [Section 2.3.3 and Section 3.2 (see also Section 3.3.2)]
    "Then we divide each node’s time-series of x intervals into smaller windows of 5 intervals that overlap by 2 intervals (i.e. the first window contains intervals 1 - 5, the second window contains intervals 3 - 7, etc.). ... Due to the class imbalance between malicious and non-malicious nodes, train/test splits are applied independently to both the set of positive (malicious) and negative (non-malicious) examples e.g."

    The 96.2% accuracy reported in Section 3.3.2 is produced by evaluation mode 3 in Section 3.2: train on 70% of windows from a combination of scenarios and test on the remaining 30%. But the windows are overlapping 5-interval segments taken from the same hosts and scenarios, so a random split can place test windows that share up to 3 of 5 input intervals with training windows, often from the same node's zero-padded time series. The LSTM is therefore able to memorize host-specific temporal patterns instead of detecting novel botnet activity, and the stated accuracy, TPR, and FPR are statistically forced by the split rather than being independent predictions.

full rationale

The algorithm itself is not a circular derivation: the input features, LSTM architecture, and CTU-13 data are all external to the claim, and the cross-scenario AUROC matrix gives genuine out-of-scenario generalization evidence. The circularity is confined to the headline within-combination evaluation: because the windows overlap and come from the same hosts, the 70%/30% random split makes the 96.2% accuracy a leaked estimate. The paper also acknowledges in Section 4.3 that feature extraction for scenario 10 took 7000 minutes, exceeding the scenario duration, which contradicts the abstract's 'amenable to real-time implementation' claim; that is a performance limitation rather than a circularity. No load-bearing self-citation, imported uniqueness theorem, or ansatz-smuggling is present.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claims rest on dataset assumptions (CTU-13 labels, graph-feature generalizability), an independence assumption in the evaluation split, and hand-chosen hyperparameters. No invented entities are introduced.

free parameters (6)
  • window_size = 300 s
    Chosen by visual exploration (Section 2.4.2); determines graph construction and temporal resolution; no sensitivity analysis.
  • step_size = 150 s
    Chosen by visual exploration (Section 2.4.2); controls overlap between windows; no sensitivity analysis.
  • undersampling_ratio = 10:1 non-malicious to malicious
    Training-phase undersampling (Section 2.3.3); affects class balance and model behavior.
  • malicious_loss_weight = 6x
    Weighted MSE loss (Section 2.4.1) used to compensate for imbalance; no ablation.
  • hidden_units = 64
    LSTM hidden layer size (Section 2.4.1); no architecture search.
  • epochs = 200
    Training duration (Section 2.4.1); no early stopping or validation-based selection reported.
assumptions (4)
  • domain assumption CTU-13 scenario ground-truth labels, including malicious hosts and infection times, are correct.
    All labels and metrics in Section 3 rely on dataset-provided ground truth.
  • domain assumption Overlapping 300-second windows can be treated as independent samples after a random 70/30 split.
    Evaluation in Section 3.2 assumes train/test independence; violated because windows overlap by 150 seconds and belong to the same hosts.
  • domain assumption Botnet activity exhibits periodic or dormant patterns that an LSTM can exploit in 300-second windows.
    Motivates LSTM in Section 2.4, citing AsSadhan et al. [12]; not verified for all scenarios.
  • domain assumption Graph communication structure is similar enough across botnet types to generalize.
    Core premise for the botnet-agnostic claim in Section 1; only tested within CTU-13 scenarios.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tracking Temporal Evolution of Network Activity for Botnet Detection." pith.science (2026). https://pith.science/paper/ODNIIPUZ

@misc{pith2026190803443,
  author       = {Pith},
  title        = {Pith review of: Tracking Temporal Evolution of Network Activity for Botnet Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ODNIIPUZ}},
  note         = {Machine review of arXiv:1908.03443}
}
read the original abstract

Botnets are becoming increasingly prevalent as the primary enabling technology in a variety of malicious campaigns such as email spam, click fraud, distributed denial-of-service (DDoS) attacks, and cryptocurrency mining. Botnet technology has continued to evolve rapidly making detection a very challenging problem. There is a fundamental need for robust detection methods that are insensitive to characteristics of a specific botnet and are generalizable across different botnet types. We propose a novel supervised approach to detect malicious botnet hosts by tracking a host's network activity over time using a Long Short-Term Memory (LSTM) based neural network architecture. We build a prototype to demonstrate the feasibility of our approach, evaluate it on the CTU-13 dataset, and compare our performance against existing detection methods. We show that our approach results in a more generalizable, botnet-agnostic detection methodology, is amenable to real-time implementation, and performs well compared to existing approaches, with an overall accuracy score of 96.2%.

Figures

Figures reproduced from arXiv: 1908.03443 by the authors.

Figure 1
Figure 1. The series of graphs is generated from the packet capture file of scenario 12 (P2P botnet) in the CTU-13 [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Schematic of the Algorithm Workflow [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Graph feature extraction from network data [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: ROC curve of model trained and tested on scenarios 6, 7, 10, 11, and 12 [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Flow-based Detection of Botnets through Bio-inspired Optimisation of Machine Learning

    cs.CR 2024-12 reject novelty 2.0 of 10

    A random forest model with genetic-algorithm hyperparameter tuning reports high accuracy on three public botnet datasets, but the evaluation relies on imbalanced accuracy and lacks independent validation.

Reference graph

Works this paper leans on

25 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    https://www.cyber.nj.gov/threat-profiles/botnet-variants/smominru , Feb 2018

    Smominru. https://www.cyber.nj.gov/threat-profiles/botnet-variants/smominru , Feb 2018

    work page 2018

  2. [2]

    Botnet technology

    R Jaiswal and Shivraj Bajgude. Botnet technology. In 3rd International Conference on Emerging Trends in Computer and Image Processing (ICETCIP’2013), pages 169–175, 2013

    work page 2013

  3. [3]

    Behavioral clustering of http-based malware and signature generation using malicious network traces

    Roberto Perdisci, Wenke Lee, and Nick Feamster. Behavioral clustering of http-based malware and signature generation using malicious network traces. In Proceedings of the 7th USENIX Conference on Networked Systems Design and Implementation, NSDI’10, pages 26–26, Berkeley, CA, USA, 2010. USENIX Association

    work page 2010

  4. [4]

    Heuristics for detecting botnet coordinated attacks

    Kazuya Kuwabara, Hiroaki Kikuchi, Masato Terada, and Masashi Fujiwara. Heuristics for detecting botnet coordinated attacks. In Proceedings of the 4th International Workshop on Advances on Information Security (WAIS2010), pages 603–607, 02 2010

    work page 2010

  5. [5]

    Timothy Strayer, David Lapsely, Robert Walsh, and Carl Livadas

    W. Timothy Strayer, David Lapsely, Robert Walsh, and Carl Livadas. Botnet Detection Based on Network Behavior, pages 1–24. Springer US, Boston, MA, 2008. 11 A PREPRINT - AUGUST 12, 2019

    work page 2008

  6. [6]

    Botnet detection using graph-based feature clustering

    Sudipta Chowdhury, Mojtaba Khanzadeh, Ravi Akula, Fangyan Zhang, Song Zhang, Hugh Medal, Mohammad Marufuzzaman, and Linkan Bian. Botnet detection using graph-based feature clustering. Journal of Big Data, 4(1):14, May 2017

    work page 2017

  7. [7]

    An empirical comparison of botnet detection methods

    Sebastian Garcia, Martin Grill, Jan Stiborek, and Alejandro Zunino. An empirical comparison of botnet detection methods. computers & security, 45:100–123, 2014

    work page 2014

  8. [8]

    An efficient flow-based botnet detection using supervised machine learning

    Matija Stevanovic and Jens Myrup Pedersen. An efficient flow-based botnet detection using supervised machine learning. In 2014 International Conference on Computing, Networking and Communications, ICNC 2014, pages 797–801, 02 2014

    work page 2014

Show all 25 references
  1. [9]

    Botgrep: Finding p2p bots with structured graph analysis

    Shishir Nagaraja, Prateek Mittal, Chi-Yao Hong, Matthew Caesar, and Nikita Borisov. Botgrep: Finding p2p bots with structured graph analysis. In Proceedings of the 19th USENIX Conference on Security, USENIX Security’10, pages 7–7, Berkeley, CA, USA, 2010. USENIX Association

    2010

  2. [10]

    Botnet detection using social graph analysis

    Jing Wang and Ioannis Paschalidis. Botnet detection using social graph analysis. 2014 52nd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2014, 03 2015

    2014

  3. [11]

    An analysis of recurrent neural networks for botnet behavior detection

    Pablo Torres, Carlos Catania, Sebastián García, and Carlos Garcia Garino. An analysis of recurrent neural networks for botnet behavior detection. In Proc. IEEE Biennial Congr. Argentina (ARGENCON), 06 2016

    2016

  4. [12]

    Basil AsSadhan, Jose Moura, and David E. Lapsley. Periodic behavior in botnet command and control channels traffic. In GLOBECOM: Global Telecommunications Conference IEEE, pages 1–6, 11 2009

    2009

  5. [13]

    Tiago P. Peixoto. The graph-tool python library. figshare, 2014

    2014

  6. [14]

    Real-time Peer-to-Peer Botnet Detection Framework based on Bayesian Regularized Neural Network

    Sharath Chandra Guntuku, Pratik Narang, and Chittaranjan Hota. Real-time Peer-to-Peer Botnet Detection Framework based on Bayesian Regularized Neural Network. jul 2013

    2013

  7. [15]

    Long short-term memory

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997

    1997

  8. [16]

    François Chollet et al. Keras. https://keras.io, 2015

    2015

  9. [17]

    Tieleman and G

    T. Tieleman and G. Hinton. Lecture 6.5—RmsProp: Divide the gradient by a running average of its recent magnitude. Coursera: Neural Networks for Machine Learning, 2012

    2012

  10. [18]

    The relationship between precision-recall and roc curves

    Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. In Proceedings of the 23rd International Conference on Machine Learning, ICML ’06, pages 233–240, New York, NY , USA, 2006. ACM

    2006

  11. [19]

    Reducing false positives of network anomaly detection by local adaptive multivariate smoothing

    Martin Grill, Tomás Pevný, and Martin Rehak. Reducing false positives of network anomaly detection by local adaptive multivariate smoothing. Journal of Computer and System Sciences, 83, 04 2016

    2016

  12. [20]

    Big data analytics for network anomaly detection from netflow data

    Duygu Sinanc, Ramazan Terzi, and Seref Sagiroglu. Big data analytics for network anomaly detection from netflow data. In Computer Science and Engineering (UBMK), 2017 International Conference on. IEEE, pages 592–597, 10 2017

    2017

  13. [21]

    Learning combination of anomaly detectors for security domain

    Martin Grill and Tomás Pevný. Learning combination of anomaly detectors for security domain. Computer Networks, 107, 06 2016

    2016

  14. [22]

    Session-based network intrusion detection using a deep learning architecture

    Yang Yu, Jun Long, and Zhiping Cai. Session-based network intrusion detection using a deep learning architecture. In Modeling Decisions for Artificial Intelligence, vol. 10571 of Lecture Notes in Computer Science, pages 144–155, 09 2017

    2017

  15. [23]

    Nur Zincir-Heywood

    Fariba Haddadi, Duong-Tien Phan, and A. Nur Zincir-Heywood. How to choose from different botnet detection systems? NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium, pages 1079–1084, 2016

    2016

  16. [24]

    A comparative study of machine learning algorithms and their ensembles for botnet detection

    Songhui Ryu and Baijian Yang. A comparative study of machine learning algorithms and their ensembles for botnet detection. Journal of Computer and Communications, 06:119–129, 01 2018

    2018

  17. [25]

    An effective conversation-based botnet detection method

    Ruidong Chen, Weina Niu, Xiaosong Zhang, Zhongliu Zhuo, and Fengmao Lv. An effective conversation-based botnet detection method. Mathematical Problems in Engineering, 2017:1–9, 04 2017. 12

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.