Temporal Analysis of NetFlow Datasets for Network Intrusion Detection Systems
Pith reviewed 2026-05-23 00:56 UTC · model grok-4.3
The pith
NetFlow datasets enhanced with temporal features reveal unique time-frequency patterns for many attacks.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
By incorporating temporal features into NetFlow datasets and applying time-frequency analysis, many attacks display unique patterns in their time-frequency signal presentations that are not apparent in conventional features.
What carries the argument
Time-frequency signal presentations (TFSPs) computed from temporal NetFlow features, which serve to expose attack-specific patterns across time and frequency domains.
If this is right
- Machine learning models for network intrusion detection can use the distinct TFSP patterns to separate attack classes more easily.
- The released datasets with temporal features enable new temporal and time-frequency studies of NIDS data.
- Time-series representations of NetFlow flows provide additional views into how attack traffic evolves over time.
- Unique attack patterns in the time-frequency domain suggest potential improvements in detection accuracy for existing ML-based systems.
Where Pith is reading between the lines
- The approach could extend to other flow-based datasets if the temporal features are added consistently.
- Combining TFSPs with existing feature sets might reduce confusion between similar attack types in operational settings.
- The public datasets allow direct replication and testing of whether the observed uniqueness holds across different network environments.
Load-bearing premise
The borrowed time-frequency analysis method from signal processing applies directly to NetFlow traffic and the added temporal features accurately represent real-world timing.
What would settle it
Running the same time-frequency analysis on the released datasets and finding that attack types produce overlapping rather than unique patterns would falsify the central observation.
Figures
read the original abstract
This paper investigates the temporal analysis of NetFlow datasets for machine learning (ML)-based network intrusion detection systems (NIDS). Although many previous studies have highlighted the critical role of temporal features, such as inter-packet arrival time and flow length/duration, in NIDS, the currently available NetFlow datasets for NIDS lack these temporal features. This study addresses this gap by creating and making publicly available a set of NetFlow datasets that incorporate these temporal features [1]. With these temporal features, we provide a comprehensive temporal analysis of NetFlow datasets by examining the distribution of various features over time and presenting time-series representations of NetFlow features. This temporal analysis has not been previously provided in the existing literature. We also borrowed an idea from signal processing, time frequency analysis, and tested it to see how different the time frequency signal presentations (TFSPs) are for various attacks. The results indicate that many attacks have unique patterns, which could help ML models to identify them more easily.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper creates and publicly releases NetFlow datasets augmented with temporal features (e.g., inter-packet arrival time, flow duration) that were previously missing from standard NIDS benchmarks. It conducts temporal analysis via feature distributions over time and time-series plots, then applies time-frequency signal processing (TFSP) borrowed from signal processing to generate representations of attack traffic, observing that many attack types display visually distinct patterns in these TFSPs and suggesting this could ease ML-based identification.
Significance. The public release of temporally augmented datasets directly addresses a documented gap in NIDS resources and could enable reproducible follow-on work. The TFSP visualizations offer a novel cross-domain perspective on network traffic; if the distinctiveness claim holds under quantitative scrutiny, it would provide a concrete direction for frequency-domain feature engineering in intrusion detection.
major comments (1)
- [Abstract] Abstract (final sentence) and corresponding results discussion: the assertion that unique TFSP patterns 'could help ML models to identify them more easily' is unsupported because the manuscript contains no downstream classification experiments (no accuracy, F1, precision-recall, or detection-rate comparisons between models using only the added temporal features versus models that also incorporate TFSP-derived features).
Simulated Author's Rebuttal
We thank the referee for their constructive and detailed feedback. We address the major comment below.
read point-by-point responses
-
Referee: [Abstract] Abstract (final sentence) and corresponding results discussion: the assertion that unique TFSP patterns 'could help ML models to identify them more easily' is unsupported because the manuscript contains no downstream classification experiments (no accuracy, F1, precision-recall, or detection-rate comparisons between models using only the added temporal features versus models that also incorporate TFSP-derived features).
Authors: We agree that the final sentence of the abstract (and the parallel phrasing in the results discussion) makes an unsupported forward-looking claim about benefits to ML models. The manuscript presents only qualitative visual observations of distinct TFSP patterns; no classification experiments were performed. In the revised manuscript we will remove or qualify this assertion in both the abstract and the discussion, limiting the text to the supported observation that many attack types exhibit visually distinct TFSP patterns. This revision will keep the contribution focused on the released datasets, the temporal analysis, and the novel cross-domain visualization while ensuring every claim is directly evidenced by the results shown. revision: yes
Circularity Check
No circularity; empirical dataset construction and direct application of existing technique
full rationale
The paper constructs new NetFlow datasets incorporating temporal features and applies a borrowed time-frequency analysis method from signal processing to generate TFSP representations, then reports direct observations of unique patterns across attack types. This chain consists of data preparation followed by empirical visualization and pattern inspection; no step reduces by construction to a fitted parameter, self-citation, or renamed input. The 'could help ML models' statement is an untested inference but does not constitute a derivation that loops back on itself.
Axiom & Free-Parameter Ledger
axioms (2)
- domain assumption NetFlow records can be reliably augmented with temporal features such as inter-packet arrival time and flow duration without altering their fundamental structure
- domain assumption Time-frequency analysis techniques from signal processing transfer meaningfully to NetFlow feature time series for revealing attack-specific patterns
Forward citations
Cited by 1 Pith paper
-
MA-IDS: Multi-Agent RAG Framework for IoT Network Intrusion Detection with an Experience Library
MA-IDS uses two collaborating LLM agents and a persistent experience library to reach 89.75% and 85.22% macro F1 on IoT intrusion datasets while supplying rule-based explanations for each decision.
Reference graph
Works this paper leans on
-
[1]
NetFlow V3 NIDS Datasets - The University of Queensland,
Majed Luay, Siamak Layeghy, Seyedehfaezeh Hosseininoorbin, Mohanad Sarhan, Nour Moustafa, and Marius Portmann. NetFlow V3 NIDS Datasets - The University of Queensland,
-
[2]
Available at: https://staff.itee.uq.edu.au/marius/NIDS_datasets/
-
[3]
Why are my flows different? a tutorial on flow exporters
Gernot Vormayr, Joachim Fabini, and Tanja Zseby. Why are my flows different? a tutorial on flow exporters. IEEE Communications Surveys & Tutorials , 22(3):2064–2103, 2020
work page 2064
-
[4]
Flow-based intrusion detection: Techniques and challenges
Muhammad Fahad Umer, Muhammad Sher, and Yaxin Bi. Flow-based intrusion detection: Techniques and challenges. Computers & Security , 70:238–254, 2017. 19
work page 2017
-
[5]
A survey of network-based intrusion detection data sets
Markus Ring, Sarah Wunderlich, Deniz Scheuring, Dieter Landes, and Andreas Hotho. A survey of network-based intrusion detection data sets. Computers & Security , 86:147–167, 2019
work page 2019
-
[6]
Research trends in network-based intrusion detection systems: A review
Satish Kumar, Sunanda Gupta, and Sakshi Arora. Research trends in network-based intrusion detection systems: A review. IEEE Access, 9:157761–157779, 2021
work page 2021
-
[7]
Oluwadamilare Harazeem Abdulganiyu, Taha Ait Tchakoucht, and Yakub Kayode Saheed. A systematic literature review for network intrusion detection system (ids).International Journal of Information Security , 22(5):1125–1162, 2023
work page 2023
-
[8]
Snort: Lightweight intrusion detection for networks
Martin Roesch et al. Snort: Lightweight intrusion detection for networks. In Lisa, volume 99, pages 229–238, 1999
work page 1999
-
[9]
A review of machine learning-based zero-day attack detection: Challenges and future directions
Yang Guo. A review of machine learning-based zero-day attack detection: Challenges and future directions. Computer communications, 198:175–185, 2023
work page 2023
-
[10]
Review on anomaly based network intrusion detection system
Rafath Samrin and D Vasumathi. Review on anomaly based network intrusion detection system. In 2017 international conference on electrical, electronics, communication, computer, and optimization techniques (ICEECCOT) , pages 141–147. IEEE, 2017
work page 2017
-
[11]
Exploring edge tpu for network intrusion detection in iot
Seyedehfaezeh Hosseininoorbin, Siamak Layeghy, Mohanad Sarhan, Raja Jurdak, and Marius Portmann. Exploring edge tpu for network intrusion detection in iot. Journal of Parallel and Distributed Computing, 179:104712, 2023
work page 2023
-
[12]
The cross-evaluation of machine learning- based network intrusion detection systems
Giovanni Apruzzese, Luca Pajola, and Mauro Conti. The cross-evaluation of machine learning- based network intrusion detection systems. IEEE Transactions on Network and Service Man- agement, 19(4):5152–5169, 2022
work page 2022
-
[13]
Artificial intelli- gence and machine learning in cyber security
Ramjee Prasad, Vandana Rohokale, Ramjee Prasad, and Vandana Rohokale. Artificial intelli- gence and machine learning in cyber security. Cyber security: the lifeline of information and communication technology, pages 231–247, 2020
work page 2020
-
[14]
Flowtransformer: A transformer framework for flow-based network intrusion detection systems
Liam Daly Manocchio, Siamak Layeghy, Wai Weng Lo, Gayan K Kulatilleke, Mohanad Sarhan, and Marius Portmann. Flowtransformer: A transformer framework for flow-based network intrusion detection systems. Expert Systems with Applications , 241:122564, 2024
work page 2024
-
[15]
A review of the advancement in intrusion detection datasets
Ankit Thakkar and Ritika Lohiya. A review of the advancement in intrusion detection datasets. Procedia Computer Science, 167:636–645, 2020. International Conference on Computational Intelligence and Data Science
work page 2020
-
[16]
Sok: Pragmatic assessment of machine learning for network intrusion detection
Giovanni Apruzzese, Pavel Laskov, and Johannes Schneider. Sok: Pragmatic assessment of machine learning for network intrusion detection. In 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P) , pages 592–614, 2023
work page 2023
-
[17]
Netflow datasets for machine learning-based network intrusion detection systems
Mohanad Sarhan, Siamak Layeghy, Nour Moustafa, and Marius Portmann. Netflow datasets for machine learning-based network intrusion detection systems. In Zeng Deze, Huan Huang, Rui Hou, Seungmin Rho, and Naveen Chilamkurti, editors, Big Data Technologies and Appli- cations, pages 117–135, Cham, 2021. Springer International Publishing. 20
work page 2021
-
[18]
Towards a standard feature set for network intrusion detection system datasets
Mohanad Sarhan, Siamak Layeghy, and Marius Portmann. Towards a standard feature set for network intrusion detection system datasets. Mobile networks and applications , pages 1–14, 2022
work page 2022
-
[19]
Cisco Systems NetFlow Services Export Version 9
Benoˆ ıt Claise. Cisco Systems NetFlow Services Export Version 9. RFC 3954, October 2004
work page 2004
-
[20]
Maseer, Robiah Yusof, Baidaa Al-Bander, Abdu Saif, and Qusay Kanaan Kad- him
Ziadoon K. Maseer, Robiah Yusof, Baidaa Al-Bander, Abdu Saif, and Qusay Kanaan Kad- him. Meta-analysis and systematic review for anomaly network intrusion detection systems: Detection methods, dataset, validation methodology, and challenges, 2023
work page 2023
-
[21]
Nour Moustafa and Jill Slay. Unsw-nb15: a comprehensive data set for network intrusion detec- tion systems (unsw-nb15 network data set). In2015 Military Communications and Information Systems Conference (MilCIS), pages 1–6, 2015
work page 2015
-
[22]
Nickolaos Koroniotis, Nour Moustafa, Elena Sitnikova, and Benjamin Turnbull. Towards the development of realistic botnet dataset in the internet of things for network forensic analytics: Bot-iot dataset. Future Generation Computer Systems , 100:779–796, 2019
work page 2019
-
[23]
Nour Moustafa. A new distributed architecture for evaluating ai-based security systems at the edge: Network ton iot datasets. Sustainable Cities and Society , 72:102994, 2021
work page 2021
-
[24]
Toward generating a new intrusion detection dataset and intrusion traffic characterization
Iman Sharafaldin, Arash Habibi Lashkari, Ali A Ghorbani, et al. Toward generating a new intrusion detection dataset and intrusion traffic characterization. ICISSp, 1:108–116, 2018
work page 2018
-
[25]
Seyedehfaezeh Hosseininoorbin, Siamak Layeghy, Brano Kusy, Raja Jurdak, and Marius Port- mann. Harbic: Human activity recognition using bi-stream convolutional neural network with dual joint time–frequency representation. Internet of Things , 22:100816, 2023
work page 2023
-
[26]
Bishop- Hurley, Paul L Greenwood, and Marius Portmann
Seyedehfaezeh Hosseininoorbin, Siamak Layeghy, Brano Kusy, Raja Jurdak, Greg J. Bishop- Hurley, Paul L Greenwood, and Marius Portmann. Deep learning-based cattle behaviour classification using joint time-frequency data representation. Computers and Electronics in Agriculture, 187:106241, 2021
work page 2021
-
[27]
Adnan Shahid Khan, Zeeshan Ahmad, Johari Abdullah, and Farhan Ahmad. A spectrogram image-based network anomaly detection system using deep convolutional neural network.IEEE Access, 9:87079–87093, 2021
work page 2021
-
[28]
S-ads: Spectrogram image- based anomaly detection system for iot networks
Zeeshan Ahmad, Adnan Shahid Khan, Sehrish Aqeel, Azlina Ahmadi Julaihi, Seleviawati Tarmizi, Noralifah Annuar, and Mohammed Sayeeduddin Habeeb. S-ads: Spectrogram image- based anomaly detection system for iot networks. In 2022 Applied Informatics International Conference (AiIC), pages 105–110, 2022
work page 2022
-
[29]
Shahid Tufail, Hugo Riggs, Mohd Tariq, and Arif I. Sarwat. Advancements and challenges in machine learning: A comprehensive review of models, libraries, applications, and algorithms. Electronics, 12(8), 2023
work page 2023
-
[30]
Network-based intrusion detection datasets: A survey
Lubna Ali Hassan Ahmed, Yahia Abdalla Mohamed Hamad, and Ahmed Abdallah Mo- hamed Ali Abdalla. Network-based intrusion detection datasets: A survey. In 2022 Inter- national Arab Conference on Information Technology (ACIT) , pages 1–7, 2022. 21
work page 2022
-
[31]
A detailed analysis of benchmark datasets for network intrusion detection system
Mossa Ghurab, Ghaleb Gaphari, Faisal Alshami, Reem Alshamy, and Suad Othman. A detailed analysis of benchmark datasets for network intrusion detection system. Asian Journal of Research in Computer Science, 7(4):14–33, 2021
work page 2021
-
[32]
Benchmarking the benchmark — comparing synthetic and real-world network ids datasets
Siamak Layeghy, Marcus Gallagher, and Marius Portmann. Benchmarking the benchmark — comparing synthetic and real-world network ids datasets. Journal of Information Security and Applications, 80:103689, 2024
work page 2024
-
[33]
Measuring the complexity of benchmark nids datasets via spectral analysis
Robert Flood and David Aspinall. Measuring the complexity of benchmark nids datasets via spectral analysis. In 2024 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), pages 335–341. IEEE, 2024
work page 2024
-
[34]
Anukool Lakhina, Konstantina Papagiannaki, Mark Crovella, Christophe Diot, Eric D. Ko- laczyk, and Nina Taft. Structural analysis of network traffic flows. SIGMETRICS Perform. Eval. Rev., 32(1):61–72, June 2004
work page 2004
-
[35]
Andersen, Hyong Kim, and Hui Zhang
George Nychis, Vyas Sekar, David G. Andersen, Hyong Kim, and Hui Zhang. An empirical evaluation of entropy-based traffic anomaly detection. In Proceedings of the 8th ACM SIG- COMM Conference on Internet Measurement , IMC ’08, page 151–156, New York, NY, USA,
-
[36]
Association for Computing Machinery
-
[37]
Anukool Lakhina, Konstantina Papagiannaki, Mark Crovella, Christophe Diot, Eric D. Ko- laczyk, and Nina Taft. Structural analysis of network traffic flows. In Proceedings of the Joint International Conference on Measurement and Modeling of Computer Systems , SIGMETRICS ’04/Performance ’04, page 61–72, New York, NY, USA, 2004. Association for Computing Ma- chinery
work page 2004
-
[38]
Flow length and size distributions in campus internet traffic
Piotr Jurkiewicz, Grzegorz Rzym, and Piotr Bory lo. Flow length and size distributions in campus internet traffic. Computer Communications, 167:15–30, 2021
work page 2021
-
[39]
Classifying elephant and mice flows in high-speed scientific networks
Anshuman Chhabra and Mariam Kiran. Classifying elephant and mice flows in high-speed scientific networks. Proc. INDIS, pages 1–8, 2017
work page 2017
-
[40]
Akhtar Ali, Muhammad Imran, and M
Mosab Hamdan, Bushra Mohammed, Usman Humayun, Ahmed Abdelaziz, Suleman Khan, M. Akhtar Ali, Muhammad Imran, and M. N. Marsono. Flow-aware elephant flow detection for software-defined networks. IEEE Access, 8:72585–72597, 2020
work page 2020
-
[41]
An elephant flow detection method based on machine learning
Kaihao Lou, Yongjian Yang, and Chuncai Wang. An elephant flow detection method based on machine learning. In Smart Computing and Communication: 4th International Confer- ence, SmartCom 2019, Birmingham, UK, October 11–13, 2019, Proceedings 4 , pages 212–220. Springer, 2019
work page 2019
-
[42]
Automatic detection of elephant flows through openflow-based openvswitch
Spurthi Mallesh. Automatic detection of elephant flows through openflow-based openvswitch . PhD thesis, Dublin, National College of Ireland, 2017
work page 2017
-
[43]
Slow-paced persis- tent network attacks analysis and detection using spectrum analysis
Li Ming Chen, Shun-Wen Hsiao, Meng Chang Chen, and Wanjiun Liao. Slow-paced persis- tent network attacks analysis and detection using spectrum analysis. IEEE Systems Journal , 10(4):1326–1337, 2016. 22
work page 2016
-
[44]
Network traffic characteristics of data centers in the wild
Theophilus Benson, Aditya Akella, and David A Maltz. Network traffic characteristics of data centers in the wild. In Proceedings of the 10th ACM SIGCOMM conference on Internet measurement, pages 267–280, 2010
work page 2010
-
[45]
The nature of data center traffic: measurements & analysis
Srikanth Kandula, Sudipta Sengupta, Albert Greenberg, Parveen Patel, and Ronnie Chaiken. The nature of data center traffic: measurements & analysis. In Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement , IMC ’09, page 202–208, New York, NY, USA, 2009. Association for Computing Machinery
work page 2009
-
[46]
Cisco systems netflow services export version 9
Benoit Claise. Cisco systems netflow services export version 9. Technical report, Cisco Systems, 2004
work page 2004
-
[47]
On the evaluation of sequential machine learning for network intrusion detection
Andrea Corsini, Shanchieh Jay Yang, and Giovanni Apruzzese. On the evaluation of sequential machine learning for network intrusion detection. In Proceedings of the 16th International Conference on Availability, Reliability and Security , ARES ’21, New York, NY, USA, 2021. Association for Computing Machinery
work page 2021
-
[48]
Stidm: A spatial and temporal aware intrusion detection model
Xueying Han, Rongchao Yin, Zhigang Lu, Bo Jiang, Yuling Liu, Song Liu, Chonghua Wang, and Ning Li. Stidm: A spatial and temporal aware intrusion detection model. In 2020 IEEE 19th International Conference on Trust, Security and Privacy in Computing and Communica- tions (TrustCom), pages 370–377, 2020
work page 2020
-
[49]
Network intrusion detection: Based on deep hierarchical network and original flow data
Yong Zhang, Xu Chen, Lei Jin, Xiaojuan Wang, and Da Guo. Network intrusion detection: Based on deep hierarchical network and original flow data. IEEE Access, 7:37004–37016, 2019
work page 2019
-
[50]
A review of computer vision methods in network security
Jiawei Zhao, Rahat Masood, and Suranga Seneviratne. A review of computer vision methods in network security. IEEE Communications Surveys & Tutorials , 23(3):1838–1878, 2021
work page 2021
-
[51]
Bench- marking datasets for anomaly-based network intrusion detection: Kdd cup 99 alternatives
Abhishek Divekar, Meet Parekh, Vaibhav Savla, Rudra Mishra, and Mahesh Shirole. Bench- marking datasets for anomaly-based network intrusion detection: Kdd cup 99 alternatives. In 2018 IEEE 3rd International Conference on Computing, Communication and Security (IC- CCS), pages 1–8, 2018
work page 2018
-
[52]
Mohanad Sarhan, Siamak Layeghy, and Marius Portmann. Evaluating standard feature sets towards increased generalisability and explainability of ml-based network intrusion detection. Big Data Research, 30:100359, 2022
work page 2022
-
[53]
nprobe, an extensible netflow v5/v9/ipfix probe for ipv4/v6, 2017
Ntop. nprobe, an extensible netflow v5/v9/ipfix probe for ipv4/v6, 2017. Accessed: 2024-05-21
work page 2017
-
[54]
Effects of cyber security knowledge on attack detec- tion
Noam Ben-Asher and Cleotilde Gonzalez. Effects of cyber security knowledge on attack detec- tion. Computers in Human Behavior , 48:51–61, 2015
work page 2015
-
[55]
Non-invasive Moni- toring of Fetal Movements Using Time-Frequency Features of Accelerometry
Siamak Layeghy, Ghasem Azemi, Paul Colditz, and Boualem Boashash. Non-invasive Moni- toring of Fetal Movements Using Time-Frequency Features of Accelerometry. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 4379–
work page 2014
-
[56]
Classification of Fe- tal Movement Accelerometry Through Time-Frequency Features
Siamak Layeghy, Ghasem Azemi, Paul Colditz, and Boualem Boashash. Classification of Fe- tal Movement Accelerometry Through Time-Frequency Features. In 2014 8th International Conference on Signal Processing and Communication Systems (ICSPCS) , pages 1–6. IEEE, 2014. 23
work page 2014
-
[57]
W.E. Leland, M.S. Taqqu, W. Willinger, and D.V. Wilson. On the Self-similar Nature of Ethernet Traffic . IEEE/ACM Transactions on Networking , 2(1):1–15, 1994
work page 1994
-
[58]
Long Term 5G Network Traffic Forecasting via Modeling Non-stationarity with Deep Learning
Yuguang Yang, Shupeng Geng, Baochang Zhang, Juan Zhang, Zheng Wang, Yong Zhang, and David Doermann. Long Term 5G Network Traffic Forecasting via Modeling Non-stationarity with Deep Learning. Communications Engineering, 2(1):33, 2023. 24
work page 2023
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.