Pith. sign in

REVIEW 2 major objections 1 minor 33 references

Comparative Analysis of Machine Learning based Intrusion Detection in Realistic IoT Networks

T0 review · 2 major / 1 minor · reviewed 2026-07-01 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Random Forest reaches 0.99 F1-score classifying attacks in the Gotham2025 IoT dataset.

desk verdict Routine classifier comparison on a new emulated IoT dataset, but the 0.99 F1 result lacks any training or validation details and the testbed realism is unshown. read the letter →

arxiv 2606.31594 v1 pith:52F7BXBD submitted 2026-06-30 cs.CR cs.AI

Rana Alharbi , Chuadhry Mujeeb Ahmed This is my paper · ORCID
classification cs.CRcs.AI
keywords intrusiondetectionIoTnetworksmachinelearningRandomForestGotham2025datasetMQTTCoAPnetworksecurity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper evaluates five machine learning algorithms to detect intrusions in IoT networks. It relies on the Gotham2025 dataset created from a testbed simulating 78 IoT devices communicating via MQTT, CoAP, and RTSP protocols. The analysis identifies Random Forest as the strongest performer, reaching an F1-score of 0.99 for attack classification. A sympathetic reader would care because IoT devices often lack strong built-in security, making external detection systems essential for protecting data and privacy. The work provides evidence that certain ML models can handle the task effectively on realistic traffic patterns.

What carries the argument

Comparative evaluation of five machine learning classifiers on the Gotham2025 dataset from the Gotham testbed emulating 78 IoT devices.

What would settle it

Evaluating the trained models on traffic collected from physical non-emulated IoT devices and checking whether the F1-score stays near 0.99.

Watch

Extended reading notes

Core claim

Through comparative analysis of Random Forest, XGBoost, Logistic Regression, Naive Bayes, and Deep Neural Network on the Gotham2025 dataset, the Random Forest Classifier is shown to be the top-performing model with an F1-score of 0.99 in classifying attacks.

Load-bearing premise

The Gotham2025 dataset from the emulated testbed with 78 devices accurately represents real-world IoT network traffic and attacks.

Editorial extensions

If this is right

  • Random Forest can serve as an effective component in intrusion detection systems for IoT networks.
  • The emulated testbed approach enables reproducible evaluation of security models without needing physical hardware.
  • High F1-scores indicate that ML classifiers can distinguish attacks from normal traffic in multi-protocol IoT setups.
  • Tree-based models like Random Forest may be prioritized for IoT security applications over simpler alternatives.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If performance holds outside the testbed, these classifiers could integrate into edge devices to reduce breach risks in expanding IoT systems.
  • The protocol mix in the dataset suggests the results may extend to applications such as smart homes or medical monitoring.
  • Lightweight versions of the top model could be tested for feasibility on resource-limited IoT hardware.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The manuscript conducts a comparative analysis of five machine learning algorithms (Random Forest, XGBoost, Logistic Regression, Naive Bayes, and Deep Neural Network) for intrusion detection on the Gotham2025 dataset generated from an emulated testbed of 78 IoT devices using MQTT, CoAP, and RTSP protocols. It claims that the Random Forest classifier is the top performer, achieving an F1-score of 0.99 in classifying attacks.

Significance. If the dataset realism and training methodology are substantiated, the empirical comparison could provide practical guidance on model selection for IoT intrusion detection. The work does not include machine-checked proofs, reproducible code releases, or parameter-free derivations.

major comments (2)
  1. [Abstract] Abstract: The reported F1-score of 0.99 for Random Forest is presented without any description of the training procedure, validation strategy (e.g., train/test split or cross-validation), class imbalance handling, or overfitting mitigation, rendering the central performance claim unverifiable from the supplied information.
  2. [Gotham2025 dataset section] Gotham2025 dataset section: The title and abstract position the results as applicable to 'realistic IoT networks,' yet the manuscript supplies no quantitative validation (distributional tests, entropy comparisons, or side-by-side traces) of the emulated traffic against external real-world IoT captures, which is load-bearing for the generalizability claim.
minor comments (1)
  1. [Abstract] Abstract: The list of protocols (MQTT, CoAP, RTSP) is given without indicating the proportion of traffic or attack types per protocol, which would aid interpretation of the results.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major point below and indicate where revisions will be made to improve clarity and transparency.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The reported F1-score of 0.99 for Random Forest is presented without any description of the training procedure, validation strategy (e.g., train/test split or cross-validation), class imbalance handling, or overfitting mitigation, rendering the central performance claim unverifiable from the supplied information.

    Authors: The abstract is intentionally brief, but the full manuscript details the methodology in Section 4: 5-fold stratified cross-validation, SMOTE for class imbalance, and early stopping plus dropout for the DNN to mitigate overfitting. To address the concern, we will revise the abstract to include a concise methods summary: 'Five models were trained and evaluated via 5-fold cross-validation on the Gotham2025 dataset, with class imbalance handled using SMOTE.' This makes the performance claim verifiable from the abstract alone. revision: yes

  2. Referee: [Gotham2025 dataset section] Gotham2025 dataset section: The title and abstract position the results as applicable to 'realistic IoT networks,' yet the manuscript supplies no quantitative validation (distributional tests, entropy comparisons, or side-by-side traces) of the emulated traffic against external real-world IoT captures, which is load-bearing for the generalizability claim.

    Authors: The Gotham2025 dataset is produced from a controlled emulation of 78 IoT devices using standard protocols (MQTT, CoAP, RTSP) to approximate realistic conditions. We agree that direct quantitative comparisons (e.g., distributional tests) to external real-world traces are absent. We will add a dedicated limitations subsection citing prior IoT testbed literature and explicitly qualifying the generalizability claim, while noting that full side-by-side validation would require additional external datasets beyond the scope of this comparative study. revision: partial

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity in empirical ML comparison on independent testbed dataset

full rationale

The paper conducts a standard empirical comparison of five ML classifiers (Random Forest, XGBoost, etc.) on the Gotham2025 dataset generated from an emulated testbed. No derivation chain, equations, or predictions are present that could reduce to inputs by construction. The reported F1-score of 0.99 is a direct evaluation outcome on held-out data, not a fitted parameter renamed as a prediction. No self-citations are load-bearing for any uniqueness claim, and the dataset is described as externally generated rather than constructed via the evaluated models. This is a self-contained empirical study with no circular steps.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The central claim rests on the assumption that the Gotham2025 dataset is a faithful proxy for real IoT attacks and that standard supervised classification metrics are sufficient to rank models for deployment.

assumptions (1)
  • domain assumption The Gotham2025 dataset generated through the Gotham testbed accurately represents realistic IoT network traffic and attacks.
    The paper uses this dataset as the sole basis for model comparison without external validation against real-world IoT deployments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Comparative Analysis of Machine Learning based Intrusion Detection in Realistic IoT Networks." pith.science (2026). https://pith.science/paper/52F7BXBD

@misc{pith2026260631594,
  author       = {Pith},
  title        = {Pith review of: Comparative Analysis of Machine Learning based Intrusion Detection in Realistic IoT Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/52F7BXBD}},
  note         = {Machine review of arXiv:2606.31594}
}
read the original abstract

The Internet of Things (IoT) is rapidly growing and expanding into various sectors, such as healthcare, transportation, smart homes, and more. Despite the benefits of using IoT devices, they present several challenges. Given the significant role these devices play in our lives, it is crucial to address issues related to their security and privacy. These devices are limited in resources, which complicates their security and the protection of the data that they manage. The paper aims to examine intrusion detection systems using the Gotham2025 dataset, generated through the Gotham testbed, which consists of 78 emulated IoT devices utilising various protocols, including MQTT, CoAP, and RTSP, to assist in safeguarding IoT networks from attacks. We conduct a comparative analysis between five machine learning algorithms, including Random Forest, XGBoost, Logistic Regression, Naive Bayes, and Deep Neural Network. We demonstrate that the Random Forest Classifier was the top-performing model, achieving an F1-score of 0.99 in classifying attacks.

Figures

Figures reproduced from arXiv: 2606.31594 by the authors.

Figure 1
Figure 1. Overview of the Methodology [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Class Distribution for Dataset [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Random Forest Classifier Confusion Matrix [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: XGB Classifier Confusion Matrix Class Precision Recall F1-Score Support 0 Brute Force 0.89 0.87 0.88 45,772 1 C&C Communication 1.00 0.99 1.00 6,182 2 DoS 1.00 1.00 1.00 4,367,650 3 Infection 0.99 0.99 0.99 5,888 4 Network Scanning 0.96 0.97 0.96 148,514 5 Normal 1.00 …
Figure 5
Figure 5. Figure 5: Logistic Regression Confusion Matrix Class Precision Recall F1-Score Support 0 Brute Force 0.52 0.27 0.36 45,772 1 C&C Communication 0.87 0.95 0.91 6,182 2 DoS 1.00 1.00 1.00 4,367,650 3 Infection 0.94 0.11 0.19 5,888 4 Network Scanning 0.79 0.92 0.85 148,514 5 Normal …
Figure 6
Figure 6. Figure 6: Naive Bayes Confusion Matrix Class Precision Recall F1-Score Support 0 Brute Force 0.07 1.00 0.14 45,772 1 C&C Communication 0.85 0.95 0.90 6,182 2 DoS 1.00 0.31 0.47 4,367,650 3 Infection 0.70 0.09 0.17 5,888 4 Network Scanning 0.09 0.52 0.15 148,514 5 Normal 0.59 1.0…
Figure 7
Figure 7. Figure 7: Deep Neural Network (DNN) Confusion Matrix [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Models Performance [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 33 canonical work pages

  1. [1]

    Internet-of-things security and vulnerabilities: Case study.Journal of Applied Security Research, 18(3):559–575, 2023

    Ghaida Alqarawi, Bashayer Alkhalifah, Najla Alharbi, and Salim El Khediri. Internet-of-things security and vulnerabilities: Case study.Journal of Applied Security Research, 18(3):559–575, 2023

    work page 2023

  2. [2]

    Internet of things: Applications and challenges in technol- ogy and standardization.Wireless Personal Communications, 58(1):49–69, 2011

    Debasis Bandyopadhyay and Jaydip Sen. Internet of things: Applications and challenges in technol- ogy and standardization.Wireless Personal Communications, 58(1):49–69, 2011

    work page 2011

  3. [3]

    Next-generation internet of things (iot): Opportunities, challenges, and solutions.Sensors, 21(4), 2021

    Yousaf Bin Zikria, Rashid Ali, Muhammad Khalil Afzal, and Sung Won Kim. Next-generation internet of things (iot): Opportunities, challenges, and solutions.Sensors, 21(4), 2021

    work page 2021

  4. [4]

    Hardware, software platforms, operating systems and routing protocols for internet of things applications.Wirel

    Amira Zrelli. Hardware, software platforms, operating systems and routing protocols for internet of things applications.Wirel. Pers. Commun., 122(4):3889–3912, February 2022

    work page 2022

  5. [5]

    Internet of things: A survey on enabling technologies, protocols, and applications.IEEE Communi- cations Surveys & Tutorials, 17(4):2347–2376, 2015

    Ala Al-Fuqaha, Mohsen Guizani, Mehdi Mohammadi, Mohammed Aledhari, and Moussa Ayyash. Internet of things: A survey on enabling technologies, protocols, and applications.IEEE Communi- cations Surveys & Tutorials, 17(4):2347–2376, 2015

    work page 2015

  6. [6]

    Chuadhry Mujeeb Ahmed, Gauthama Raman M R, and Aditya P. Mathur. Challenges in machine learning based approaches for real-time anomaly detection in industrial control systems. InPro- ceedings of the 6th ACM on Cyber-Physical System Security Workshop, CPSS ’20, page 23–29, New York, NY, USA, 2020. Association for Computing Machinery

    work page 2020

  7. [7]

    Machine learning in iot security: Current solutions and future challenges.IEEE Communications Surveys & Tutorials, 22(3):1686–1721, 2020

    Fatima Hussain, Rasheed Hussain, Syed Ali Hassan, and Ekram Hossain. Machine learning in iot security: Current solutions and future challenges.IEEE Communications Surveys & Tutorials, 22(3):1686–1721, 2020

    work page 2020

  8. [8]

    Scanning the cycle: Timing-based authentication on plcs

    Chuadhry Mujeeb Ahmed, Martin Ochoa, Jianying Zhou, and Aditya Mathur. Scanning the cycle: Timing-based authentication on plcs. InProceedings of the 2021 ACM Asia Conference on Computer and Communications Security, ASIA CCS ’21, page 886–900, New York, NY, USA, 2021. Association for Computing Machinery

    work page 2021

Show all 33 references
  1. [9]

    Gotham dataset 2025: A reproducible large-scale iot network dataset for intrusion detection and security research, 2025

    Othmane Belarbi, Theodoros Spyridopoulos, Eirini Anthi, Omer Rana, Pietro Carnelli, and Aftab Khan. Gotham dataset 2025: A reproducible large-scale iot network dataset for intrusion detection and security research, 2025

    2025

  2. [10]

    Gotham testbed: A reproducible iot testbed for security experiments and dataset generation.IEEE Transactions on Dependable and Secure Computing, 21(1):186–203, 2024

    Xabier S´ aez-de C´ amara, Jose Luis Flores, Crist´ obal Arellano, Aitor Urbieta, and Urko Zurutuza. Gotham testbed: A reproducible iot testbed for security experiments and dataset generation.IEEE Transactions on Dependable and Secure Computing, 21(1):186–203, 2024

    2024

  3. [11]

    Internet of things and ransomware: Evolution, mitigation and prevention.Egyptian Informatics Journal, 22(1):105–117, 2021

    Mamoona Humayun, NZ Jhanjhi, Ahmed Alsayat, and Vasaki Ponnusamy. Internet of things and ransomware: Evolution, mitigation and prevention.Egyptian Informatics Journal, 22(1):105–117, 2021

    2021

  4. [12]

    Alex Halderman, Luca Invernizzi, Michalis Kallitsis, Deepak Kumar, Chaz Lever, Zane Ma, Joshua Mason, Damian Menscher, Chad Seaman, Nick Sullivan, Kurt Thomas, and Yi Zhou

    Manos Antonakakis, Tim April, Michael Bailey, Matt Bernhard, Elie Bursztein, Jaime Cochran, Zakir Durumeric, J. Alex Halderman, Luca Invernizzi, Michalis Kallitsis, Deepak Kumar, Chaz Lever, Zane Ma, Joshua Mason, Damian Menscher, Chad Seaman, Nick Sullivan, Kurt Thomas, and Y...

    2017

  5. [13]

    Limitations of state estimation based cyber attack detection schemes in industrial control systems

    Chuadhry Mujeeb Ahmed, Sridhar Adepu, and Aditya Mathur. Limitations of state estimation based cyber attack detection schemes in industrial control systems. In2016 Smart City Security and Privacy Workshop (SCSP-W), pages 1–5, 2016

    2016

  6. [14]

    Ghorbani

    Barjinder Kaur, Sajjad Dadkhah, Farzaneh Shoeleh, Euclides Carlos Pinto Neto, Pulei Xiong, Shahrear Iqbal, Philippe Lamontagne, Suprio Ray, and Ali A. Ghorbani. Internet of things (iot) security dataset evolution: Challenges and future directions.Internet of Things, 22:100780, 2023

    2023

  7. [15]

    Frank Chen

    Mohammad Shahin, Mazdak Maghanaki, Ali Hosseinzadeh, and F. Frank Chen. Advancing net- work security in industrial iot: A deep dive into ai-enabled intrusion detection systems.Advanced Engineering Informatics, 62:102685, 2024

    2024

  8. [16]

    Mohamed Faisal Elrawy, Ali Ismail Awad, and Hesham F. A. Hamed. Intrusion detection systems for iot-based smart environments: a survey.Journal of Cloud Computing, 7(1):21, 2018

    2018

  9. [17]

    A review on evolving domains of internet of things: Architecture, applications, and technical challenges.International Journal of Communication Systems, 36(18):e5613, 2023

    Jahanvi Sharma, Anju Sangwan, and Rishi Pal Singh. A review on evolving domains of internet of things: Architecture, applications, and technical challenges.International Journal of Communication Systems, 36(18):e5613, 2023

    2023

  10. [18]

    Machine learning for intru- sion detection in industrial control systems: challenges and lessons from experimental evaluation

    Gauthama Raman MR, Chuadhry Mujeeb Ahmed, and Aditya Mathur. Machine learning for intru- sion detection in industrial control systems: challenges and lessons from experimental evaluation. Cybersecurity, 4(1):27, 2021

    2021

  11. [19]

    Gutierrez

    Andrea Pinto, Luis-Carlos Herrera, Yezid Donoso, and Jairo A. Gutierrez. Survey on intrusion detection systems based on machine learning techniques for the protection of critical infrastructure. Sensors, 23(5), 2023

    2023

  12. [20]

    Usman Adedayo Adeniyi and Akinyemi Moruff Oyelakin. A survey on promising datasets and recent machine learning approaches for the classification of attacks in internet of things.Journal of Information Technology and Computing, 4(2):31–38, Dec. 2023

    2023

  13. [21]

    Ghorbani

    Euclides Carlos Pinto Neto, Sajjad Dadkhah, Raphael Ferreira, Alireza Zohourian, Rongxing Lu, and Ali A. Ghorbani. Ciciot2023: A real-time dataset and benchmark for large-scale attacks in iot environment.Sensors, 23(13), 2023

    2023

  14. [22]

    Attack rules: an adversarial approach to generate attacks for industrial control systems using machine learning

    Muhammad Azmi Umer, Chuadhry Mujeeb Ahmed, Muhammad Taha Jilani, and Aditya P Mathur. Attack rules: an adversarial approach to generate attacks for industrial control systems using machine learning. InProceedings of the 2th Workshop on CPS&IoT Security and Privacy, pages 35–40, 2021

    2021

  15. [23]

    A survey on intrusion detection system in iot networks.Cyber Security and Applications, 3:100082, 2025

    Md Mahbubur Rahman, Shaharia Al Shakil, and Mizanur Rahman Mustakim. A survey on intrusion detection system in iot networks.Cyber Security and Applications, 3:100082, 2025

    2025

  16. [24]

    MQTT Version 5.0.https://docs

    Andrew Banks, Ed Briggs, Ken Borgendale, and Rahul Gupta. MQTT Version 5.0.https://docs. oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html, March 2019. OASIS Standard

    2019

  17. [25]

    The Constrained Application Protocol (CoAP)

    Zach Shelby, Klaus Hartke, and Carsten Bormann. The Constrained Application Protocol (CoAP). RFC 7252, June 2014

    2014

  18. [26]

    Real Time Streaming Protocol (RTSP)

    Anup Rao, Rob Lanphier, and Henning Schulzrinne. Real Time Streaming Protocol (RTSP). RFC 2326, April 1998

    1998

  19. [27]

    O’Reilly Media, Inc., 2nd edition, 2019

    Aurelien Geron.Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems. O’Reilly Media, Inc., 2nd edition, 2019

    2019

  20. [28]

    Mitchell

    Tom M. Mitchell. Generative and discriminative classifiers: Naive bayes and logistic regression. https://www.cs.cmu.edu/~tom/mlbook/NBayesLogReg.pdf, 2020. Supplementary chapter to *Ma- chine Learning*

    2020

  21. [29]

    Scikit- learn: Machine learning in Python.Journal of Machine Learning Research, 12:2825–2830, 2011

    Fabian Pedregosa, Ga¨ el Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexan- dre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and ´Edouard Duc...

    2011

  22. [30]

    Transforming large-size to lightweight deep neural networks for iot applications.ACM Comput

    Rahul Mishra and Hari Gupta. Transforming large-size to lightweight deep neural networks for iot applications.ACM Comput. Surv., 55(11), February 2023

    2023

  23. [31]

    Performance anal- ysis and comparison of machine and deep learning algorithms for iot data classification.ArXiv, abs/2001.09636, 2020

    Meysam Vakili, Mohammad Khosheghbal Ghamsari, and Masoumeh Rezaei. Performance anal- ysis and comparison of machine and deep learning algorithms for iot data classification.ArXiv, abs/2001.09636, 2020

    2001

  24. [32]

    Yu Han, Aaron Ceross, and Jeroen H. M. Bergmann. AI for regulatory affairs: Balancing accuracy, interpretability, and computational cost in medical device classification, 2025

    2025

  25. [33]

    Federated machine learning to enable intrusion detection systems in iot networks.Electronics, 14(6), 2025

    Mark Devine, Saeid Pourroostaei Ardakani, Mohammed Al-Khafajiy, and Yvonne James. Federated machine learning to enable intrusion detection systems in iot networks.Electronics, 14(6), 2025

    2025

Pith tools

Reviewed July 1, 2026 · model on record in the stance chip above.