Pith. sign in

REVIEW 4 major objections 5 minor 10 references

Network Threat Detection: Addressing Class Imbalanced Data with Deep Forest

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pairing class-imbalance resampling with the cascade-forest ensemble gcForest detects malicious IoT traffic with near-perfect recall (0.99999) and ROC AUC (0.99983) on the IoT-23 dataset, outperforming logistic regression, SVM, decision…

desk verdict Routine gcForest benchmark on IoT-23 with a plausible but unsupported empirical claim; no split or leakage controls make the headline numbers uninterpretable. read the letter →

arxiv 2506.08383 v1 pith:XUPZZ2EV submitted 2025-06-10 cs.LG cs.CR

Jiaqi Chen , Rongbin Ye This is my paper
classification cs.LGcs.CR
keywords DeepForestMalwareDetectionImbalancedClassificationMachineLearningIoT-23gcNetworkIntrusionSMOTEENN
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a non-neural ensemble can reliably detect malicious traffic in a heavily imbalanced IoT dataset. Using the IoT-23 corpus and three resampling schemes (SMOTE, Hybrid SMOTE-plus-undersampling, and SMOTEENN), it compares logistic regression, a decision tree, an SVM, an MLP, and the cascade-forest ensemble gcForest across 20 experimental conditions. The central claim is that coupling a proper imbalance treatment with an ensemble method, especially gcForest, beats traditional classifiers on this task. A sympathetic reader would care because the reported numbers suggest a low-cost, low-tuning model can catch rare malicious flows almost perfectly while staying stable across resampling choices.

What carries the argument

gcForest (deep forest): a cascade of decision-tree ensembles in which each layer's class-probability outputs are concatenated with the original features and fed to the next layer, plus multi-grained scanning that uses sliding windows to produce richer feature representations. Because it is trained layer-wise without backpropagation, it needs little hyperparameter tuning; the cascade structure is what the paper credits for resistance to overfitting and stability across resampling schemes.

What would settle it

Retrain the same models on a strict time-ordered holdout (e.g., train on earlier flows, test on later flows) or a stratified 70/30 split, compute recall and ROC AUC on the held-out portion, and check whether gcForest's 0.99999 recall and 0.99983 ROC AUC survive; if those numbers drop materially, the central comparison fails.

Watch

Extended reading notes

Core claim

The paper reports that gcForest, an ensemble of decision-tree forests arranged in a cascade with multi-grained scanning, achieves recall 0.99999 and ROC AUC 0.99983 on the IoT-23 dataset when trained with SMOTEENN resampling, and nearly identical scores under Hybrid Sampling and plain SMOTE. This near-perfect recall means the model misses essentially no malicious flows, while its precision (0.88906) keeps the false-alarm burden moderate. Compared with logistic regression, SVM, decision tree, and an MLP on the same data and resampling conditions, gcForest dominates on recall and ROC AUC and is notably more stable across sampling interventions. The paper interprets this as evidence that the deep forest approach is intrinsically resistant to class imbalance and to the choice of resampling strategy, combining the practical advantages of decision-tree ensembles with the representation depth usually associated with neural networks.

Load-bearing premise

The reported near-perfect metrics are computed on a proper evaluation split; the paper never states a train/test split, cross-validation, or leakage controls, and feature importance is ranked on the full labeled dataset before the models are evaluated.

Editorial extensions

If this is right

  • With SMOTEENN, gcForest's recall of 0.99999 would translate to almost no missed malicious IoT flows in a deployed detector.
  • The model's near-identical scores under SMOTE, Hybrid, and SMOTEENN imply that an operational team would not need to fine-tune the resampling strategy.
  • High recall does not collapse precision: an F1 of 0.94127 and precision 0.88906 keep the false-positive burden manageable.
  • A non-neural model with shallow decision trees can match or exceed neural classifiers on this benchmark, which matters for edge and resource-constrained deployments.
  • The rank-based XGBoost/Random Forest feature importance identifies protocol, origin_port, and connection_state as the top three features, pointing to a small feature subset for faster detection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The near-perfect scores likely overstate real-world performance: the paper never specifies a train/test split or cross-validation, and feature importance is ranked on the full dataset before model evaluation, so leakage may inflate the comparison.
  • A natural extension is to recompute feature importance inside cross-validation folds and to evaluate on a time-ordered split of IoT-23, mimicking deployment on traffic that arrives after training.
  • If the stability claim holds beyond IoT-23, gcForest plus SMOTEENN should show similar dominance on other imbalanced security corpora, a testable transfer the paper leaves implicit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes using the deep forest (gcForest) ensemble with class-imbalance resampling for network threat detection on the IoT-23 dataset. It compares three resampling strategies (SMOTE, Hybrid Sampling, SMOTEENN) and five models (SVM, Logistic Regression, Decision Tree, MLP, Deep Forest) across accuracy, F1, precision, recall, and ROC AUC. The central claim is that combining imbalance treatment with ensemble methods, particularly gcForest, yields better detection performance than traditional approaches. The paper also presents a rank-based feature selection procedure using XGBoost and Random Forest, selecting ten features for the models.

Significance. If the reported results were valid, they would be practically significant: a non-neural ensemble achieving near-perfect recall (0.99999) and ROC AUC (0.99983) on rare malicious IoT flows, with stability across resampling choices, would offer a computationally efficient alternative to deep neural networks in security monitoring. The study also uses a public, widely studied dataset and considers multiple sampling strategies and models, which is a reasonable empirical design in principle. However, the manuscript does not describe a valid evaluation protocol, so the reported numbers cannot be taken as generalization estimates. The strength of the claimed contribution is entirely contingent on an evaluation setup that the paper never states.

major comments (4)
  1. [§III-D, Table II] The central claim that Deep Forest 'achieves better detection performance compared to traditional approaches' is contradicted by the paper's own results on the Original data. In Table II, Decision Tree outperforms Deep Forest on accuracy (0.99510 vs. 0.99179), F1 (0.96964 vs. 0.95238), and precision (1.00000 vs. 0.91935). The claimed superiority holds only for recall and ROC AUC, not for detection performance overall. The abstract and Section IV should be revised to reflect the actual comparative results, or the claim must be supported with a justified metric weighting.
  2. [§III-B–§III-D] No train/test split, cross-validation scheme, or temporal partition is described. Sections III-B through III-D report accuracy, F1, precision, recall, and ROC AUC without stating how the data were divided for training and evaluation. Without a clear held-out evaluation protocol, the reported metrics, including the near-perfect recall and ROC AUC for Deep Forest, are uninterpretable as estimates of generalization. This is a load-bearing omission because the paper's only evidence for its central claim is these numbers.
  3. [§III-E] Feature importance is computed on the full labeled dataset using XGBoost and Random Forest, and the resulting top-10 features are presumably used for the Table II models. The manuscript does not state that feature selection was performed only on training folds. If the final feature subset was chosen using the full dataset, then the evaluation data have influenced model construction, constituting a second leakage path independent of the missing train/test split. This must be clarified or the experimental protocol must be changed.
  4. [§III-D, Table II] The Deep Forest metrics under SMOTE, Hybrid Sampling, and SMOTEENN are identical to five decimal places for accuracy, F1, precision, and recall (only ROC AUC varies slightly). The same phenomenon occurs for Decision Tree under all three resampling methods. Given that resampling changes the training distribution, such exact numerical agreement is surprising and suggests either a reporting artifact or an evaluation protocol in which the test set is unaffected by resampling. The authors should explain how the reported metrics are computed and why the results are invariant across resampling schemes.
minor comments (5)
  1. [§III-B] The dataset description does not state the number of flows, the class ratio, or the distribution of benign versus malicious samples. These basic statistics are needed to understand the imbalance and the effect of resampling.
  2. [§III-C] The paper says '20 distinct experiments' but lists five models and four sampling conditions, which is 20 combinations. That wording is fine, but the paper should clarify whether each experiment is run once or with repeated seeds, since no variance or confidence intervals are reported.
  3. [§II] The section title contains a typo: 'DEEPFORST' should be 'DEEPFOREST'. Additionally, the sentence 'As an alternative to traditional deep neural networks (Zhou and Feng, 2017) Deep Forest...' is grammatically incomplete and should be rephrased.
  4. [References] Reference [10] ends with an unrelated bibliographic fragment, 'Trans. Roy. Soc. London, vol. A247, pp. 529–551, April 1955', which appears to be a copy-paste error. The reference list should also be checked for consistent formatting.
  5. [§III-B] The preprocessing section includes an orphaned line 'Identify applicable sponsor/shere. (sponsors)', which appears to be a template remnant and should be removed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular dependency: the paper is an empirical benchmark with external baselines and no fitted-to-prediction reduction.

full rationale

This paper is an empirical comparison of classifiers under different resampling strategies, not a derivation in which an output is constructed from its own input. The main results in Table II are measured performance metrics, and the paper does not define any of these metrics in terms of the models' outputs, nor does it fit a parameter and then rename that fit as a prediction. The feature-importance procedure in Section III-E is a transparent mean-rank aggregation of XGBoost and Random Forest importance scores; even though computing importance on the full labeled dataset may raise evaluation-leakage concerns, that is a data-handling validity issue, not a circularity of the kind where a claimed result is equivalent to its inputs by construction. The Deep Forest method is cited to the external work of Zhou and Feng (arXiv:1702.08835), and the IoT-23 dataset is cited to Stratosphere Laboratory; neither citation is a self-citation bearing the paper's central claim. No equation in the paper sets a predicted quantity equal to a fitted input, and no uniqueness theorem is imported from the authors' own prior work. The near-identical Deep Forest metrics across sampling methods are an empirical observation, not a circular construction. Therefore no significant circularity is present, and the appropriate score is 0.

Assumptions & free parameters 12 free parameters · 4 assumptions · 0 invented entities

The paper introduces no free parameter in the physics sense, but every classifier's hyperparameters are hand-chosen constants that the comparison depends on. The knowledge pulled from prior work includes the gcForest algorithm, the resampling methods, and the IoT-23 dataset labels. The central unstated assumption is that the evaluation is leak-free; the paper also assumes the correctness of the external labels and the fidelity of its deep forest implementation.

free parameters (12)
  • SVM C = 0.001
    Hand-chosen in Table I; no tuning or sensitivity analysis, yet central to SVM results.
  • SVM gamma = 0.001
    Hand-chosen in Table I for the RBF kernel.
  • Decision Tree max_depth = 4
    Hand-chosen in Table I; limits tree complexity.
  • Decision Tree min_samples_split = 10
    Hand-chosen in Table I.
  • Decision Tree ccp_alpha = 0.01
    Cost-complexity pruning parameter, hand-chosen.
  • MLP hidden_layer_sizes = (100, 50)
    Architecture chosen by hand in Table I.
  • MLP alpha = 1.0
    L2 regularization strength, hand-chosen.
  • MLP max_iter = 100
    Optimization cap, hand-chosen.
  • gcForest cascade layers = 8
    Architecture choice in Table I; no explanation or sweep.
  • gcForest n_cascadeRF = 8
    Number of forests per cascade level, hand-chosen.
  • gcForest sliding window = 2
    Multi-grained scanning window size, hand-chosen.
  • Hybrid sampling target size = 12,000 per class
    The paper chooses 12,000 for both classes in hybrid sampling without justification.
assumptions (4)
  • domain assumption IoT-23 labels from Stratosphere Laboratory are accurate ground truth.
    Section III-A treats the dataset labels as correct without external validation.
  • domain assumption The gcForest implementation used matches the description in Zhou and Feng (2017).
    Section II-A describes the cascade and multi-grained scanning, but no code or library version is given; results depend on implementation fidelity.
  • domain assumption Evaluation metrics are computed on a held-out test set independent of training and feature selection.
    Section III-D reports metrics but never states a split or cross-validation; this assumption is load-bearing and unstated.
  • domain assumption XGBoost and Random Forest feature importance rankings are valid for selecting discriminative features.
    Section III-E uses the mean rank across the two algorithms without validating stability or measuring the impact of the selected features.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Network Threat Detection: Addressing Class Imbalanced Data with Deep Forest." pith.science (2026). https://pith.science/paper/XUPZZ2EV

@misc{pith2026250608383,
  author       = {Pith},
  title        = {Pith review of: Network Threat Detection: Addressing Class Imbalanced Data with Deep Forest},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XUPZZ2EV}},
  note         = {Machine review of arXiv:2506.08383}
}
read the original abstract

With the rapid expansion of Internet of Things (IoT) networks, detecting malicious traffic in real-time has become a critical cybersecurity challenge. This research addresses the detection challenges by presenting a comprehensive empirical analysis of machine learning techniques for malware detection using the IoT-23 dataset provided by the Stratosphere Laboratory. We address the significant class imbalance within the dataset through three resampling strategies. We implement and compare a few machine learning techniques. Our findings demonstrate that the combination of appropriate imbalance treatment techniques with ensemble methods, particularly gcForest, achieves better detection performance compared to traditional approaches. This work contributes significantly to the development of more intelligent and efficient automated threat detection systems for IoT environments, helping to secure critical infrastructure against sophisticated cyber attacks while optimizing computational resource usage.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

10 extracted references · 8 canonical work pages

  1. [1]

    and Kamruzzaman J

    Khraisat, A., Gondal, I., Vamplew, P. and Kamruzzaman J. (2019). Survey of intrusion detection systems: techniques, datasets and challenges. Cybersecurity, 2, 20. https://doi.org/10.1186/s42400-019- 0038-7

    work page doi:10.1186/s42400-019- 2019

  2. [2]

    and Moon, J

    Shon, T. and Moon, J. (2007). A hybrid machine learning approach to network anomaly detection. Information Sciences, 177(18), 3799-3821. https://doi.org/10.1016/j.ins.2007.03.025

    work page doi:10.1016/j.ins.2007.03.025 2007

  3. [3]

    and Haque, A

    Zhang, J., Zulkernine, M. and Haque, A. (2008). Random-forests-based network intrusion detection systems. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(5), 649- 659.https://doi.org/10.1109/TSMCC.2008.923876

    work page arXiv 2008

  4. [4]

    Wang, W., Zhu, M.,Zeng, X., Ye, X.,and Sheng, Y. (2017). Malware traffic classification using convolutional neural network for representation learning. 2017 International Conference on Information Networking (ICOIN), Da Nang, Vietnam, 712-717. https://doi.org/10.1109/ICOIN.2017.7899588

    work page arXiv 2017

  5. [5]

    P., and Poornachandran, P

    Vinayakumar, R., Soman K. P., and Poornachandran, P. (2017). Applying convolutional neural network for network intrusion detection. 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI), 1222-1228. https://doi.org/10.1109/ICACCI.2017.8126009

    arXiv 2017

  6. [6]

    Deep Forest

    Zhou, Z., Feng, J., “Deep Forest”, arXiv:1702.08835v4 https://doi.org/10.48550/arXiv.1702.08835

    work page Pith review arXiv doi:10.48550/arxiv.1702.08835

  7. [7]

    and He, J

    Dong, Y., Wang, R. and He, J. (2019). Real-time network intrusion detection system based on deep learning. 2019 IEEE 10th International ConferenceonSoftwareEngineeringandServiceScience(ICSESS),1-4. https://doi.org/10.1109/ICSESS47205.2019.9040718

    work page arXiv 2019

  8. [8]

    Garcia, S., Parmisano, A., and Erquiaga, M. (2020). IoT-23: A labeled dataset with malicious and benign IoT network traffic (Version 1.0.0) [Dataset].Zenodo.http://doi.org/10.5281/zenodo.4743746

    doi:10.5281/zenodo.4743746 2020

Show all 10 references
  1. [9]

    Abdelkhalek, A., and Mashaly, M. (2023). Addressing the class imbalance problem in network intrusion detection systems using data resampling and deep learning. The Journal of Supercomputing, 79, 10611–10644.https://doi.org/10.1007/s11227-023-05073-x

    2023 doi

  2. [10]

    Zhang, Y., and Zhang, Y. (2023). Deep Forest-Based E-Commerce Recommendation Attack Detection Method. Security and Privacy, 6(1), e8413247. https://doi.org/10.1155/2023/8413247 Trans. Roy. Soc. London,vol.A247,pp.529–551,April1955. (references)

    2023 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.