Pith. sign in

REVIEW 4 major objections 5 minor 23 references

Random Forest Stratified K-Fold Cross Validation on SYN DoS Attack SD-IoV

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read A fine-tuned Random Forest detects SYN DoS attacks in vehicle networks in 0.24 seconds with near-perfect accuracy.

desk verdict Standard RF on CIC-DDoS2019 with internally consistent near-perfect metrics, but the SD-IoV framing and the split protocol are too thin to support the claimed significance. read the letter →

arxiv 2509.07016 v1 pith:FIDHXBLD submitted 2025-09-07 cs.CR cs.AIcs.LG

classification cs.CRcs.AIcs.LG
keywords RandomForestSYNfloodDenialofServiceSD-IoVStratifiedK-FoldCIC-DDoS2019intrusiondetectiontime
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a deliberately simple Random Forest model can detect TCP SYN flood denial-of-service attacks in software-defined vehicular networks almost perfectly and fast enough for real-time use. After jointly tuning three hyperparameters and validating with Stratified K-Fold cross-validation on the CIC-DDoS2019 dataset, the authors report accuracy, precision, recall, and F1-score all averaging 0.999998, with a detection time of 0.24 seconds. The claim matters because vehicular networks are safety-critical and SYN floods are one of the cheapest attacks to launch against them. The paper positions the 20-estimator, depth-10 configuration as a practical benchmark for SD-IoV intrusion detection.

What carries the argument

The load-bearing object is the Random Forest ensemble, whose hyperparameters (number of trees, maximum tree depth, and feature selection rule) are tuned jointly by exhaustive search over 4x4x3 combinations. Validation is carried out with Stratified K-Fold cross-validation, which keeps each fold's class distribution identical to the full dataset and is meant to give an unbiased estimate on the imbalanced CIC-DDoS2019 data. StandardScaler feature scaling and label encoding make the numeric features comparable before training. The model's output is judged on accuracy, precision, recall, F1, ROC AUC, and prediction time, and the winning configuration is chosen by highest accuracy with prediction time as the tie-breaker.

What would settle it

Re-run the published 20-estimator, depth-10 Random Forest with the StandardScaler fitted inside each training fold rather than on the full dataset before the split, and also evaluate the model on a held-out capture of real or simulated SD-IoV traffic containing SYN floods; if accuracy drops materially or the scaler leaks, the central claim is unsupported.

Watch

Extended reading notes

Core claim

The central discovery, on the paper's own terms, is that a fine-tuned Random Forest with 20 trees, maximum depth 10, and no feature-subset restriction ('None' for max_features) separates SYN attack traffic from normal traffic essentially without error. Evaluated with 5-fold Stratified K-Fold cross-validation on 82 features extracted from CIC-DDoS2019, the model reaches about 0.999998 for accuracy, precision, recall, and F1, and 0.99997 ROC AUC. The confusion matrix aggregated over the folds shows only 2 false positives and 7 false negatives among millions of samples. The authors interpret the 0.24-second prediction time as the key advance over prior Random Forest DDoS detectors, which they report at 0.4 seconds, and conclude this is a state-of-the-art algorithm for SYN DoS detection in SD-IoV.

Load-bearing premise

The paper's SD-IoV conclusion rests on the assumption that the CIC-DDoS2019 dataset, a general-purpose internet DDoS benchmark, behaves like SYN flood traffic inside a software-defined vehicular network; no vehicular traffic data is used.

Editorial extensions

If this is right

  • A 20-estimator, depth-10 Random Forest with 'None' feature selection is presented as the recommended configuration for SYN DoS detection on this data, balancing near-perfect scores with 0.24 s prediction time.
  • With only 2 false positives and 7 false negatives across the cross-validation test set, the model is claimed to be suitable for deployment where missing an attack is costly.
  • The comparison table places the method ahead of prior Random Forest and alternative classifiers, with detection time halved from 0.4 s to 0.24 s.
  • Stratified K-Fold validation with reported confusion matrix totals gives a standardized way to evaluate imbalanced attack-detection data.
  • The detection speed, if it carries to real deployments, would allow SD-IoV controllers to react to SYN floods within the same second the traffic is inspected.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The near-perfect results on CIC-DDoS2019 do not by themselves establish performance on real vehicular traffic; a test on actual SD-IoV captures or a high-fidelity vehicular network simulator would be needed to confirm the SD-IoV claim.
  • The 0.24 s figure is a batch prediction time on a test set, not a per-packet line-rate measurement; whether it stays below control-loop deadlines in dense vehicle traffic remains open.
  • If the StandardScaler is fitted before the Stratified K-Fold split, as Algorithm 1 suggests, some cross-validation folds may have seen test-fold statistics during scaling, which would inflate the reported scores; repeating the run with scaling fitted inside each fold is a concrete robustness check.
  • The same tuning recipe could be tested against other DDoS families in CIC-DDoS2019 (for example, UDP or DNS floods) to see whether the 20/10 configuration generalizes or is specific to SYN traffic.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a fine-tuned Random Forest classifier for detecting TCP SYN flood attacks in Software-Defined Internet of Vehicles (SD-IoV). Using the CIC-DDoS2019 dataset, the authors apply feature scaling, label encoding, and Stratified K-Fold cross-validation, and perform a grid search over the number of estimators, tree depth, and feature-selection strategy. They report near-perfect average accuracy, precision, recall, and F1-score of about 0.999998, a ROC AUC of about 0.99997, and a detection time of 0.24 seconds with the chosen configuration (20 estimators, depth 10). The paper claims this constitutes a significant advancement and a state-of-the-art algorithm for SYN flood detection in SD-IoV.

Significance. If the reported generalization performance could be substantiated on SD-IoV-relevant traffic, the result would be practically interesting: it would show that a simple, fast Random Forest configuration can handle a large-scale, imbalanced attack-detection task with extremely low false-positive and false-negative rates. The internal consistency of the reported confusion matrix with the derived metrics is a strength, and the use of a public benchmark (CIC-DDoS2019) aids reproducibility. However, the paper's central claim is for SD-IoV, and the evidence is entirely from a general-purpose Internet DDoS dataset with no vehicular-network data; this transferability assumption is load-bearing and currently unsupported. The lack of detail on feature handling and the final data split also leaves open a leakage risk that could explain the near-perfect scores.

major comments (4)
  1. [Section III-B1] The central claim is about SYN DoS detection in SD-IoV, but no SD-IoV or vehicular traffic data is used; the paper states that 'the CIC-DDoS2019 dataset is employed to simulate DoS attacks in IoV contexts,' which is an assumption, not a validation. This is load-bearing because the near-perfect metrics are obtained on general Internet DDoS flows, and the paper provides no evidence that these flows are representative of fast-moving vehicle-initiated attacks in an SDN-controlled IoV environment. The authors should either evaluate on an SD-IoV dataset (e.g., generated vehicular traffic or a vehicular-network benchmark) or explicitly reframe the contribution as a benchmark result on CIC-DDoS2019 and remove the 'significant advancement' claim for SD-IoV.
  2. [Section III-B1 and Algorithm 1] The paper does not disclose which 6 of the 88 CIC-DDoS2019 features were discarded to obtain the '82 features,' nor does it state whether instance-level identifiers such as IP addresses, ports, or timestamps were retained. If such identifiers are present, the random Stratified K-Fold split can leak information, because flows from the same attack session or source IP can appear in both training and test folds; a Random Forest can memorize these instance-level fingerprints and produce near-perfect scores that would not transfer to unseen traffic. The authors must report the exact feature list, specify which features were removed and why, and use a split that respects flow/session boundaries (e.g., grouping by source IP or flow ID) to rule out leakage.
  3. [Section IV-A and Algorithm 1] The paper never states the final train/test split ratio or the size of the held-out test set, and it does not report per-fold metrics or the variance of accuracy, precision, recall, and F1 across folds. The confusion matrix (TP=5,867,033, TN=36,180, FP=2, FN=7) indicates a highly imbalanced test set with roughly 99.4% attack flows, so the near-perfect accuracy is driven by the majority class; the same limitation applies to the majority of the other metrics. Reporting class distributions, the split ratio, and per-fold statistics is necessary to assess whether the result is robust or an artifact of imbalance and a particular split.
  4. [Section IV, Table I] The claimed 'state-of-the-art' and 'significant advancement' status is not supported by the comparison in Table I: the three prior works are evaluated on different datasets and under different protocols, and the table does not include confidence intervals, statistical significance tests, or an evaluation of any baseline method under the same experimental setup. A fair comparison would require running at least one or two baseline classifiers (e.g., decision tree, SVM, or a simpler RF configuration) on the same train/test folds and reporting comparable metrics and timing measurements.
minor comments (5)
  1. [Throughout] The dataset name is written inconsistently as both 'CIC-DDoS2019' and 'CCIC-DDoS19'; please use a single consistent name.
  2. [Section III-C] The text says 'F1 score merges both precision and recall into a single metric. It is essentially a balanced mean of accuracy and recall,' which is incorrect; the F1 score is the harmonic mean of precision and recall, not a balanced mean of accuracy and recall.
  3. [Section IV] Figures 2 through 7 are referenced but their captions are generic placeholders and the content is not described in enough detail for a reader to interpret the trends in accuracy, F1, recall, ROC AUC, prediction time, and the confusion matrix; please provide descriptive captions and, where relevant, axis labels.
  4. [Reference [12]] Reference [12] has a duplicated title fragment; please clean up the citation.
  5. [Algorithm 1] Algorithm 1 is described in prose only and is not typeset as an actual algorithm listing; please format it as a proper numbered algorithm with clear input/output and step labels.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported metrics are measured outcomes of an empirical benchmark study, not quantities defined into existence by the paper's own equations or citations.

full rationale

This paper is an empirical machine-learning evaluation rather than a derivational chain. The central quantitative claims (accuracy, precision, recall, F1, ROC AUC, detection time of 0.24 s) are presented as measured results of a Random Forest model trained and evaluated on CIC-DDoS2019 using Stratified K-Fold cross-validation. No equation in the paper defines the reported metric values in terms of the input labels or the hyperparameters; the formulas for precision, recall, and F1 are standard evaluation definitions, and the values themselves come from the confusion matrix counts (TP 5,867,033, TN 36,180, FP 2, FN 7). Hyperparameter selection (estimators 20, depth 10, feature option None) is described as an exhaustive search over combinations and is not a fitted parameter renamed as a prediction. The paper does not invoke a self-citation chain to justify its central claim; the references cited are general prior works or an external comparison baseline (Ma et al.), not the present authors' own unverified uniqueness theorems. The main weaknesses are methodological rather than circular: feature scaling is applied before splitting in Algorithm 1, the specific 82-feature subset is not disclosed, and the use of a general DDoS benchmark to represent SD-IoV SYN flood traffic is an external-validity assumption. These are correctness or reproducibility concerns and do not make the reported result equivalent to its inputs by construction. Therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim depends on three hand-chosen hyperparameters and on the assumption that a general DDoS benchmark represents SD-IoV traffic. No new entities are introduced.

free parameters (3)
  • random_forest.n_estimators = 20
    Selected by grid search over [10,20,50,100] as the best balance of accuracy and prediction time.
  • random_forest.max_depth = 10
    Selected by grid search over [5,10,15,20]. The paper claims this gives near-perfect metrics.
  • random_forest.max_features = None
    The paper concludes the 'None' feature type is optimal (all features at each split).
assumptions (3)
  • domain assumption CIC-DDoS2019 traffic is representative of SYN DoS attacks in SD-IoV.
    The paper states in Sec. III-B1 that DDoS attacks in SD-IoV mirror traditional attacks and uses CIC-DDoS2019 to simulate them, without providing any SD-IoV data or validation.
  • domain assumption The 82 extracted features correctly isolate SYN attack and normal traffic.
    The feature extraction procedure (from 88 to 82 features) is not described, so the reader cannot verify that no information from the test set influenced feature selection or that the features are appropriate.
  • domain assumption Accuracy is a meaningful objective for this imbalanced dataset.
    The test set contains 5.9M attack samples and 36K normal samples, so accuracy is dominated by the majority class. The paper uses accuracy, along with other metrics, to select the model, but does not discuss the class imbalance beyond using stratified folds.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Random Forest Stratified K-Fold Cross Validation on SYN DoS Attack SD-IoV." pith.science (2026). https://pith.science/paper/FIDHXBLD

@misc{pith2026250907016,
  author       = {Pith},
  title        = {Pith review of: Random Forest Stratified K-Fold Cross Validation on SYN DoS Attack SD-IoV},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FIDHXBLD}},
  note         = {Machine review of arXiv:2509.07016}
}
read the original abstract

In response to the prevalent concern of TCP SYN flood attacks within the context of Software-Defined Internet of Vehicles (SD-IoV), this study addresses the significant challenge of network security in rapidly evolving vehicular communication systems. This research focuses on optimizing a Random Forest Classifier model to achieve maximum accuracy and minimal detection time, thereby enhancing vehicular network security. The methodology involves preprocessing a dataset containing SYN attack instances, employing feature scaling and label encoding techniques, and applying Stratified K-Fold cross-validation to target key metrics such as accuracy, precision, recall, and F1-score. This research achieved an average value of 0.999998 for all metrics with a SYN DoS attack detection time of 0.24 seconds. Results show that the fine-tuned Random Forest model, configured with 20 estimators and a depth of 10, effectively differentiates between normal and malicious traffic with high accuracy and minimal detection time, which is crucial for SD-IoV networks. This approach marks a significant advancement and introduces a state-of-the-art algorithm in detecting SYN flood attacks, combining high accuracy with minimal detection time. It contributes to vehicular network security by providing a robust solution against TCP SYN flood attacks while maintaining network efficiency and reliability.

Figures

Figures reproduced from arXiv: 2509.07016 by the authors.

Figure 1
Figure 1. Random Forest Algorithm Process [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 5
Figure 5. ROC AUC [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figures from the paper (1 more)
Figure 7
Figure 7. Figure 7: Confusion Matrix Heatmap [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 17 canonical work pages

  1. [19]

    doi: 10.1109/ICICT55121.2022.10064554

  2. [1]

    Security in VANET,

    R. Hostak and I. Baronak, “Security in VANET,” 2023, doi: 10.21203/rs.3.rs-2406430/v1

  3. [2]

    Lightweight Statistical Approach towards TCP SYN Flood DDoS Attack Detection and Mitigation in SDN Environment,

    S. Batool et al., “Lightweight Statistical Approach towards TCP SYN Flood DDoS Attack Detection and Mitigation in SDN Environment,” Security and Communication Networks, vol. 2022, 2022, doi: 10.1155/2022/2593672

  4. [3]

    Detection and Mitigation of SYN Flooding Attacks through SYN/ACK Packets and Black/White Lists,

    C. H. Yang, J. P. Wu, F. Y. Lee, T. Y. Lin, and M. H. Tsai, “Detection and Mitigation of SYN Flooding Attacks through SYN/ACK Packets and Black/White Lists,” Sensors, vol. 23, no. 8, Apr. 2023, doi: 10.3390/s23083817

  5. [4]

    SDN enabled DDoS attack detection and mitigation for 5G networks,

    B. Aryal, R. Abbas, and I. B. Collings, “SDN enabled DDoS attack detection and mitigation for 5G networks,” JoC, vol. 16, no. 7, pp. 267–275, Jul. 2021, doi: 10.12720/jcm.16.7.267-275

  6. [5]

    TCP and Priority Queue based Emergency Data Transmission in VANETs,

    A. Mukhopadhyay, B. S. Limitha, R. Anusha, and V. Gowthami, “TCP and Priority Queue based Emergency Data Transmission in VANETs,” in 2022 IEEE GCAT

  7. [6]

    Adaptive Congestion Control Mechanism to Enhance TCP Performance in Cooperative IoV,

    T. K. Mishra, K. S. Sahoo, “Adaptive Congestion Control Mechanism to Enhance TCP Performance in Cooperative IoV,” IEEE Access, vol. 11, pp. 9000–9013, 2023, doi: 10.1109/ACCESS.2023.3239302

  8. [7]

    On the realization of VANET using named data networking: On improvement of VANET using NDN-based routing, caching, and security,

    E. T. da Silva, A. L. D. Costa, and J. M. H. de Macedo, “On the realization of VANET using named data networking: On improvement of VANET using NDN-based routing, caching, and security,” IJCS, vol. 35, no. 18, Dec. 2022, doi: 10.1002/dac.5348

Show all 23 references
  1. [9]

    Design and Implementation of TCP-Friendly Meters in P4 Switches,

    S. Y. Wang, H. W. Hu, and Y. B. Lin, “Design and Implementation of TCP-Friendly Meters in P4 Switches,” vol. 28, no. 4, pp. 1885–1898, Aug. 2020, doi: 10.1109/TNET.2020.3002074

  2. [11]

    Detection and Prevention of TCP SYN Flooding Attack in WSN Using Protocol Dependent Detection and Classification System,

    B. J. Santhosh Kumar and K. R. Karthik Gowda, “Detection and Prevention of TCP SYN Flooding Attack in WSN Using Protocol Dependent Detection and Classification System,” in ICDSIS 2022,

  3. [12]

    doi: 10.1109/ICDSIS55133.2022.9915949

  4. [13]

    The potential of SDNs and multihoming in VANETs: A The potential of SDNs and multihoming in VANETs: A Comprehensive Survey Comprehensive Survey

    “The potential of SDNs and multihoming in VANETs: A The potential of SDNs and multihoming in VANETs: A Comprehensive Survey Comprehensive Survey”, doi: 10.36227/techrxiv.24552964.v1

  5. [14]

    Detection of DDOS Attack using Machine Learning Models,

    S. Santhosh, M. Sambath, and J. Thangakumar, “Detection of DDOS Attack using Machine Learning Models,” ICNWC 2023, doi: 10.1109/ICNWC57852.2023.10127537

  6. [15]

    A SYN flooding attack detection approach with hierarchical policies based on self-information,

    J. R. Sun, C. T. Huang, “A SYN flooding attack detection approach with hierarchical policies based on self-information,” ETRI Journal, vol. 44, no. 2, pp. 346–354, Apr. 2022, doi: 10.4218/etrij.2018-0382

  7. [16]

    Towards near-real-time intrusion detection for IoT devices using supervised learning and apache spark,

    V. Morfino and S. Rampone, “Towards near-real-time intrusion detection for IoT devices using supervised learning and apache spark,” vol. 9, no. 3, Mar. 2020, doi: 10.3390/electronics9030444

  8. [17]

    A Transfer Double Deep Q Network Based DDoS Detection Method for Internet of Vehicles,

    Z. Li, Y. Kong, and C. Jiang, “A Transfer Double Deep Q Network Based DDoS Detection Method for Internet of Vehicles,” IEEE Trans Veh Technol, vol. 72, no. 4, pp. 5317–5331, Apr. 2023, doi: 10.1109/TVT.2022.3233880

  9. [18]

    Comparison of Machine Learning Techniques on Snort for Predicting Realtime DoS and Probe Attack,

    A. H. N. Faizi, D. F. Priambodo, and F. D. Rahmawati, “Comparison of Machine Learning Techniques on Snort for Predicting Realtime DoS and Probe Attack,” in ICIMCIS 2022, 2022, pp. 224–229. doi: 10.1109/ICIMCIS56303.2022.10017776

  10. [20]

    Real-Time Detection of DDoS Attacks Based on Random Forest in SDN,

    R. Ma, Q. Wang, X. Bu, and X. Chen, “Real-Time Detection of DDoS Attacks Based on Random Forest in SDN,” Applied Sciences (Switzerland), vol. 13, no. 13, Jul. 2023, doi: 10.3390/app13137872

  11. [21]

    DoS Attack Detection in Wireless Sensor Networks (WSN) Using Hybrid Machine Learning Model,

    G. S. Rao, M. Harshitha, V. R. Joshitha, S. S. Sravya, and M. V Priya, “DoS Attack Detection in Wireless Sensor Networks (WSN) Using Hybrid Machine Learning Model,” in 2023 SPIN, 2023, pp. 384–388. doi: 10.1109/SPIN57001.2023.10117098

  12. [22]

    Classification Using Random Forest on Imbalanced Credit Card Transaction Data,

    H. Aktar, M. A. Masud, N. J. Aunto “Classification Using Random Forest on Imbalanced Credit Card Transaction Data,” (STI), 2021, pp. 1–4. doi: 10.1109/STI53101.2021.9732553

  13. [23]

    SKCV: Stratified K-fold cross-validation on ML classifiers for predicting cervical cancer,

    S. Prusty, S. Patnaik “SKCV: Stratified K-fold cross-validation on ML classifiers for predicting cervical cancer,” vol. 4, 2022, doi: 10.3389/fnano.2022.972421. Author Meth. Acc. Prec. Rec. F1-sc. T(s) Ma et al

  14. [2022]

    doi: 10.1109/GCAT55367.2022.9971951

  15. [2023]

    doi: 10.3390/smartcities6050121

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.