{"id":"8bee35f4-aeeb-4962-a9f7-fd6bd697cc06","arxiv_id":"2509.07016","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":1.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A Random Forest with 20 estimators and depth 10 achieves 99.9998 percent accuracy on SYN DoS attacks from CIC-DDoS2019, with results claimed to generalize to SD-IoV.","lead":"This paper applies a fine-tuned Random Forest classifier to the public CIC-DDoS2019 dataset and reports near-perfect detection of TCP SYN flood attacks with a 0.24 second prediction time. It frames the work as a solution for Software-Defined Internet of Vehicles (SD-IoV), but uses no vehicular traffic data.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Near-perfect results may reflect dataset leakage, not SD-IoV detection; feature list and split protocol are undisclosed.","rationale":"The reader's weakest assumption (dataset transferability) is real, but the more load-bearing issue is that the benchmark evaluation itself may be invalid because the paper does not disclose the feature set or split procedure. If IP/port/timestamp features are retained or flows from the same IP are shared across folds, the reported 0.999998 metrics are expected under leakage and say nothing about any real deployment, including on CIC-DDoS2019's held-out flows from new hosts. The confusion matrix's near-zero errors on 5.9M samples is a red flag that the decision boundary is trivial (e.g., a port value or IP-embedded pattern). Since Algorithm 1's only preprocessing step is StandardScaler (which does not affect decision trees) and there is no mention of dropping identifiers, the risk is substantial. The concrete test would settle it: a grouped split and identifier removal are standard checks. If the test passes, the paper's internal result is likely valid, but the SD-IoV relevance still needs independent vehicular data. Therefore the overall verdict should remain CONDITIONAL, with the condition extended to include a leakage check and a cross-domain evaluation. This partially agrees with the reader: we both flag the dataset gap, but I pin the highest risk on undisclosed feature/split details rather than on representativeness alone.","tokens_in":8272,"tokens_out":7964,"duration_ms":69558,"concrete_test":"Run the reported model on the CIC-DDoS2019 SYN subset twice: (A) replicating the paper's random stratified K-Fold, and (B) using a group split by source IP (all flows from one IP in the same fold) with IP/port/timestamp features removed; also train on CIC-DDoS2019 and test on an independent SD-IoV/VANET SYN flood dataset. If accuracy in (B) or the cross-domain accuracy is substantially below 0.999998, the near-perfect result stems from leakage and the SD-IoV claim is falsified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—0.999998 accuracy/precision/recall/F1 and 0.24 s detection for SYN DoS in SD-IoV—rests on an unverified benchmark-to-domain transfer and, more fundamentally, on the absence of any demonstration that the CIC-DDoS2019 split measures real generalization. The paper never states which of CIC-DDoS2019's 88 features were retained as the '82 features' (Sec. III-B1), and it never says whether IP addresses, ports, or timestamps were discarded. Stratified K-Fold randomly splits flows, so records from the same attack IP or the same traffic burst appear in both training and test folds; a Random Forest with depth 10 can memorize such instance-level fingerprints and produce near-perfect scores (TP 5,867,033, TN 36,180, FP 2, FN 7) that would not survive contact with unseen vehicular traffic. Because the paper uses no SD-IoV-specific data and provides no feature-importance or ablation analysis, the reported metrics could be an artifact of distributional leakage inside one public benchmark. The 'significant advancement' for SD-IoV therefore has no support until this is ruled out.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a fine-tuned Random Forest classifier for detecting TCP SYN flood attacks in Software-Defined Internet of Vehicles (SD-IoV). Using the CIC-DDoS2019 dataset, the authors apply feature scaling, label encoding, and Stratified K-Fold cross-validation, and perform a grid search over the number of estimators, tree depth, and feature-selection strategy. They report near-perfect average accuracy, precision, recall, and F1-score of about 0.999998, a ROC AUC of about 0.99997, and a detection time of 0.24 seconds with the chosen configuration (20 estimators, depth 10). The paper claims this constitutes a significant advancement and a state-of-the-art algorithm for SYN flood detection in SD-IoV.","tokens_in":8493,"tokens_out":3102,"duration_ms":29069,"significance":"If the reported generalization performance could be substantiated on SD-IoV-relevant traffic, the result would be practically interesting: it would show that a simple, fast Random Forest configuration can handle a large-scale, imbalanced attack-detection task with extremely low false-positive and false-negative rates. The internal consistency of the reported confusion matrix with the derived metrics is a strength, and the use of a public benchmark (CIC-DDoS2019) aids reproducibility. However, the paper's central claim is for SD-IoV, and the evidence is entirely from a general-purpose Internet DDoS dataset with no vehicular-network data; this transferability assumption is load-bearing and currently unsupported. The lack of detail on feature handling and the final data split also leaves open a leakage risk that could explain the near-perfect scores.","major_comments":[{"comment":"The central claim is about SYN DoS detection in SD-IoV, but no SD-IoV or vehicular traffic data is used; the paper states that 'the CIC-DDoS2019 dataset is employed to simulate DoS attacks in IoV contexts,' which is an assumption, not a validation. This is load-bearing because the near-perfect metrics are obtained on general Internet DDoS flows, and the paper provides no evidence that these flows are representative of fast-moving vehicle-initiated attacks in an SDN-controlled IoV environment. The authors should either evaluate on an SD-IoV dataset (e.g., generated vehicular traffic or a vehicular-network benchmark) or explicitly reframe the contribution as a benchmark result on CIC-DDoS2019 and remove the 'significant advancement' claim for SD-IoV.","section":"Section III-B1"},{"comment":"The paper does not disclose which 6 of the 88 CIC-DDoS2019 features were discarded to obtain the '82 features,' nor does it state whether instance-level identifiers such as IP addresses, ports, or timestamps were retained. If such identifiers are present, the random Stratified K-Fold split can leak information, because flows from the same attack session or source IP can appear in both training and test folds; a Random Forest can memorize these instance-level fingerprints and produce near-perfect scores that would not transfer to unseen traffic. The authors must report the exact feature list, specify which features were removed and why, and use a split that respects flow/session boundaries (e.g., grouping by source IP or flow ID) to rule out leakage.","section":"Section III-B1 and Algorithm 1"},{"comment":"The paper never states the final train/test split ratio or the size of the held-out test set, and it does not report per-fold metrics or the variance of accuracy, precision, recall, and F1 across folds. The confusion matrix (TP=5,867,033, TN=36,180, FP=2, FN=7) indicates a highly imbalanced test set with roughly 99.4% attack flows, so the near-perfect accuracy is driven by the majority class; the same limitation applies to the majority of the other metrics. Reporting class distributions, the split ratio, and per-fold statistics is necessary to assess whether the result is robust or an artifact of imbalance and a particular split.","section":"Section IV-A and Algorithm 1"},{"comment":"The claimed 'state-of-the-art' and 'significant advancement' status is not supported by the comparison in Table I: the three prior works are evaluated on different datasets and under different protocols, and the table does not include confidence intervals, statistical significance tests, or an evaluation of any baseline method under the same experimental setup. A fair comparison would require running at least one or two baseline classifiers (e.g., decision tree, SVM, or a simpler RF configuration) on the same train/test folds and reporting comparable metrics and timing measurements.","section":"Section IV, Table I"}],"minor_comments":[{"comment":"The dataset name is written inconsistently as both 'CIC-DDoS2019' and 'CCIC-DDoS19'; please use a single consistent name.","section":"Throughout"},{"comment":"The text says 'F1 score merges both precision and recall into a single metric. It is essentially a balanced mean of accuracy and recall,' which is incorrect; the F1 score is the harmonic mean of precision and recall, not a balanced mean of accuracy and recall.","section":"Section III-C"},{"comment":"Figures 2 through 7 are referenced but their captions are generic placeholders and the content is not described in enough detail for a reader to interpret the trends in accuracy, F1, recall, ROC AUC, prediction time, and the confusion matrix; please provide descriptive captions and, where relevant, axis labels.","section":"Section IV"},{"comment":"Reference [12] has a duplicated title fragment; please clean up the citation.","section":"Reference [12]"},{"comment":"Algorithm 1 is described in prose only and is not typeset as an actual algorithm listing; please format it as a proper numbered algorithm with clear input/output and step labels.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"This is a straightforward empirical study whose main weakness is the mismatch between the claimed domain (SD-IoV) and the data used (general-purpose CIC-DDoS2019). The near-perfect results are internally consistent but currently lack the reproducibility details needed to rule out leakage. I think a major revision with added experimental detail, a group-wise split, and either an SD-IoV evaluation or a reframing of the claims is the right path; the paper is not beyond repair."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a plain Random Forest with a small grid search applied to the SYN subset of CIC-DDoS2019. The reported metrics (0.999998 accuracy/precision/recall/F1, confusion matrix TP 5,867,033 / TN 36,180 / FP 2 / FN 7) are internally consistent, and the paper is coherent about what it did. The problem is that what it did is not enough to support what it claims.\n\nWhat is actually new: essentially nothing scientific. Table 1 shows Ma et al. [19] already report 0.9999 accuracy and F1 with RF on the same family of data. The only concrete delta is the detection time (0.24 s vs 0.4 s), but that number is reported without hardware, batch size, or per-fold variance, so it is not a meaningful scientific claim. The hyperparameter selection (20 estimators, depth 10, max_features none) is a minor configuration detail.\n\nWhere the paper does well: the methodology section is transparent about the grid search space, the use of Stratified K-Fold, and the evaluation setup. The confusion matrix is consistent with the metrics, which is more than many papers of this type manage. The comparison table is honest about prior work, even if the framing around it is not.\n\nNow the soft spots, in decreasing order of severity. First, and this is the one that really matters: the paper never says which 82 features were kept from CIC-DDoS2019's 88, and it never says whether IP addresses, ports, or timestamps were discarded. Stratified K-Fold with random shuffling splits flows, so records from the same attack burst or same source IP can appear in both training and test folds. A depth-10 RF can memorize those instance-level fingerprints and produce exactly this kind of near-perfect score. That is not a hypothetical risk for network flow datasets; it is a known failure mode. The stress-test note is right to raise it.\n\nSecond, there is no clear description of a truly held-out test set. Algorithm 1 says \"train on the entire training set\" and \"evaluate on the test set,\" but where that test set comes from is never specified. If the CV folds are used for model selection and the same data are used for final evaluation, the estimates are optimistic.\n\nThird, the SD-IoV framing rests on a single sentence: the CIC-DDoS2019 dataset is used to simulate DoS attacks in IoV contexts. No vehicular traffic, no SDN topology, no mobility model. That is not an evaluation of SD-IoV security; it is a benchmark exercise with a domain label attached.\n\nWho gets value out of this? Someone who wants a baseline RF number on CIC-DDoS2019 SYN flows, or a teaching example of hyperparameter tuning. It is not a substantive contribution to SD-IoV security.\n\nIf I were editor, I would not desk reject it outright, because the central question—does the leak risk invalidate the headline number?—is answerable. I would send it to a referee with a clear request: release the code and preprocessing pipeline, state the feature list, and re-run with group-based splits (e.g., by source IP or by time window). Without those, the 0.999998 is not believable. With them, it might become a useful incremental data point.\n\nBottom line: the paper deserves a serious look, but only with the expectation of substantial revision and toned-down claims.","headline":"Standard RF on CIC-DDoS2019 with internally consistent near-perfect metrics, but the SD-IoV framing and the split protocol are too thin to support the claimed significance.","tokens_in":9016,"tokens_out":3195,"would_cite":false,"duration_ms":30959,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-tuned Random Forest detects SYN DoS attacks in vehicle networks in 0.24 seconds with near-perfect accuracy.","keywords":["Random Forest","SYN flood","Denial of Service","SD-IoV","Stratified K-Fold","CIC-DDoS2019","intrusion detection","detection time"],"falsifier":"Re-run the published 20-estimator, depth-10 Random Forest with the StandardScaler fitted inside each training fold rather than on the full dataset before the split, and also evaluate the model on a held-out capture of real or simulated SD-IoV traffic containing SYN floods; if accuracy drops materially or the scaler leaks, the central claim is unsupported.","tokens_in":8075,"feed_emoji":"🛡️","tokens_out":5874,"duration_ms":46862,"temperature":0.7,"pith_summary":"This paper claims that a deliberately simple Random Forest model can detect TCP SYN flood denial-of-service attacks in software-defined vehicular networks almost perfectly and fast enough for real-time use. After jointly tuning three hyperparameters and validating with Stratified K-Fold cross-validation on the CIC-DDoS2019 dataset, the authors report accuracy, precision, recall, and F1-score all averaging 0.999998, with a detection time of 0.24 seconds. The claim matters because vehicular networks are safety-critical and SYN floods are one of the cheapest attacks to launch against them. The paper positions the 20-estimator, depth-10 configuration as a practical benchmark for SD-IoV intrusion detection.","feed_headline":"Random Forest flags SYN floods in 0.24 seconds","feed_subtitle":"20-tree model on CIC-DDoS2019 hits 0.999998 accuracy, precision, recall, and F1.","key_machinery":"The load-bearing object is the Random Forest ensemble, whose hyperparameters (number of trees, maximum tree depth, and feature selection rule) are tuned jointly by exhaustive search over 4x4x3 combinations. Validation is carried out with Stratified K-Fold cross-validation, which keeps each fold's class distribution identical to the full dataset and is meant to give an unbiased estimate on the imbalanced CIC-DDoS2019 data. StandardScaler feature scaling and label encoding make the numeric features comparable before training. The model's output is judged on accuracy, precision, recall, F1, ROC AUC, and prediction time, and the winning configuration is chosen by highest accuracy with prediction time as the tie-breaker.","core_discovery":"The central discovery, on the paper's own terms, is that a fine-tuned Random Forest with 20 trees, maximum depth 10, and no feature-subset restriction ('None' for max_features) separates SYN attack traffic from normal traffic essentially without error. Evaluated with 5-fold Stratified K-Fold cross-validation on 82 features extracted from CIC-DDoS2019, the model reaches about 0.999998 for accuracy, precision, recall, and F1, and 0.99997 ROC AUC. The confusion matrix aggregated over the folds shows only 2 false positives and 7 false negatives among millions of samples. The authors interpret the 0.24-second prediction time as the key advance over prior Random Forest DDoS detectors, which they report at 0.4 seconds, and conclude this is a state-of-the-art algorithm for SYN DoS detection in SD-IoV.","pith_inferences":["The near-perfect results on CIC-DDoS2019 do not by themselves establish performance on real vehicular traffic; a test on actual SD-IoV captures or a high-fidelity vehicular network simulator would be needed to confirm the SD-IoV claim.","The 0.24 s figure is a batch prediction time on a test set, not a per-packet line-rate measurement; whether it stays below control-loop deadlines in dense vehicle traffic remains open.","If the StandardScaler is fitted before the Stratified K-Fold split, as Algorithm 1 suggests, some cross-validation folds may have seen test-fold statistics during scaling, which would inflate the reported scores; repeating the run with scaling fitted inside each fold is a concrete robustness check.","The same tuning recipe could be tested against other DDoS families in CIC-DDoS2019 (for example, UDP or DNS floods) to see whether the 20/10 configuration generalizes or is specific to SYN traffic."],"forward_implications":["A 20-estimator, depth-10 Random Forest with 'None' feature selection is presented as the recommended configuration for SYN DoS detection on this data, balancing near-perfect scores with 0.24 s prediction time.","With only 2 false positives and 7 false negatives across the cross-validation test set, the model is claimed to be suitable for deployment where missing an attack is costly.","The comparison table places the method ahead of prior Random Forest and alternative classifiers, with detection time halved from 0.4 s to 0.24 s.","Stratified K-Fold validation with reported confusion matrix totals gives a standardized way to evaluate imbalanced attack-detection data.","The detection speed, if it carries to real deployments, would allow SD-IoV controllers to react to SYN floods within the same second the traffic is inspected."],"supporting_citations":[{"why":"Supplies the CIC-DDoS2019 dataset with 88 features, from which 82 SYN-related features are extracted, and appears as the prior Random Forest baseline in the comparison table.","marker":"[19]"},{"why":"Prior work where Random Forest reaches 100% accuracy on SYN-DoS detection; motivates the choice of classifier in this paper.","marker":"[15]"},{"why":"The IoV-focused DDoS detection baseline (Double Deep Q-Network) that the result is compared against and improves on.","marker":"[16]"},{"why":"The Snort-based SVM benchmark included in the comparison table as an alternative detection method.","marker":"[17]"},{"why":"The Stratified K-Fold cross-validation method the paper adopts for robust evaluation on imbalanced data.","marker":"[23]"},{"why":"Supports the use of Random Forest with class-stratified evaluation on imbalanced datasets.","marker":"[22]"},{"why":"Context for lightweight SYN flood detection in SDN; used to motivate the SD-IoV threat scenario.","marker":"[2]"},{"why":"Describes Random Forest as an ensemble algorithm and is cited when the paper explains why RF was selected for DoS detection.","marker":"[20]"}],"fun_headline_variants":["SYN floods ID'd in 0.24s by Random Forest","Random Forest hits 0.999998 accuracy on SYN attacks","20-tree Random Forest stops SYN DoS with 99.9998% accuracy","0.24s SYN flood detection: Random Forest excels in SD-IoV"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's SD-IoV conclusion rests on the assumption that the CIC-DDoS2019 dataset, a general-purpose internet DDoS benchmark, behaves like SYN flood traffic inside a software-defined vehicular network; no vehicular traffic data is used.","fun_headline_variants_meta":{"raw":{"variants":["SYN floods ID'd in 0.24s by Random Forest","Random Forest hits 0.999998 accuracy on SYN attacks","20-tree Random Forest stops SYN DoS with 99.9998% accuracy","0.24s SYN flood detection: Random Forest excels in SD-IoV"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001284,"raw_usage":{"total_tokens":5251,"prompt_tokens":951,"completion_tokens":4300,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":4219}},"tokens_in":567,"tokens_out":4300,"duration_ms":25844,"temperature":1.0,"reasoning_tokens":4219,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:20:10.143272+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the published 20-estimator, depth-10 Random Forest with the StandardScaler fitted inside each training fold rather than on the full dataset before the split, and also evaluate the model on a held-out capture of real or simulated SD-IoV traffic containing SYN floods; if accuracy drops materially or the scaler leaks, the central claim is unsupported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CIC-DDoS2019 dataset with 88 features, from which 82 SYN-related features are extracted, and appears as the prior Random Forest baseline in the comparison table."},{"cited_title":"A SYN flooding attack detection approach with hierarchical policies based on self-information,","cited_arxiv_id":null,"evidence_quote":"Prior work where Random Forest reaches 100% accuracy on SYN-DoS detection; motivates the choice of classifier in this paper."},{"cited_title":"Towards near-real-time intrusion detection for IoT devices using supervised learning and apache spark,","cited_arxiv_id":null,"evidence_quote":"The IoV-focused DDoS detection baseline (Double Deep Q-Network) that the result is compared against and improves on."},{"cited_title":"SKCV: Stratified K-fold cross-validation on ML classifiers for predicting cervical cancer,","cited_arxiv_id":null,"evidence_quote":"The Stratified K-Fold cross-validation method the paper adopts for robust evaluation on imbalanced data."},{"cited_title":"Classification Using Random Forest on Imbalanced Credit Card Transaction Data,","cited_arxiv_id":null,"evidence_quote":"Supports the use of Random Forest with class-stratified evaluation on imbalanced datasets."},{"cited_title":"Lightweight Statistical Approach towards TCP SYN Flood DDoS Attack Detection and Mitigation in SDN Environment,","cited_arxiv_id":null,"evidence_quote":"Context for lightweight SYN flood detection in SDN; used to motivate the SD-IoV threat scenario."},{"cited_title":"Real-Time Detection of DDoS Attacks Based on Random Forest in SDN,","cited_arxiv_id":null,"evidence_quote":"Describes Random Forest as an ensemble algorithm and is cited when the paper explains why RF was selected for DoS detection."}],"review_version":2}