{"id":"d6946338-302d-46c8-9672-a9e085bf95fc","arxiv_id":"2411.14512","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Logistic regression and SVM achieve 98.65% accuracy on a DDoS dataset, but the claim is not reliable due to a non-random-looking test split and duplicated results.","lead":"The paper applies two standard machine learning classifiers, logistic regression and SVM, to a public DDoS attack dataset and reports 98.65% accuracy. The result is undermined by a test set that is not representative of the dataset and by identical confusion matrices for two different models.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set class supports in Tables III/V contradict Section III.D: 187,848 UDP-Flood and 19,637 HTTP-Flood test instances exceed the full dataset totals, and Tables II/IV are cell-for-cell identical.","rationale":"The reader identified the 80/20 split as the weakest assumption. I agree with the direction, but the problem is stronger than non-representativeness: the reported test set contains more UDP-Flood and HTTP-Flood records than the full dataset contains, so the confusion matrices cannot be correct as printed. A label-ordering error could in principle rescue the support counts, because the reported numbers are close to a permuted stratified split, but that would make the printed class names wrong and would require rerunning the evaluation with corrected labels. The identical LR/SVM confusion matrices also require explanation; if both outputs were produced from a single run, the comparative claim is void. The paper has no code, no error bars, and no independent evaluation, and formal verification is absent. Thus the empirical claim is not verifiable from the manuscript. Honest non-finding is not appropriate here because the internal inconsistency is concrete and decisive. The verdict should remain REJECT; if the authors provide reproducible code and corrected labels, the result could be re-evaluated.","tokens_in":7605,"tokens_out":8522,"duration_ms":80914,"concrete_test":"Rerun the described pipeline with sklearn's train_test_split(stratify=True), a fixed random seed, and the same 80/20 split, then compare the resulting class-wise test supports and confusion matrices to Tables II through V. If the reported supports (187,848 UDP-Flood, 19,637 HTTP-Flood) are not reproduced under a consistent label mapping, then the 98.65% accuracy claim is not supported by the paper's stated dataset and procedure.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim is empirical: both classifiers reach 98.65% accuracy, the highest reported for this dataset (Abstract; Section V.C). The only supporting evidence is the confusion matrices (Tables II and IV) and classification reports (Tables III and V). These exhibits are internally inconsistent with the dataset description in Section III.D. The full dataset is stated to contain 1,048,574 records, with 97,521 UDP-Flood and about 1,997 HTTP-Flood records. Yet the reported test set alone supports 187,848 UDP-Flood instances and 19,637 HTTP-Flood instances (row sums of Tables II/IV and support column in Tables III/V). A 20% test subset cannot contain more instances of a class than exist in the entire dataset. This means either the split was not a 20% split of the described data, or the class labels attached to the matrices are wrong. Either way, the 98.65% figure and the per-class precision/recall values cannot be attributed to the named classes as printed. Additionally, Tables II and IV are identical cell-for-cell for two different sklearn estimators; reproducing that by chance on 209,715 test instances is implausible, and the paper provides no evidence that the two models were actually fitted independently. Because the manuscript's entire contribution is the accuracy number and the claimed comparison, this internal failure is load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes logistic regression and support vector machine classifiers for detecting and classifying DDoS attacks (Normal, UDP-Flood, Smurf, SIDDOS, HTTP-Flood) on a dataset of about 1,048,574 records with 27 features. The authors report that both classifiers achieve 98.65% accuracy on a 20% test split, claim this is the highest accuracy on this dataset compared with prior work (MLP: 98.63%, Random Forest: 98.02%, Naive Bayes: 96.91%), and provide confusion matrices and classification reports in Sections V.A and V.B.","tokens_in":7870,"tokens_out":2005,"duration_ms":18591,"significance":"If the reported results were valid, the contribution would be modest: two standard classifiers applied to a public dataset with a small accuracy improvement over an existing MLP result. The paper does not provide code, hyperparameter details beyond a few settings (e.g., gamma=scale, ovo decision function, LBFGS solver), or statistical significance tests, so reproducibility and evidentiary value are limited. The paper does make an explicit and falsifiable comparison claim against prior work, which is a strength in principle, but the internal inconsistencies detailed below destroy the basis for that claim.","major_comments":[{"comment":"The test-set class counts reported in Tables II and III are impossible given the dataset description in Section III.D. The full dataset is stated to contain 97,521 UDP-Flood records and 1,997 HTTP-Flood records. A 20% test split (about 209,715 records) cannot contain 187,848 UDP-Flood instances and 19,637 HTTP-Flood instances, as shown by the row sums in Table II and the support column in Table III. This means the matrix/table labels, the split description, or the dataset description are wrong; in any case, the reported 98.65% accuracy cannot be attributed to the named classes on the described dataset.","section":"Section III.D and Section V.A, Tables II and III"},{"comment":"Tables II and IV are cell-for-cell identical for two different scikit-learn estimators (LogisticRegression with lbfgs and SVC with gamma=scale and ovo decision function). Producing exactly the same 209,715-instance confusion matrix with two distinct models, including all 1,899 misclassifications of HTTP-Flood as UDP-Flood and 800 SIDDOS misclassifications, is extremely implausible. The paper provides no explanation, and this strongly suggests the two results were not obtained from independent model fits, undermining the claimed 'comparative study'.","section":"Tables II and IV (Sections V.A and V.B)"},{"comment":"The central claim that both models achieve 'the highest achieved accuracy' depends entirely on the contradictory confusion matrices and classification reports. Since the class distributions in those exhibits are inconsistent with the dataset totals, the accuracy value of 98.65% is not a meaningful measure of generalization to the stated data distribution. The comparison to previous accuracies (96.91%, 98.02%, 98.63%) therefore cannot be accepted as stated.","section":"Section V.C, Accuracy claim"}],"minor_comments":[{"comment":"The text says 'The SMURF, SIDDOS, and HTTPS are 6211, 3198, and 1997 of the whole observations respectively,' but Table II and the class descriptions use 'HTTP-Flood'; also the sum of the stated class counts (939,648 + 97,521 + 6,211 + 3,198 + 1,997 = 1,048,575) differs by one from the stated total of 1,048,574 records.","section":"Section III.D"},{"comment":"The solver is referred to as 'LBSFGS solver'; the correct scikit-learn name is 'lbfgs'.","section":"Section IV.B"},{"comment":"Several entries in Table I are labeled 'Continuous' despite being symbolic or categorical (e.g., SRC_ADD, DES_ADD, PKT_ID, FLAGS, NODE_NAME_FROM), and the min-max normalization formula is referenced but not actually shown in the text.","section":"Section IV.A and Table I"},{"comment":"Reference [10] contains garbled non-English text instead of a proper title and venue, and reference [6] appears to combine two different papers (Loukas et al. and Sabrina et al.) under a single citation to a 2017 LSTM technical report.","section":"References"}],"recommendation":"reject","confidential_remarks":"The identical confusion matrices in Tables II and IV for two different estimators, combined with test-set class counts that exceed full-dataset class totals, are not the kind of issue that can be fixed by a revision; they call the integrity of the experimental results into question. The manuscript would need to be rerun from scratch with a verifiable train/test split and independently fitted models, and even then the contribution would be incremental. I see no path within the current manuscript's scope to a publishable result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper is a routine benchmark application of logistic regression and SVM to the Alkasassbeh DDoS dataset, and the only new result is that these two classifiers hadn't been run on this particular dataset before. That is a very small contribution, but it isn't nothing: if the numbers were sound, the paper would be a marginally useful data point, comparable to the 98.63% MLP reported earlier.\n\nIt does some things well. The preprocessing pipeline is described plainly: label encoding, min-max normalization, 80/20 split. The related work is a fair sample of prior DDoS detection work, including the paper that created the dataset, and the comparison numbers in Section V.C match what that earlier paper reported. The citation pattern is not a problem.\n\nThe load-bearing problem is that the reported test-set counts cannot come from the dataset as described. Table III and Table V give support values of 187,848 for UDP-Flood and 19,637 for HTTP-Flood, out of 209,715 test instances. Section III.D says the entire dataset contains 97,521 UDP-Flood and 1,997 HTTP-Flood records. A 20% test split cannot contain more instances of a class than exist in the whole dataset. The reader's report flagged a related distribution mismatch; I think the absolute counts are even stronger evidence. The test set is not a random or stratified sample of the described data, and the 98.65% accuracy therefore cannot be attributed to the five classes as labeled.\n\nThe second red flag is that Tables II and IV are identical cell-for-cell for two different sklearn estimators. On 209,715 test instances, exact agreement on every one of the 25 cells is not credible unless the two fits were not actually independent or the tables were copied. The text says the models give 'almost the same' results; the tables show exactly the same results, including the exact same misclassifications. That needs explanation.\n\nThere is no circular reasoning here; the accuracy is empirical. But empirical support collapses when the test-set counts contradict the dataset description. The paper also provides no code, no seeds, and no error bars, so there is no way to verify what was actually split. These are not minor omissions given that the entire contribution is one accuracy number.\n\nThis paper is for a reader who wants a quick benchmark number for this specific dataset. It is not ready for a serious referee. I would desk reject it, and the message back to the authors should be concrete: verify the split, rerun the two models, report the actual test distribution, and explain the identical matrices. If the rerun confirms 98.65% on a representative test set, the benchmark point is worth a short workshop paper. As written, the central claim is unsupported.","headline":"The 98.65% claim is not supported by the paper's own tables, and the identical confusion matrices make the whole empirical section look unreliable.","tokens_in":8381,"tokens_out":3170,"would_cite":false,"duration_ms":29598,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that logistic regression and SVM both reach 98.65% accuracy on a five-class DDoS detection benchmark, the highest reported for that dataset.","keywords":["DDoS detection","logistic regression","support vector machine","multiclass classification","intrusion detection","network security","UDP flood","SIDDOS"],"falsifier":"Re-run both models with a stratified random 80/20 split, or with 10-fold cross-validation, that preserves the dataset's roughly 90% Normal and 9% UDP-Flood proportions, and report per-class recall. If the accuracy drops materially below 98.65%, or if the Normal-class recall changes from the 0.94 shown in Tables III and V, the headline number is an artifact of the skewed test split rather than a property of the classifiers.","tokens_in":7420,"feed_emoji":"🛡️","tokens_out":7777,"duration_ms":70255,"temperature":0.7,"pith_summary":"This paper sets out to show that two standard, inexpensive classifiers—logistic regression and support vector machines—can detect and label distributed denial-of-service attacks in a public dataset of 1,048,574 records described by 27 network attributes. The authors report that both models achieve 98.65% classification accuracy across five classes (normal traffic, UDP flood, Smurf, SIDDOS, and HTTP flood), which they say is the highest accuracy so far reported on this particular dataset, edging past the 98.63% of a multilayer perceptron. The practical interest is that if this holds, a network operator can get near-best detection with a model that is simple to train and explain, without a deep network. The paper also reports per-class precision and recall, showing both models classify UDP flood and HTTP flood well but recover only about a third of SIDDOS attacks.","feed_headline":"Both logistic regression and SVM hit 98.65% DDoS accuracy","feed_subtitle":"On a five-class public benchmark, the two classifiers match or beat prior MLP, random forest, and Naive Bayes scores.","key_machinery":"The load-bearing object is the five-class DDoS dataset taken from the prior study [5]: 1,048,574 rows, 27 continuous and symbolic network attributes plus a class label, with about 90% normal traffic and 10% attacks (UDP flood 9%, with Smurf, SIDDOS, and HTTP flood making up the rest). The argument is carried by a standard supervised pipeline—label encoding of categorical features, min-max normalization to the range 0 to 1, an 80/20 train-test split, and two off-the-shelf classifiers, logistic regression and a support vector machine (a max-margin classifier that separates classes by a learned boundary)—and by confusion matrices and classification reports showing that the two models behave identically on the test set. The reported accuracy is the weighted fraction of test records whose predicted class matches the true class.","core_discovery":"On the authors' own terms, the central discovery is that logistic regression (with an LBFGS solver) and SVM (scikit-learn SVC with gamma=scale and one-vs-one decision function) produce identical performance on the DDoS dataset: the same 98.65% accuracy, the same confusion matrix, and the same per-class precision and recall. Both models correctly classify nearly all UDP-Flood test records and most HTTP-Flood records, but SIDDOS recall is only 0.33, meaning two of every three SIDDOS attacks are missed. Because the two classifiers give the same numbers and the same matrix, the authors treat the result as a mutual confirmation and conclude that these models set a new high mark for this dataset, above the Naive Bayes 96.91%, Random Forest 98.02%, and MLP 98.63% reported for the same data in the original dataset paper.","pith_inferences":["The test set's composition—187,848 UDP-Flood records but only 380 Normal records, against a full dataset that is 90% Normal—suggests the reported accuracy mostly measures the model's ability to recognize UDP flood, not its ability to recognize normal traffic; a stratified random split would be a stricter test.","The fact that two different classifiers produce identical confusion matrices raises the possibility that one dominant class (UDP flood) drives both models' decisions; testing on per-class balanced subsets would show whether the equivalence survives.","A natural extension the authors do not run is feature selection or ablation: with 27 mostly continuous network attributes, removing correlated features (packet counts, rates, sizes) could show which attributes actually carry the DDoS signal.","Combining this dataset with additional DDoS datasets, as the authors say they plan to do, would test whether logistic regression and SVM keep their edge when class proportions and attack mixes change."],"forward_implications":["If the reported accuracy holds, a network intrusion system could use logistic regression instead of a multilayer perceptron and obtain essentially the same accuracy with lower training cost and easier interpretability.","The identical confusion matrices imply that on this dataset two very different classifiers have converged on the same decision behavior, which supports the idea that the 27 features carry enough signal to separate most classes.","The SIDDOS recall of 0.33 means that the headline 98.65% accuracy does not imply dependable detection of every attack class; a deployment focused on SQL-injection DDoS would need additional features or a different model.","The comparison suggests the practical ceiling on this dataset is near 98.6% for simple classifiers, so meaningful further gains would require new features or hybrid methods rather than another off-the-shelf model.","Because the original dataset paper [5] did not report precision and recall for its models, the per-class numbers here are the first direct comparison for those metrics on this data."],"supporting_citations":[{"why":"Supplies the exact dataset (27 attributes, five classes, 1,048,574 records) and the three previous accuracies (Naive Bayes 96.91%, Random Forest 98.02%, MLP 98.63%) against which the paper's 'highest accuracy' claim is measured.","marker":"[5]"},{"why":"Establishes the earlier machine-learning DDoS detection baseline (delta-time features, multiple classifiers, 93.67% best accuracy) that the proposed classifiers are intended to improve upon.","marker":"[4]"},{"why":"Provides the paper's cited grounding for the SVM method and for applying machine learning to DDoS detection, the second classifier in the comparison.","marker":"[7]"}],"fun_headline_variants":["SVM and logistic regression tie at 98.65% DDoS accuracy","Identical DDoS detection: SVM and logistic regression both hit 98.65%","DDoS attacks detected with 98.65% accuracy by both SVM and logistic regression","Two ML models match at 98.65% accuracy on DDoS detection","SVM and logistic regression both reach 98.65% on DDoS benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on assuming that the 80/20 train-test split gives a fair sample of all five classes, but the test set shown in the confusion matrices is overwhelmingly UDP-Flood and nearly empty of Normal records, so the reported 98.65% figure depends on that assumption.","fun_headline_variants_meta":{"raw":{"variants":["SVM and logistic regression tie at 98.65% DDoS accuracy","Identical DDoS detection: SVM and logistic regression both hit 98.65%","DDoS attacks detected with 98.65% accuracy by both SVM and logistic regression","Two ML models match at 98.65% accuracy on DDoS detection","SVM and logistic regression both reach 98.65% on DDoS benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000869,"raw_usage":{"total_tokens":3771,"prompt_tokens":956,"completion_tokens":2815,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":2719}},"tokens_in":572,"tokens_out":2815,"duration_ms":16177,"temperature":1.0,"reasoning_tokens":2719,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:30:34.389706+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run both models with a stratified random 80/20 split, or with 10-fold cross-validation, that preserves the dataset's roughly 90% Normal and 9% UDP-Flood proportions, and report per-class recall. If the accuracy drops materially below 98.65%, or if the Normal-class recall changes from the 0.94 shown in Tables III and V, the headline number is an artifact of the skewed test split rather than a property of the classifiers.","supporting_citations":[{"cited_title":"Accur acy is the measurement of the rate of the correctly classi fied instances","cited_arxiv_id":null,"evidence_quote":"Supplies the exact dataset (27 attributes, five classes, 1,048,574 records) and the three previous accuracies (Naive Bayes 96.91%, Random Forest 98.02%, MLP 98.63%) against which the paper's 'highest accuracy' claim is measured."},{"cited_title":"Unlike SIDDOS attackers se nd valid messages instead of illegitimate messages but at a very slow rate to a web application container web s erver","cited_arxiv_id":null,"evidence_quote":"Establishes the earlier machine-learning DDoS detection baseline (delta-time features, multiple classifiers, 93.67% best accuracy) that the proposed classifiers are intended to improve upon."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the paper's cited grounding for the SVM method and for applying machine learning to DDoS detection, the second classifier in the comparison."}],"review_version":1}