{"id":"7a2f0396-c4c3-4722-b2ea-9854a31ac83d","arxiv_id":"2506.19877","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A benchmark finds supervised MLP and CNN detect known attacks almost perfectly but miss most novel attacks, while OCSVM generalizes better to unseen threats.","lead":"This paper compares four machine learning models on the CICIDS2017 network traffic dataset, testing them on both known and novel attack types. It reports that supervised models detect known attacks well but miss most novel attacks, while a one-class SVM offers the most balanced performance across both scenarios.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OCSVM's 'best for unseen attacks' conclusion is driven by one of three held-out types (Slowhttptest); its per-class Bot accuracy is 4.4%, so the generalization claim is unsupported.","rationale":"The reader's weakest assumption identifies the representativeness of the three held-out attack types and the absence of deduplication. I agree that representativeness is a serious problem, and the more decisive evidence is internal: Table IV itself shows OCSVM's pooled unknown-set advantage is driven by DoS Slowhttptest, while its per-class accuracy on Bot is only 0.0443. This does not require external data about CICIDS2017 duplicates; it is visible in the paper's own numbers. The paper offers no macro-averaged per-attack-type result, no leave-one-out experiment, and no confidence intervals, so the headline conclusion that OCSVM is 'the most practical approach for real-world intrusion detection' is not supported. Deduplication and threshold calibration are secondary but would also need to be addressed. Since the reported evidence is internally insufficient for the central claim, REJECT is the appropriate verdict, consistent with the reader's overall assessment.","tokens_in":9589,"tokens_out":7119,"duration_ms":73901,"concrete_test":"Recompute the Unknown Attack Test Set metrics three ways: (1) macro-average per attack type, weighting slowloris, Slowhttptest, and Bot equally; (2) leave-one-out evaluation with a different set of attacks excluded from training and tested (e.g., DDoS, PortScan, FTP-Patator); (3) recalibrate each supervised model's decision threshold to match OCSVM's false-positive rate before comparing F1. If OCSVM does not remain best under macro-averaging or a different holdout set, the paper's practical recommendation should be withdrawn.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim that OCSVM is the most practical model for real-world intrusion detection rests on its Unknown Attack Test Set performance (Table III: F1=0.7575). But the pooled unknown-attack metrics are micro-averages over three attack types with very different sample sizes, and DoS Slowhttptest (5,499 of 13,261 attack samples, 41.5%) dominates. Table IV shows OCSVM's per-class accuracy is 0.9480 on Slowhttptest but only 0.0443 on Bot, whereas LOF scores 0.4680 on Bot. Thus OCSVM's leading F1 is largely an artifact of one easy attack type; the paper's claim that OCSVM 'maintains strong detection rates... even when facing unknown attacks' is contradicted by its near-total failure on the Bot class. The three held-out attacks are also too narrow to represent 'unseen malicious behaviors' in general, and no leave-one-out analysis or macro-averaged per-attack-type summary is provided. Without such evidence, the practical recommendation exceeds what the reported per-class results support.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares four classifiers—MLP, 1D-CNN, OCSVM, and LOF—on the CICIDS2017 dataset in two scenarios: detecting known attack types and detecting three held-out 'unknown' attack types (DoS slowloris, DoS Slowhttptest, Bot). It reports that supervised MLP and CNN achieve high accuracy on known attacks but collapse on unknown ones, LOF gives moderate accuracy with high recall on unknown attacks, and OCSVM best balances precision and recall, leading the authors to recommend OCSVM as the most practical approach for real-world intrusion detection.","tokens_in":9804,"tokens_out":4552,"duration_ms":45326,"significance":"If the central claim were supported, a controlled comparison of supervised, boundary-based, and density-based detectors on a standard benchmark would be a useful practical reference for IDS design. The paper has some strengths: a fixed random seed for reproducible splits, binary classification metrics, and per-attack-type accuracy breakdowns. However, the main conclusion is not supported by the paper's own tables, and the experimental design has unresolved methodological issues. The contribution is therefore largely a negative or cautionary result unless the analysis is substantially reworked.","major_comments":[{"comment":"The central recommendation that OCSVM is the most practical model is contradicted by the paper's own results. On the Overall Test Set, OCSVM achieves F1=0.5520 with recall 0.4784, while MLP and CNN achieve F1=0.9446 and 0.9160, respectively; this is not 'robust detection across both scenarios' as claimed in the Summary of Observed Trends. On the Unknown Attack Test Set, OCSVM's pooled F1=0.7575 is a micro-average dominated by DoS Slowhttptest (5,499 of 13,261 attack samples), yet Table IV shows OCSVM's per-class accuracy on Bot is only 0.0443, compared with LOF's 0.4680. The claim that OCSVM 'maintains strong detection rates... even when facing unknown attacks' is therefore unsupported; the authors should report macro-averaged per-attack-type metrics and a leave-one-out analysis before drawing this conclusion.","section":"§IV, Tables II–IV and §V"},{"comment":"The construction of the Unknown Attack Test Set is a convenience sample. The three held-out attack types are chosen because their sample sizes are 'moderate,' but no argument is given that they are representative of unseen attacks in general, and all three may share distributional properties (low-rate or botnet-like traffic). Without leave-one-out evaluation or additional held-out attack combinations, the paper cannot support the broad claim that OCSVM generalizes to 'previously unseen threats.'","section":"§III-A"},{"comment":"No deduplication of CICIDS2017 is reported. CICIDS2017 is known to contain duplicate flows, and leaving duplicates in the training and test splits can inflate accuracy and mask true generalization performance. The authors should state whether duplicates were removed and, if not, rerun the experiments on a deduplicated version of the dataset.","section":"§III-A"},{"comment":"The description of StandardScaler usage is ambiguous for the unsupervised models. The text says the scaler is fit on 'the training set' and then applied to both test splits; for OCSVM and LOF the training set is benign-only, while for MLP and CNN it includes known attack classes. Since the test splits contain attack features far outside the benign training range, the choice of scaling distribution can materially change anomaly scores. Please specify exactly which samples were used to fit the scaler for each model and justify that choice.","section":"§III-A"},{"comment":"No variance or significance information is provided. All experiments use a single split with random_state=42, with no repeated runs, bootstrap confidence intervals, or statistical tests. Given the class imbalance and the small per-class counts for rare attacks (e.g., Bot, Web Attack – SQL Injection), the reported differences between models, and especially the OCSVM-versus-LOF comparison on unknown attacks, may not be stable.","section":"§IV"}],"minor_comments":[{"comment":"The CNN subsection opens with an incomplete phrase, 'benign tr,' which appears to be a fragment of a sentence; it should be completed.","section":"§III-C2"},{"comment":"The confusion-matrix numbers for MLP on the Overall Test Set (TN=453,433, FP=1,187, FN=11,771, TP=110,367) sum to 576,758, which does not match the reported test-set size; please verify the arithmetic and the split counts.","section":"§IV"},{"comment":"The 'Conclusions from Model Behavior' paragraph repeats the unsupported claim that OCSVM is 'especially well-suited for real-world intrusion detection'; this should be revised to match the evidence presented in Tables II–IV.","section":"§V"},{"comment":"Reference [24] lists an arXiv preprint but the 'Available' field is empty; the URL should be provided.","section":"References"},{"comment":"There is a punctuation and grammar issue in the sentence 'this capability to detect novel threats is well-supported in the anomaly-detection literature [4]. Any significant deviation...'—the period before 'Any' should likely be a comma or semicolon.","section":"§II"}],"recommendation":"reject","confidential_remarks":"The main empirical claim is contradicted by the paper's own Table IV (OCSVM per-class accuracy on Bot is 0.0443), and the pooled unknown-attack result is dominated by one easy attack type. The experiment also lacks deduplication, repeated runs, and a principled unknown-attack selection. These are load-bearing issues, not presentation fixes. A resubmission that reframes the conclusion to a narrower, evidence-based claim and adds macro-averaged, deduplicated, and significance-aware results could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — the useful part of this paper is Table IV, the per-class accuracy on CICIDS2017 for four models. The headline recommendation, however, is not supported by that table: OCSVM is billed as the most practical model, yet its per-class accuracy on the Bot class is 4.4% while LOF manages 46.8%. The pooled unknown-attack F1 that drives the OCSVM claim is micro-averaged, and DoS Slowhttptest (41.5% of the attack samples) dominates it. OCSVM gets 94.8% on Slowhttptest, 57.3% on slowloris, and 4.4% on Bot. So the 'robust detection of unseen attacks' claim is essentially an artifact of one easy attack type.\n\nWhat the paper does well: it is clearly written, specifies hyperparameters, random seeds, and the AWS environment, and it reproduces a known result — supervised models (MLP, CNN) collapse on held-out attack types while unsupervised methods retain some recall. The per-class table is genuinely informative for model selection, though the models are standard and the dataset is a single benchmark.\n\nSoft spots, in order of severity. First, the central conclusion is contradicted by the paper's own Table IV; no macro-averaged or per-attack-type summary on the unknown set is provided, so the aggregate favors the largest class. Second, the three held-out attacks (slowloris, Slowhttptest, Bot) are a convenience sample, not a principled representation of 'unseen malicious behaviors'; no leave-one-out analysis. Third, CICIDS2017 is used without deduplication, which is known to inflate accuracy. Fourth, there are no error bars or repeated runs; everything rests on random_state=42. Fifth, the finding itself is not new: refs [5] and [8] already report that supervised models fail on unseen attacks while unsupervised/semi-supervised methods generalize. The paper adds a per-class breakdown and a specific model quartet, but that is incremental. The reference list also pads with unrelated work (recommender unlearning, autonomous driving, credit risk) that does not support the IDS discussion.\n\nWho is this for? A reader who wants a quick, reproducible comparison of four off-the-shelf models on CICIDS2017 might use Table IV. The paper as it stands should not be accepted because the main claim is wrong. But the per-class evidence is useful and could support a corrected version. I would send this to review with a request for major revision: report macro-averaged metrics on the unknown set, deduplicate, add runs/error bars, and rewrite the conclusion to match the per-class numbers. As is, reject; as a candidate to revise, it is worth referee attention.","headline":"Useful per-class benchmark on CICIDS2017, but the OCSVM 'most practical' conclusion is not supported by the paper's own numbers—Bot accuracy is 4.4%.","tokens_in":10319,"tokens_out":4378,"would_cite":false,"duration_ms":44393,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A one-class SVM trained only on benign network traffic detects previously unseen attack types more reliably than supervised deep-learning models on CICIDS2017.","keywords":["Intrusion Detection System (IDS)","Anomaly Detection","Unsupervised Learning","CICIDS2017","Network Security","Machine Learning","One-Class SVM","Local Outlier Factor"],"falsifier":"Run the same comparison on a deduplicated CICIDS2017 split plus a second benchmark such as UNSW-NB15, holding out a wider and more varied set of unseen attack types (including low-volume, stealthy ones); if OCSVM's F1 on unknown attacks no longer exceeds MLP/CNN or falls below them, the paper's central generalization claim is falsified.","tokens_in":9364,"feed_emoji":"🛡️","tokens_out":5164,"duration_ms":52262,"temperature":0.7,"pith_summary":"This paper compares four machine-learning models—two supervised deep networks, a one-class support vector machine, and a density-based outlier detector—on the CICIDS2017 network-traffic dataset. The authors hold out three attack types as \"unknown\" and find that supervised models (MLP, CNN) excel on familiar attacks but largely fail on novel ones, with recall dropping to 0.175 and 0.195. The one-class SVM, trained only on benign traffic, achieves the best balance on the unknown-attack test set (F1 0.758) and is argued to be the most practical model for real-world intrusion detection. The paper aims to establish that learning a boundary around normal behavior generalizes better to unseen threats than learning signatures of known attacks.","feed_headline":"A benign-only model beats deep nets at spotting novel attacks","feed_subtitle":"On CICIDS2017, an OCSVM trained only on normal traffic best balances precision and recall against unfamiliar threats.","key_machinery":"The load-bearing mechanism is the One-Class SVM, a boundary-based anomaly detector that learns a compact decision surface around normal (benign) traffic in a kernel feature space and flags anything outside that surface as an attack. The experimental setup reinforces this by holding out three attack types—DoS slowloris, DoS Slowhttptest, and Bot—from all training, so the unknown-attack test measures genuine generalization.","core_discovery":"The paper's central discovery is that a One-Class SVM (OCSVM) that trains exclusively on benign traffic detects novel attack types more reliably than supervised deep models: on the Unknown Attack Test Set it reaches Accuracy 0.7919 and F1 0.7575, while MLP and CNN collapse to F1 below 0.33. The authors interpret this as evidence that \"training exclusively on benign data supports better generalization to unseen malicious behaviors,\" and they conclude that boundary-based anomaly detection is the most practical approach for real-world intrusion detection.","pith_inferences":["The three held-out attacks are all relatively high-volume, volumetric or botnet-like, so the generalization advantage may not extend to stealthy, low-rate, or highly evasive attacks; testing on such types would sharpen the claim.","CICIDS2017 is known to contain duplicate flows; deduplicating before splitting could change the reported accuracies, so the numbers should be read as upper-bound estimates.","OCSVM's boundary could be combined with feature selection or ensemble scoring to reduce the precision drop seen on the Overall Test Set (Precision 0.6525), a trade-off the paper does not explore.","A hybrid pipeline that uses supervised models for known attacks and OCSVM for residuals could yield higher combined F1 than either paradigm alone."],"forward_implications":["Deployments facing frequent novel attacks should consider benign-only boundary models like OCSVM over supervised classifiers.","Supervised deep models should not be relied on for zero-day detection unless retrained or augmented with anomaly-based heads.","The F1 gap on unknown attacks quantifies the cost of supervised overfitting: more than 0.4 F1 points on this test.","Tuning OCSVM parameters such as ν and the kernel could further reduce false alarms below the reported 882 benign misclassifications on the unknown-attack set."],"supporting_citations":[{"why":"Supplies the CICIDS2017 dataset and the prior semi-supervised baseline of training on benign data, which the paper adopts for OCSVM and LOF.","marker":"[5]"},{"why":"Provides the one-class support-estimation formulation that underlies the OCSVM decision boundary and the design principle of training on normal data only.","marker":"[13]"},{"why":"Grounds the anomaly-detection approach by surveying how deviations from a normal baseline can reveal novel attacks.","marker":"[4]"},{"why":"Establishes the evaluation metrics and confusion-matrix methodology used for all models in the comparison.","marker":"[10]"}],"fun_headline_variants":["One-class SVM beats deep nets on unseen attack detection","Benign-only training beats deep nets on novel attacks","OCSVM outperforms deep learning on unfamiliar attacks","Unsupervised detector beats deep nets at new attacks","Boundary model bests deep nets on unknown threats"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the three held-out attack types (DoS slowloris, DoS Slowhttptest, Bot) stand in for novel attacks in general, and that CICIDS2017—used without deduplication—gives a valid test distribution for measuring that generalization.","fun_headline_variants_meta":{"raw":{"variants":["One-class SVM beats deep nets on unseen attack detection","Benign-only training beats deep nets on novel attacks","OCSVM outperforms deep learning on unfamiliar attacks","Unsupervised detector beats deep nets at new attacks","Boundary model bests deep nets on unknown threats"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000454,"raw_usage":{"total_tokens":2216,"prompt_tokens":813,"completion_tokens":1403,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":429,"completion_tokens_details":{"reasoning_tokens":1337}},"tokens_in":429,"tokens_out":1403,"duration_ms":9974,"temperature":1.0,"reasoning_tokens":1337,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:15:13.911605+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same comparison on a deduplicated CICIDS2017 split plus a second benchmark such as UNSW-NB15, holding out a wider and more varied set of unseen attack types (including low-volume, stealthy ones); if OCSVM's F1 on unknown attacks no longer exceeds MLP/CNN or falls below them, the paper's central generalization claim is falsified.","supporting_citations":[{"cited_title":"Evaluation of CI- CIDS2017 with qualitative comparison of machine learning algorithm,","cited_arxiv_id":null,"evidence_quote":"Supplies the CICIDS2017 dataset and the prior semi-supervised baseline of training on benign data, which the paper adopts for OCSVM and LOF."},{"cited_title":"Estimating the support of a high-dimensional distribution,","cited_arxiv_id":null,"evidence_quote":"Provides the one-class support-estimation formulation that underlies the OCSVM decision boundary and the design principle of training on normal data only."},{"cited_title":"A detailed analysis of the KDD Cup 99 data set,","cited_arxiv_id":null,"evidence_quote":"Establishes the evaluation metrics and confusion-matrix methodology used for all models in the comparison."}],"review_version":1}