{"id":"f764213b-aedd-4b03-8beb-601f2b56ebb7","arxiv_id":"2501.05387","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":13,"one_line_summary":"An application of standard ensemble classifiers and SHAP to a private dataset reports >99% detection accuracy, but the evaluation may be leaky and no baselines are given.","lead":"This paper applies tree-based ensemble models and SHAP to detect malware in encrypted traffic, reporting over 99% accuracy on two datasets. The authors say their dataset of 1,127 connections is the largest open-source collection, but the data and code are not released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ADASYN oversampling is not specified as nested inside the CV folds; if applied before splitting, the reported >99% metrics are inflated by test-set leakage.","rationale":"The paper's central claim is the near-perfect detection performance on encrypted traffic, so the validity of the evaluation protocol is the most load-bearing condition. The reader identified this precisely: Section 3.6 describes ADASYN oversampling and 10-fold cross-validation without specifying whether oversampling is nested inside the folds, leaving a plausible leakage path that would explain the unusually high and consistent metrics. My reading of the text finds no statement that rules out global oversampling before splitting, and the lack of software or dataset artifacts makes it impossible to verify. Other issues, such as the incorrect formula for Random Forest in Equation 1 and the logical inconsistency in Algorithm 1, are real but secondary because they do not necessarily falsify the reported empirical results as directly as leakage does. The proposed concrete test would settle the matter by reproducing the evaluation under a sound protocol and comparing the metrics. Since this concern matches the reader's weakest assumption and supports the REJECT verdict, the verdict should remain unchanged.","tokens_in":10914,"tokens_out":3423,"duration_ms":33613,"concrete_test":"Re-run the custom-dataset experiment with ADASYN correctly nested inside a 10-fold cross-validation, e.g., using imbalanced-learn's ADASYN inside a scikit-learn Pipeline passed to cross_validate, so that the synthetic sampler is fitted only on the training folds and the held-out fold is never seen during fitting or oversampling. Also report the same metrics with no oversampling and with a plain train/test split. If the XGB accuracy and F1 drop by more than a few percentage points relative to the paper's 99.32%/99.43%, the published results are not reproducible under a leak-free protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.6 states that 'we oversample our normal samples using an adaptive synthetic sampling technique' and then 'We train the model based on 10-fold cross-validation,' but it never states that oversampling is applied independently inside each training fold. The most natural reading is that ADASYN is a global preprocessing step performed before the folds are created. Because the dataset is 98.97% normal and 1.03% malicious, ADASYN generates synthetic samples that depend on nearest neighbors across the entire dataset. If the oversampling is done before the train/test split, synthetic instances derived from test-fold samples appear in the training folds, so the classifier effectively sees information from the test set during training. This would directly inflate the headline 99.32% accuracy and 99.43% F1 on the custom dataset and the >99% figures on CTU-13. The paper provides no code, no released dataset, and no explicit ordering of oversampling relative to splitting, so the reported numbers are not reproducible under a sound protocol. The suspiciously extreme hyperparameters (e.g., max_depth=43 for XGB with only 1,127 samples) are consistent with a model that memorizes a small, possibly leaked training set. A correct protocol would fit ADASYN only on the training portion of each fold and leave the validation/test portion untouched.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an explainable malware-detection pipeline for encrypted network traffic. It constructs bidirectional flows from raw pcap data, extracts multi-view features (handshake metadata, certificate fields, timing and packet-length statistics, Markov-chain-derived states, and connection metadata), trains three tree-ensemble classifiers (Random Forest, XGBoost, Extra Trees), and applies SHAP TreeExplainer for global and local explanations. The authors report near-perfect performance on a self-compiled dataset of 1,127 malicious captures spanning 54 families, and on the CTU-13 dataset, with XGBoost achieving 99.32% accuracy, 99.53% precision, and 99.43% F1-score on the custom dataset. They also claim a dataset contribution and identify maximum packet size, mean inter-arrival time, and TLS version as the most critical features.","tokens_in":11228,"tokens_out":5470,"duration_ms":55224,"significance":"If the evaluation protocol were sound, the paper would be a useful empirical contribution to encrypted-traffic malware detection and XAI: it combines multi-view feature extraction, tree ensembles, a CTU-13 benchmark, MCC reporting, and SHAP-based explanations, and it attempts to assemble a multi-family malware capture set. The main strengths are the breadth of feature families and the use of a public benchmark dataset. However, the headline claims are currently not credible because the relationship between oversampling, hyperparameter selection, and cross-validation is not specified, no same-data baselines are provided, and the self-compiled dataset mixes traffic from different sources in ways that may introduce confounds. The SHAP findings are also presented with causal language that the method does not support. With a corrected evaluation protocol and additional baselines, the central idea could be salvageable, but the present version does not substantiate its central quantitative claims.","major_comments":[{"comment":"The ADASYN oversampling procedure is not described as nested inside the cross-validation folds. The text states that \"we oversample our normal samples using an adaptive synthetic sampling technique\" and then \"We train the model based on 10-fold cross-validation,\" but it never states that oversampling is applied independently to each training fold. Because the dataset is 98.97% normal and 1.03% malicious, ADASYN generates synthetic samples from nearest neighbors computed over the whole dataset; if this is done before splitting, synthetic instances derived from test-fold observations leak into the training folds. This would directly inflate the reported 99% accuracy, precision, and F1 metrics in Section 4.1. The authors must rerun the experiments with oversampling performed only inside each training fold, or use a pipeline that makes the ordering explicit, and report both results.","section":"Section 3.6"},{"comment":"Hyperparameter selection is also not described as nested within the cross-validation procedure. The RF parameters are chosen from \"validation curves\" and the XGB parameters from a \"nature-inspired genetic algorithm,\" but the paper does not state whether these searches are performed inside each fold or on the full dataset. If the hyperparameters are tuned on the full dataset before cross-validation, the reported metrics are optimistically biased. The XGBoost setting max_depth=43 on a dataset of 1,127 samples is particularly concerning and is consistent with memorization rather than generalization. The authors should either use nested cross-validation or fixed, conservatively chosen hyperparameters and report the associated performance.","section":"Section 3.6"},{"comment":"The comparison with existing work is not conducted on the same data or protocol. Section 4.4 cites a single literature accuracy of 96.71% from [34] and states that \"Our system outperforms,\" but no baseline model is trained or evaluated on the custom dataset or on the same CTU-13 split. Different datasets, feature sets, and evaluation protocols make this comparison uninformative. The central claim of superior performance needs same-data baselines, at minimum standard classifiers such as logistic regression or a simple neural network, and ideally prior encrypted-malware-detection methods re-run under the same evaluation pipeline.","section":"Section 4.4"},{"comment":"The self-compiled dataset has a likely source-domain confound. Malicious traffic was collected from six different public repositories, while normal traffic was collected from the Addis Ababa Science and Technology campus network and from CTU-13. A classifier may therefore separate traffic by collection environment, capture time, or IP/TLS fingerprint rather than by maliciousness. The paper provides no per-source evaluation, no domain-adaptation analysis, and no check of whether the same features separate normal and malicious traffic within each source. This threatens both the detection claims and the SHAP-based feature-importance conclusions. The authors should evaluate with held-out sources or include a per-source breakdown.","section":"Sections 3.1 and 4.1"},{"comment":"The abstract claims the compiled dataset has \"1,127 unique connections, more than any other available open-source dataset,\" but Section 3.2 states that the CTU-13 dataset alone contains 38,898 botnet samples and 53,314 normal samples. This is internally inconsistent and the dataset-size claim is factually incorrect. The claim should be removed or replaced with an accurate statement about the number of malware families or the availability of labeled malicious captures.","section":"Abstract and Sections 3.2/3.6"},{"comment":"The SHAP global explanations are descriptive properties of the fitted model, but the discussion interprets them as causal or behavioral facts about malware. For example, Section 4.3 states that \"malware traffic is characterized by a higher volume of packets\" and that \"malware authors tend to prefer older TLS versions.\" These conclusions are not established by SHAP values alone, since SHAP summarizes the model's decision function rather than ground-truth malware behavior. To support such claims, the authors should either validate the features through ablation experiments or rephrase the statements as observations about the trained model.","section":"Sections 4.2 and 4.3"}],"minor_comments":[{"comment":"There is a grammatical error in the sentence \"By observing the disparities between malicious and normal network flow's contextual information were can capture strong discriminatory feature set\"; it should be \"we can capture.\" Similar language issues appear elsewhere and should be corrected in a thorough copyedit.","section":"Section 2"},{"comment":"The description of the SHAP summary plot colors is inaccurate: the text says \"blue indicating positive influences and red indicating negative ones,\" but in a standard SHAP summary plot the color bar encodes feature values (high versus low), not the direction of the Shapley value. Please clarify the color semantics.","section":"Section 4.2"},{"comment":"The local explanation text is hard to follow: \"The explanation of an expected feature that affects the target class centers on the plot around the x-axis\" and \"was 4.97, which is higher than the base value\" lack a clear subject and would benefit from rewriting.","section":"Section 4.2"},{"comment":"Equation (1) for Random Forest is nonstandard and its variables are not fully defined; in particular, the meaning of W_i(x_j,z) and the summation limits are unclear. Please replace it with a standard description or a proper reference.","section":"Section 3.3"},{"comment":"The phrase \"we oversample our normal samples\" appears to contradict the goal of balancing the dataset, since normal samples are the majority class (98.97%). Please clarify whether the minority or majority class was oversampled, and how the \"balanced\" dataset in Figure 2(b) was constructed.","section":"Section 3.6"},{"comment":"The paper states the dataset is \"presented for the research community,\" but no download link, repository, or data availability statement is given. Please add an availability statement or remove the claim.","section":"Section 3.1 and Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical systems paper whose main contribution rests on reported quantitative results. The lack of a clear statement that ADASYN oversampling is nested inside cross-validation, combined with the extreme hyperparameters and the absence of same-data baselines, makes the headline 99% numbers unreliable as currently presented. I do not see evidence of intentional misconduct, but the evaluation protocol must be corrected and re-run before the results can be taken at face value. The source-domain confound in the self-compiled dataset is a further structural issue that the authors need to address, possibly by per-source evaluation. The paper may be suitable for a journal after major revision, but not in its present form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the paper’s headline numbers — 99%+ accuracy on both datasets — are very likely inflated by a data leakage mistake. Section 3.6 says the normal class is oversampled with ADASYN and then the model is trained with 10-fold CV, but it never states that oversampling is done inside each fold. With a 98.97/1.03 class split, doing ADASYN globally means synthetic instances derived from test-fold data leak into training folds. The paper gives no code, no dataset, and no explicit ordering, so the results are not reproducible under a sound protocol. That makes the central claim unsupported as written.\n\nCredit where it is due: the authors assembled a malware corpus from several public sources covering 54 families — that takes real effort — and they report MCC alongside accuracy, which is better than many papers in this area. The SHAP global and local explanation analysis is presented cleanly, and benchmarking on CTU-13, if the protocol were fixed, would be a useful sanity check. The related work is adequate and the paper cites its own prior ensemble-learning work, which is not a problem by itself.\n\nThe soft spots beyond the leak: no baseline comparison on the same data. Section 4.4 compares against one published accuracy number, but the models are not re-run under the same conditions. Equation 1 is not a correct formula for random forest. Algorithm 1’s flow-window logic is hard to follow. Hyperparameters like max_depth=43 for XGB on 1,127 samples are extreme and consistent with memorization or leakage. The explanation findings are post hoc — SHAP describes the fitted model, not an external ground truth, and the paper overstates them as insights into malware behavior.\n\nThe paper is for a reader who wants a worked example of SHAP on encrypted traffic, but those readers should not trust the reported metrics. The dataset would be a contribution if released, but it is not. This deserves a serious referee only in the sense that the evaluation could be repaired; as written, I would not accept it. My recommendation: send it back for major revision with a mandatory request to redo the whole evaluation with oversampling inside CV and to release code and data. If that cannot be done, reject.","headline":"The paper's >99% metrics are very likely an artifact of ADASYN oversampling before cross-validation; the SHAP analysis is clean but the evaluation is not sound.","tokens_in":11756,"tokens_out":3571,"would_cite":false,"duration_ms":33843,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A tree-ensemble model trained on connection metadata can detect malware in encrypted traffic at over 99% accuracy without decrypting payloads, and SHAP explanations show which features drive each decision.","keywords":["encrypted traffic","malware detection","explainable AI","SHAP","XGBoost","ensemble learning","network security","CTU-13"],"falsifier":"Re-run the XGBoost pipeline with the adaptive oversampling applied only within each training fold, and also with no oversampling at all, then compare accuracy, precision, and F1-score; if the scores drop well below 99%, the original numbers are partly an artifact of data leakage. A simpler check is to inspect the preprocessing code to see whether any synthetic sample generated from a test-set connection appears in a training set.","tokens_in":10703,"feed_emoji":"🛡️","tokens_out":8585,"duration_ms":77265,"temperature":0.7,"pith_summary":"The paper argues that malware hiding inside encrypted network connections can be detected without decryption by training tree-ensemble classifiers on metadata that describes how the connection behaves, not what it carries. It reports that XGBoost identifies malicious traffic with above 99% accuracy, precision, and F1-score on its own collection of 1,127 malicious flows spanning 54 families, and above 99% on the CTU-13 benchmark. What makes the approach explainable is the use of SHAP to show which features drive each decision: maximum packet size, mean inter-arrival time, and the TLS version dominate the global explanation. If those numbers hold, security analysts could get reliable encrypted-malware alerts with reasons attached, without breaking encryption.","feed_headline":"Tree ensembles flag encrypted malware at 99% without decrypting","feed_subtitle":"Packet size, arrival timing, and TLS version carry the signal, and SHAP explains every alert.","key_machinery":"The load-bearing machinery is the pairing of a multi-view feature representation of encrypted flows with a tree-ensemble classifier and Shapley-value attribution. The feature representation is what lets the model see malicious behavior without payload access: it includes handshake details such as offered and accepted cipher suites, TLS extensions and version numbers, certificate validity, time and packet-length statistics, connection metadata, and a 3x3 Markov-chain transition matrix whose states are fixed 150-byte-per-millisecond bins for packet size and inter-arrival time. The classifier side is a set of bagged and boosted trees, namely random forest, XGBoost, and extra trees, with XGBoost reported as best. The explanation side is SHAP, Shapley Additive Explanations, an additive feature-attribution method from cooperative game theory, applied through a tree-specific explainer that decomposes each prediction into additive per-feature contributions, both globally and locally.","core_discovery":"In the paper's own framing, the central discovery is that the behavior of encrypted malware is visible in side-channel features of the TLS connection itself, and tree ensembles can exploit that signal nearly perfectly. The authors build each flow from a 5-tuple bidirectional conversation, filter to completed handshakes, and extract features across handshake offers and accepted ciphers, TLS extensions and versions, certificate validity days, packet length and inter-arrival-time statistics, connection byte and packet counters, and a 3x3 Markov transition matrix built from 150-byte-per-millisecond states. On this representation, XGBoost achieves 99.32% accuracy, 99.53% precision, and 99.43% F1-score on the author-collected dataset and over 99% on CTU-13. The paper also claims that global SHAP explanations consistently rank maximum packet size, mean inter-arrival time of packets, and TLS version as the most influential signals, and that local force plots explain individual traffic samples.","pith_inferences":["If the reported numbers are leak-free, the same pipeline could seed lightweight in-network detectors that fire on packet-size and timing statistics before any machine-learning model runs; the paper does not itself propose such a detector.","A fair re-evaluation should nest the adaptive oversampling inside each cross-validation fold and also try simpler models, because the reported best hyperparameters, such as a tree depth of 43 with only 23 estimators, are typical of overfitting rather than of a discovery about encrypted traffic.","The paper does not test zero-day malware; whether the features generalize to unseen families is a natural next experiment, and the local explanations would be the way to audit failures."],"forward_implications":["If the near-perfect metrics are leak-free, the pipeline gives a practical way to alert on encrypted malware without decrypting any payload.","The global SHAP ranking gives security analysts three concrete, prioritized signals to monitor: maximum packet size, mean packet inter-arrival time, and TLS version.","Local force plots turn each alert into an explanation an analyst can inspect, a step toward trustable automated triage.","The author-collected dataset, 1,127 malicious flows across 54 families, becomes a public benchmark for future explainable-detection work.","The strong performance on CTU-13 suggests the features transfer beyond the specific malware families used for training."],"supporting_citations":[{"why":"Supplies the comparison study showing random forests beating deep models on encrypted malware, the baseline that motivates the paper's tree-ensemble choice.","marker":"[29]"},{"why":"Supplies the flow-generation tool the paper uses to turn raw pcaps into 5-tuple bidirectional flows before feature extraction.","marker":"[16]"},{"why":"Provides the CTU-13 benchmark dataset and the normal-traffic samples used for the second dataset and the comparison.","marker":"[15]"},{"why":"Provides 175 trojan traffic samples, including Zeus and Emotet, used to build the custom multi-family dataset.","marker":"[32]"},{"why":"Provides 305 malware traffic captures used as another source for the custom dataset.","marker":"[14]"},{"why":"Defines SHAP, the Shapley-value attribution method the paper uses for global and local explanations.","marker":"[24]"},{"why":"Supplies the tree-specific SHAP explainer that makes per-feature attributions tractable for XGBoost.","marker":"[25]"},{"why":"Supports the premise that packet-length and timing features separate malicious from normal encrypted flows.","marker":"[17]"}],"fun_headline_variants":["Explainable AI spots malware in encrypted traffic at 99% accuracy","XAI reveals encrypted malware signals: packet size, timing, TLS version","Encrypted malware exposed: tree ensembles hit 99% with SHAP clues","No decryption needed: XAI detects malware in TLS traffic at 99%","SHAP explains how AI detects malware hidden in encrypted streams"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported near-perfect accuracy depends on the balancing step not letting any test-set information reach the training data; the paper does not state whether the synthetic oversampling is performed inside each cross-validation fold, and if it is not, the 99% figures would be inflated.","fun_headline_variants_meta":{"raw":{"variants":["Explainable AI spots malware in encrypted traffic at 99% accuracy","XAI reveals encrypted malware signals: packet size, timing, TLS version","Encrypted malware exposed: tree ensembles hit 99% with SHAP clues","No decryption needed: XAI detects malware in TLS traffic at 99%","SHAP explains how AI detects malware hidden in encrypted streams"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000625,"raw_usage":{"total_tokens":2925,"prompt_tokens":1007,"completion_tokens":1918,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":1823}},"tokens_in":623,"tokens_out":1918,"duration_ms":12153,"temperature":1.0,"reasoning_tokens":1823,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:13:36.038803+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the XGBoost pipeline with the adaptive oversampling applied only within each training fold, and also with no oversampling at all, then compare accuracy, precision, and F1-score; if the scores drop well below 99%, the original numbers are partly an artifact of data leakage. A simpler check is to inspect the preprocessing code to see whether any synthetic sample generated from a test-set connection appears in a training set.","supporting_citations":[{"cited_title":"McGrew, B","cited_arxiv_id":null,"evidence_quote":"Supplies the flow-generation tool the paper uses to turn raw pcaps into 5-tuple bidirectional flows before feature extraction."},{"cited_title":"Stratosphere Laboratory Datasets,","cited_arxiv_id":null,"evidence_quote":"Provides the CTU-13 benchmark dataset and the normal-traffic samples used for the second dataset and the comparison."},{"cited_title":"Master’s thesis, Imperial College, London, UK (2019)","cited_arxiv_id":null,"evidence_quote":"Provides 175 trojan traffic samples, including Zeus and Emotet, used to build the custom multi-family dataset."},{"cited_title":"Malware-Traffic-Analysis,","cited_arxiv_id":null,"evidence_quote":"Provides 305 malware traffic captures used as another source for the custom dataset."},{"cited_title":"In: Advances in Neural Information Processing Systems 30 (NIPS 2017), pp","cited_arxiv_id":null,"evidence_quote":"Defines SHAP, the Shapley-value attribution method the paper uses for global and local explanations."},{"cited_title":"Machine Learning for Encrypted Malware Traffic Classification: Accounting for Noisy Labels and Non-Stationarity,","cited_arxiv_id":null,"evidence_quote":"Supports the premise that packet-length and timing features separate malicious from normal encrypted flows."}],"review_version":1}