REVIEW 4 major objections 4 minor 3 references
Interpretable Anomaly Detection in Encrypted Traffic Using SHAP with Machine Learning Models
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Flow features plus SHAP flag encrypted attacks at 99.94%
desk verdict The paper's central accuracy claim is unsupported by its own inconsistent table; combined with the already-established ML+SHAP recipe, this is a desk reject. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
SHAP (SHapley Additive exPlanations) is the central mechanism: a game-theoretic method that assigns each input feature a number for its contribution to a given prediction. The authors compute these values per model rather than for a combined ensemble, then rank features by average absolute SHAP value and produce summary, dependence, and force plots. This is what converts a classifier's output into a list of 'why' statements, and it is also the component that would need approximation to run in real time.
What would settle it
Re-run XGBoost on the three named public datasets under one explicitly fixed protocol, report per-dataset accuracy, and compare with 99.94%; a material gap in any configuration would show the reported peak does not transfer as stated.
Extended reading notes
Core claim
The authors' central claim is that an off-the-shelf gradient-boosted tree model operating purely on metadata can match or beat specialized encrypted-traffic detectors, and that SHAP attributions turn that detector into an explainable tool. On the three benchmark datasets they report XGBoost at 99.94% accuracy, with Random Forest close behind, and SHAP summaries showing features such as total forward packets, destination port, and forward window size as the main drivers. The intended upshot is that encryption does not have to mean blind detection: flow statistics remain observable, and post-hoc explanations let security teams see why a flow was flagged.
Load-bearing premise
The headline 99.94% accuracy rests on an evaluation procedure that the paper never clearly specifies, since Section 4.1 says both an 80/20 train-test split and 10-fold cross-validation; if those produce different numbers, the claim cannot be reproduced.
Editorial extensions
If this is right
- If the 99.94% figure holds under a defined protocol, perimeter defense can flag malicious encrypted flows using only packet sizes, timings, and directions.
- Per-model SHAP rankings give security teams a short list of features to monitor, such as forward packet counts and destination port.
- The same pipeline can be re-run on new flow datasets without re-engineering, since the models are standard and SHAP is model-agnostic.
- Explainable alerts could support compliance and incident response by documenting why a flow was blocked.
Reading between the lines
- One consequence the paper leaves implicit is that SHAP's feature rankings on these benchmarks are a testable claim about the traffic itself, so the same top features should appear when the models are retrained on any similarly collected enterprise TLS flow data.
- A second inference is that the practical bottleneck is not accuracy but attribution cost; tree-SHAP is fast, but at line rate the paper's per-prediction explanations would need sampling or approximation, and a future study could measure how much explanation fidelity is lost.
- Third, the reported class-specific SHAP patterns for botnet, DDoS, and exfiltration flows suggest a cheap extension: cluster alerts by their SHAP profiles and auto-tag likely attack families before deep triage.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes an interpretable anomaly-detection framework for encrypted network traffic by training machine learning models on flow-level features and applying SHAP post hoc to explain individual predictions. The authors name three public datasets (CIC-Darknet2020, USTC-TFC2016, CSE-CIC-IDS2018), describe a preprocessing pipeline, and report that XGBoost achieved a peak classification accuracy of 99.94% and outperformed the other models. The paper also presents a workflow algorithm, SHAP summary plots, and three short case studies, and it claims that the framework is a novel interpretable anomaly detection system for encrypted environments.
Significance. If the reported results were reliable, the paper would offer a modest but practical demonstration that off-the-shelf gradient boosting combined with SHAP can detect anomalies in encrypted flows without payload decryption, giving security analysts a post-hoc explanation layer. The manuscript has some positive features: it names public datasets, provides a code repository link, structures the pipeline clearly in an algorithm, and includes a useful summary of prior work in Table 1. However, the central performance claim is contradicted by the paper's own Table 2, the evaluation protocol is ambiguous, no per-dataset or error-bar results are reported, and the interpretability findings are anecdotal. As presented, the study does not establish its stated contribution, and the novelty relative to existing SHAP-plus-ML studies is incremental.
major comments (4)
- [§4.2, Table 2] The F1-scores in Table 2 are mathematically incompatible with the precision and recall values in the same rows, using the F1 formula stated in §4.2. For XGBoost, precision 90.9% and recall 88.2% give F1 = 2·90.9·88.2/(90.9+88.2) ≈ 89.5%, not 93.0%; for Random Forest, precision 92.8% and recall 88.3% give F1 ≈ 90.5%, not 89.5%. At least one metric in each row must come from a different evaluation than the others. Since Table 2 is the only quantitative support for the headline claim in the abstract and §4.2 that XGBoost reached 99.94% accuracy and outperformed the other models, that claim is unsupported by the presented evidence.
- [§4.1 and §3.1] The experimental protocol is not reproducible as written. Section 4.1 states 'with train-test split as 80:20, we used 10-fold cross-validation,' but these are alternative evaluation schemes unless their nesting is explicitly defined; the paper never explains which protocol produced the 99.94% figure. In addition, Section 3.1 says features were normalized using Min-Max scaling, while Section 4.1 says z-score normalization was used. A reader cannot determine the exact preprocessing or the exact split/folding procedure behind Table 2.
- [Abstract; §4.2; §3.2; Algorithm] The claim that XGBoost 'outperformed the other two models across multiple datasets' is not backed by the reported results. Table 2 contains only two models and only one aggregate row per model, with no per-dataset accuracy, precision, recall, F1, class balance, or error bars. Section 3.2 and the Algorithm describe three models, but Section 4.2 says 'The two machine learning models primarily XGBoost, Random Forest were evaluated'; Isolation Forest appears only in the anecdotal Example 3 of Section 4.4. The evidence therefore does not support the abstract's 'across multiple datasets' claim.
- [§4.3 and §4.4] The interpretability claims are not quantitatively validated. The SHAP analysis is presented only as selected summary plots and three post hoc case studies, with no measure of explanation fidelity, feature-rank stability, or consistency between SHAP values and the models' actual decision behavior. The text around Figure 7 is also internally unclear: it says all SHAP values are negative, then describes high feature values as making flows 'less anomalous,' without defining the label encoding. The abstract's statement that SHAP 'successfully revealed the most influential traffic features' therefore goes beyond what the experiments demonstrate.
minor comments (4)
- [References] Reference [4] contains the literal placeholder 'arXiv preprint (if applicable, add identifier)', and several core methodology references ([12]–[16]) are blog posts or web tutorials rather than peer-reviewed sources; these should be replaced or supplemented.
- [Figures] Figure numbering is inconsistent: Figure 1 is used for both XGBoost and Isolation Forest, Figure 2 is used for both Random Forest and SHAP, and the captions do not clearly state which model produced Figures 6 and 7.
- [§3.1] Section 3.1 says the experiments used 'a publicly available encrypted traffic dataset' (singular), even though Sections 3 and 4.1 list three datasets; this should be harmonized.
- [Table 1 and Abstract] The paper claims originality relative to the prior work in Table 1 but provides no quantitative comparison with those methods, so the 'novel interpretable anomaly detection system' claim is not positioned against the cited baselines.
Circularity Check
No circularity found: the reported accuracy and SHAP explanations are empirical outputs of fitted models, not definitions of the anomaly-detection target.
full rationale
This paper reports an empirical machine-learning pipeline in Section 3 and Algorithm 1: flow-level features are preprocessed, XGBoost and Random Forest are trained, predictions are evaluated with standard metrics, and SHAP is applied post hoc to explain individual predictions. There is no derivation chain in which a predicted quantity is defined in terms of, or fitted from, the quantity it is claimed to predict; the 99.94% accuracy figure and the SHAP feature rankings are outputs of already-fitted models. The self-citations in the reference list ([2], [17], [18]) are background material and do not carry the central detection claim. The SHAP case studies in Section 4.4 are illustrative post-hoc descriptions of selected individual predictions, but the text never uses those examples to define or force the anomaly labels, so they are not circular. The internal inconsistencies in the paper, such as the contradiction between the 80:20 split and 10-fold cross-validation in Section 4.1, the F1 values in Table 2 being incompatible with the reported precision and recall, and the mismatch between the three models promised in Section 3 and the two evaluated in Section 4.2, are reproducibility and correctness concerns rather than circularity. Under the rule that circularity must be demonstrated by quoting a specific reduction, no circular step is present, so the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (2)
- Model hyperparameters (XGBoost, Random Forest) =
not reported
- Class imbalance handling =
not reported
assumptions (3)
- domain assumption Features across CIC-Darknet2020, USTC-TFC2016, and CSE-CIC-IDS2018 are comparable and can be processed with a single pipeline.
- domain assumption SHAP's additive feature attribution is a valid explanation of model decisions in this domain.
- domain assumption Binary labels (normal vs anomaly) are accurate and consistent across datasets.
Cite this review
Pith. "Pith review of Interpretable Anomaly Detection in Encrypted Traffic Using SHAP with Machine Learning Models." pith.science (2026). https://pith.science/paper/O6K7ZPW7
@misc{pith2026250516261,
author = {Pith},
title = {Pith review of: Interpretable Anomaly Detection in Encrypted Traffic Using SHAP with Machine Learning Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/O6K7ZPW7}},
note = {Machine review of arXiv:2505.16261}
}
read the original abstract
The widespread adoption of encrypted communication protocols such as HTTPS and TLS has enhanced data privacy but also rendered traditional anomaly detection techniques less effective, as they often rely on inspecting unencrypted payloads. This study aims to develop an interpretable machine learning-based framework for anomaly detection in encrypted network traffic. This study proposes a model-agnostic framework that integrates multiple machine learning classifiers, with SHapley Additive exPlanations SHAP to ensure post-hoc model interpretability. The models are trained and evaluated on three benchmark encrypted traffic datasets. Performance is assessed using standard classification metrics, and SHAP is used to explain model predictions by attributing importance to individual input features. SHAP visualizations successfully revealed the most influential traffic features contributing to anomaly predictions, enhancing the transparency and trustworthiness of the models. Unlike conventional approaches that treat machine learning as a black box, this work combines robust classification techniques with explainability through SHAP, offering a novel interpretable anomaly detection system tailored for encrypted traffic environments. While the framework is generalizable, real-time deployment and performance under adversarial conditions require further investigation. Future work may explore adaptive models and real-time interpretability in operational network environments. This interpretable anomaly detection framework can be integrated into modern security operations for encrypted environments, allowing analysts not only to detect anomalies with high precision but also to understand why a model made a particular decision a crucial capability in compliance-driven and mission-critical settings.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[4]
Anomaly detection using XGBoost ensemble of deep neural network models,
S. T. Ikram et al., "Anomaly detection using XGBoost ensemble of deep neural network models," 2021. [Online]. Available: arXiv preprint (if applicable, add identifier) [5] M.-D. Nguyen, V.-H. La, W. Mallouli, A. R. Cavalli, and E. M. de Oca, "Toward Anomaly Detection Using Explainable AI," in CyberSecurity in a DevOps Environment: From Requirements to Mon...
arXiv 2021
-
[5]
A survey on encrypted network traffic analysis applications, techniques, and countermeasures,
Visualize_And_Analyze(shap_XGB, shap_RF, shap_ISO): a. Generate SHAP summary plots (bar, beeswarm) b. Generate force plots and dependence plots c. Analyze feature contributions for individual predictions 6. Identify_Influential_Features(shap_values): For each model: Compute average |SHAP value| per feature Rank features by importance Return: ranked_featur...
work page 2018
-
[17]
Encrypted network traffic classification using intelligent techniques,
S. Mali, M. Gujral, and A. Cherukuri, “Encrypted network traffic classification using intelligent techniques,” Cureus J. Comput. Sci., vol. 2, Jan. 16, 2025, Art. no. es44389-024-02701-2. doi: https://doi.org/10.7759/s44389-024-02701-2 [18] Cherukuri AK, Ikram ST, Li G, Liu X: Encrypted Network Traffic Analysis. Springer, Cham; 2024. 10.1007/978-3-031-62909-9
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.