{"id":"0fe64b3b-35c1-4c7f-8f29-13107f935ab4","arxiv_id":"2505.16261","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A benchmark report claims 99.94% accuracy for XGBoost with SHAP on encrypted traffic datasets, but the reported metrics are internally inconsistent and the evaluation protocol is contradictory.","lead":"This paper applies three machine learning classifiers, mostly XGBoost and Random Forest, plus SHAP explanations to detect anomalies in encrypted network traffic using flow statistics instead of payload contents. A smart generalist might read it to see how far off-the-shelf explainability tools get on public benchmark datasets, but the reported evaluation is too incomplete to trust.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's reported F1 scores are mathematically incompatible with its own precision and recall; the headline 99.94% accuracy is therefore unsupported and unreproducible.","rationale":"The reader's weakest assumption identifies the contradictory evaluation protocol in §4.1, which is a real reproducibility problem. My stress-test sharpens this into a stronger, protocol-independent objection: the F1 values in Table 2 cannot be derived from the precision and recall in the same row by the paper's own definition of F1. This is not a missing detail or an ambiguity—it is an internal mathematical inconsistency that no binary classification run can produce. Therefore even if the 80:20/10-fold contradiction were resolved, the headline numerical claim would still lack a valid supporting table. The paper also states that only two of the three listed models were evaluated, directly undercutting the comparative claim 'XGBoost outperformed the other two models,' and no per-dataset breakdown supports the 'across multiple datasets' assertion. These issues are load-bearing because the central contribution is an empirical accuracy claim plus SHAP-based interpretation; if the evaluation table is internally inconsistent, the empirical claim cannot be trusted, and SHAP analysis on unvalidated models does not repair it. The reader's REJECT verdict remains appropriate, and my concern does not move the verdict; it reinforces it with an independent, concrete defect.","tokens_in":8104,"tokens_out":2342,"duration_ms":18595,"concrete_test":"Recompute Table 2 from the raw predictions or confusion matrices in the linked repository (github.com/aayush251102/...). If the reported precision and recall are correct, F1 must be ≈89.5% for XGBoost and ≈90.5% for Random Forest; a reported 93.0% or 89.5% indicates the table mixes results from different evaluation protocols. Then run both protocols separately on each of the three datasets—an 80:20 split and 10-fold cross-validation—reporting mean±std accuracy, precision, recall, and F1 per dataset, and check whether the 99.94% figure survives under a fully specified protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that XGBoost reaches 99.94% accuracy and outperforms all models across datasets—rests entirely on Table 2 and the protocol in §4.1. Two independent failures make that claim unsupported. First, the F1 column is internally inconsistent: for XGBoost, P=90.9%, R=88.2% gives F1 = 2·90.9·88.2/(90.9+88.2) ≈ 89.5%, not 93.0%; for Random Forest, P=92.8%, R=88.3% gives ≈90.5%, not 89.5%. A single binary classification experiment cannot produce those three numbers simultaneously, so at least one metric comes from a different run or evaluation than the others. Second, §4.1 simultaneously says 'with train-test split as 80:20' and 'we used 10-fold cross-validation'; the boundary between split and folds is never specified, and the 99.94% number is not tied to any dataset, class balance, confusion matrix, or error bar. Because the abstract claims 'across multiple datasets' but no per-dataset results are shown, the headline is not reproducible. Additionally, §3 promises three models, but §4.2 states only XGBoost and Random Forest were evaluated, so the claim that XGBoost 'outperformed the other two models' has no support for Isolation Forest.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes an interpretable anomaly-detection framework for encrypted network traffic by training machine learning models on flow-level features and applying SHAP post hoc to explain individual predictions. The authors name three public datasets (CIC-Darknet2020, USTC-TFC2016, CSE-CIC-IDS2018), describe a preprocessing pipeline, and report that XGBoost achieved a peak classification accuracy of 99.94% and outperformed the other models. The paper also presents a workflow algorithm, SHAP summary plots, and three short case studies, and it claims that the framework is a novel interpretable anomaly detection system for encrypted environments.","tokens_in":8513,"tokens_out":8216,"duration_ms":63573,"significance":"If the reported results were reliable, the paper would offer a modest but practical demonstration that off-the-shelf gradient boosting combined with SHAP can detect anomalies in encrypted flows without payload decryption, giving security analysts a post-hoc explanation layer. The manuscript has some positive features: it names public datasets, provides a code repository link, structures the pipeline clearly in an algorithm, and includes a useful summary of prior work in Table 1. However, the central performance claim is contradicted by the paper's own Table 2, the evaluation protocol is ambiguous, no per-dataset or error-bar results are reported, and the interpretability findings are anecdotal. As presented, the study does not establish its stated contribution, and the novelty relative to existing SHAP-plus-ML studies is incremental.","major_comments":[{"comment":"The F1-scores in Table 2 are mathematically incompatible with the precision and recall values in the same rows, using the F1 formula stated in §4.2. For XGBoost, precision 90.9% and recall 88.2% give F1 = 2·90.9·88.2/(90.9+88.2) ≈ 89.5%, not 93.0%; for Random Forest, precision 92.8% and recall 88.3% give F1 ≈ 90.5%, not 89.5%. At least one metric in each row must come from a different evaluation than the others. Since Table 2 is the only quantitative support for the headline claim in the abstract and §4.2 that XGBoost reached 99.94% accuracy and outperformed the other models, that claim is unsupported by the presented evidence.","section":"§4.2, Table 2"},{"comment":"The experimental protocol is not reproducible as written. Section 4.1 states 'with train-test split as 80:20, we used 10-fold cross-validation,' but these are alternative evaluation schemes unless their nesting is explicitly defined; the paper never explains which protocol produced the 99.94% figure. In addition, Section 3.1 says features were normalized using Min-Max scaling, while Section 4.1 says z-score normalization was used. A reader cannot determine the exact preprocessing or the exact split/folding procedure behind Table 2.","section":"§4.1 and §3.1"},{"comment":"The claim that XGBoost 'outperformed the other two models across multiple datasets' is not backed by the reported results. Table 2 contains only two models and only one aggregate row per model, with no per-dataset accuracy, precision, recall, F1, class balance, or error bars. Section 3.2 and the Algorithm describe three models, but Section 4.2 says 'The two machine learning models primarily XGBoost, Random Forest were evaluated'; Isolation Forest appears only in the anecdotal Example 3 of Section 4.4. The evidence therefore does not support the abstract's 'across multiple datasets' claim.","section":"Abstract; §4.2; §3.2; Algorithm"},{"comment":"The interpretability claims are not quantitatively validated. The SHAP analysis is presented only as selected summary plots and three post hoc case studies, with no measure of explanation fidelity, feature-rank stability, or consistency between SHAP values and the models' actual decision behavior. The text around Figure 7 is also internally unclear: it says all SHAP values are negative, then describes high feature values as making flows 'less anomalous,' without defining the label encoding. The abstract's statement that SHAP 'successfully revealed the most influential traffic features' therefore goes beyond what the experiments demonstrate.","section":"§4.3 and §4.4"}],"minor_comments":[{"comment":"Reference [4] contains the literal placeholder 'arXiv preprint (if applicable, add identifier)', and several core methodology references ([12]–[16]) are blog posts or web tutorials rather than peer-reviewed sources; these should be replaced or supplemented.","section":"References"},{"comment":"Figure numbering is inconsistent: Figure 1 is used for both XGBoost and Isolation Forest, Figure 2 is used for both Random Forest and SHAP, and the captions do not clearly state which model produced Figures 6 and 7.","section":"Figures"},{"comment":"Section 3.1 says the experiments used 'a publicly available encrypted traffic dataset' (singular), even though Sections 3 and 4.1 list three datasets; this should be harmonized.","section":"§3.1"},{"comment":"The paper claims originality relative to the prior work in Table 1 but provides no quantitative comparison with those methods, so the 'novel interpretable anomaly detection system' claim is not positioned against the cited baselines.","section":"Table 1 and Abstract"}],"recommendation":"reject","confidential_remarks":"The problems in Table 2 are not a single typographical slip: both rows violate the harmonic-mean definition, and the text contradicts itself about how many models were evaluated and under which protocol. A revision would require rerunning the experiments and reporting complete per-dataset results, which is closer to a new evaluation than to a local correction. I therefore recommend rejection, while noting that a substantially rewritten and re-evaluated version could be considered as a new submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, take this one off your desk. The paper applies XGBoost, Random Forest, and (nominally) Isolation Forest with SHAP to three public encrypted-traffic datasets, and claims 99.94% accuracy. That headline number doesn't survive contact with the paper's own Table 2: for XGBoost, precision 90.9% and recall 88.2% give an F1 of about 89.5%, not the reported 93.0%; for Random Forest, 92.8% and 88.3% give about 90.5%, not 89.5%. Those three numbers cannot come from the same experiment. And Section 4.1 simultaneously says 'train-test split as 80:20' and 'we used 10-fold cross-validation' — the actual protocol is unspecified, so the 99.94% is not reproducible.\n\nThe stress-test note holds up fully on reading. The related work table (Table 1) lists Zeleke et al. and Gummadi et al., which already combine SHAP with ML for anomaly detection, including encrypted traffic, so the claimed novelty is mostly a packaging difference. The paper does have some positive features: it uses three standard public benchmarks, gives a GitHub link, and its limitations paragraph is honest about real-time deployment issues. The workflow algorithm is clearly written. But these don't compensate for a load-bearing evaluation error.\n\nMinor soft spots: only two models appear in Table 2 despite three being promised, and the abstract says 'outperforming other models across multiple datasets' with no per-dataset results anywhere. The Section 4.4 case studies are post-hoc narratives, not validations.\n\nRecommendation: desk reject. This is a course-project-level manuscript that needs a full re-run with a consistent protocol, per-dataset metrics, and verification of the F1 arithmetic before any referee time. Not worth sending out.","headline":"The paper's central accuracy claim is unsupported by its own inconsistent table; combined with the already-established ML+SHAP recipe, this is a desk reject.","tokens_in":8930,"tokens_out":2671,"would_cite":false,"duration_ms":18810,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Flow features plus SHAP flag encrypted attacks at 99.94%","keywords":["anomaly detection","encrypted traffic","SHAP","explainable AI","machine learning","XGBoost","random forest","flow-based features"],"falsifier":"Re-run XGBoost on the three named public datasets under one explicitly fixed protocol, report per-dataset accuracy, and compare with 99.94%; a material gap in any configuration would show the reported peak does not transfer as stated.","tokens_in":7889,"feed_emoji":"🛡️","tokens_out":4189,"duration_ms":31480,"temperature":0.7,"pith_summary":"This paper tries to show that machine learning can find anomalies in encrypted network traffic without ever decrypting the payload. The authors train XGBoost, Random Forest, and Isolation Forest on flow-level statistics from three public encrypted-traffic datasets, and report peak accuracy of 99.94% for XGBoost. They then apply SHAP to each model separately to identify which features pushed individual predictions toward 'anomaly.' The point of the exercise is practical: an analyst who trusts the model can also see why it fired an alert.","feed_headline":"SHAP explains encrypted-traffic alerts with 99.94% accuracy","feed_subtitle":"XGBoost reads only packet metadata, then SHAP shows which features triggered each alert.","key_machinery":"SHAP (SHapley Additive exPlanations) is the central mechanism: a game-theoretic method that assigns each input feature a number for its contribution to a given prediction. The authors compute these values per model rather than for a combined ensemble, then rank features by average absolute SHAP value and produce summary, dependence, and force plots. This is what converts a classifier's output into a list of 'why' statements, and it is also the component that would need approximation to run in real time.","core_discovery":"The authors' central claim is that an off-the-shelf gradient-boosted tree model operating purely on metadata can match or beat specialized encrypted-traffic detectors, and that SHAP attributions turn that detector into an explainable tool. On the three benchmark datasets they report XGBoost at 99.94% accuracy, with Random Forest close behind, and SHAP summaries showing features such as total forward packets, destination port, and forward window size as the main drivers. The intended upshot is that encryption does not have to mean blind detection: flow statistics remain observable, and post-hoc explanations let security teams see why a flow was flagged.","pith_inferences":["One consequence the paper leaves implicit is that SHAP's feature rankings on these benchmarks are a testable claim about the traffic itself, so the same top features should appear when the models are retrained on any similarly collected enterprise TLS flow data.","A second inference is that the practical bottleneck is not accuracy but attribution cost; tree-SHAP is fast, but at line rate the paper's per-prediction explanations would need sampling or approximation, and a future study could measure how much explanation fidelity is lost.","Third, the reported class-specific SHAP patterns for botnet, DDoS, and exfiltration flows suggest a cheap extension: cluster alerts by their SHAP profiles and auto-tag likely attack families before deep triage."],"forward_implications":["If the 99.94% figure holds under a defined protocol, perimeter defense can flag malicious encrypted flows using only packet sizes, timings, and directions.","Per-model SHAP rankings give security teams a short list of features to monitor, such as forward packet counts and destination port.","The same pipeline can be re-run on new flow datasets without re-engineering, since the models are standard and SHAP is model-agnostic.","Explainable alerts could support compliance and incident response by documenting why a flow was blocked."],"supporting_citations":[{"why":"Supplies the XGBoost-based encrypted traffic detection baseline that this work extends with SHAP.","marker":"[4]"},{"why":"Demonstrates SHAP combined with ensemble classifiers on encrypted malware traffic, the main precedent for feature-level explanations.","marker":"[6]"},{"why":"Motivates the need for interpretability in anomaly detection for incident response and compliance.","marker":"[5]"},{"why":"Introduces a SHAP-based anomaly detection framework that serves as the model for actionable explanations.","marker":"[8]"},{"why":"Frames why encryption defeats payload-based detection and motivates flow-level analysis.","marker":"[1]"},{"why":"Provides the machine-learning classification baseline for encrypted traffic that this work compares against.","marker":"[3]"}],"fun_headline_variants":["SHAP explains encrypted traffic alerts at 99.94% accuracy","Interpretable anomaly detection in encrypted flows via SHAP","Metadata-only detection with SHAP: why each alert fired","XGBoost reads metadata, SHAP explains its flags","Encrypted-traffic detection explained, not just predicted"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline 99.94% accuracy rests on an evaluation procedure that the paper never clearly specifies, since Section 4.1 says both an 80/20 train-test split and 10-fold cross-validation; if those produce different numbers, the claim cannot be reproduced.","fun_headline_variants_meta":{"raw":{"variants":["SHAP explains encrypted traffic alerts at 99.94% accuracy","Interpretable anomaly detection in encrypted flows via SHAP","Metadata-only detection with SHAP: why each alert fired","XGBoost reads metadata, SHAP explains its flags","Encrypted-traffic detection explained, not just predicted"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1333,"prompt_tokens":925,"completion_tokens":408,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":326}},"tokens_in":541,"tokens_out":408,"duration_ms":3694,"temperature":1.0,"reasoning_tokens":326,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:03:37.370586+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run XGBoost on the three named public datasets under one explicitly fixed protocol, report per-dataset accuracy, and compare with 99.94%; a material gap in any configuration would show the reported peak does not transfer as stated.","supporting_citations":[{"cited_title":"A survey on encrypted network traffic analysis applications, techniques, and countermeasures,","cited_arxiv_id":null,"evidence_quote":"Motivates the need for interpretability in anomaly detection for incident response and compliance."}],"review_version":1}