{"id":"94d1d2d9-ecfc-4d0d-95f6-fff20e02aded","arxiv_id":"2501.07801","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"An evaluation framework for white-box XAI in network intrusion detection reports high robustness, but its own completeness results contradict the claimed advantage.","lead":"This paper evaluates three white-box explainable AI methods (LRP, IG, DeepLift) for deep learning based network intrusion detection on three datasets, using six quality metrics. It claims these methods are more robust and complete than black-box explainers like SHAP and LIME, but its own completeness tables often show low or zero scores, undermining that claim.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Completeness claim is contradicted by the paper's own Tables 9-11 and rests on a theory that may not transfer to the softmax-free DNNs; the cross-method comparison also uses non-identical protocols.","rationale":"The reader's weakest assumption identifies precisely the most load-bearing weakness: the paper claims high completeness for white-box XAI based on a theoretical argument from [17], while its own empirical completeness tables (Tables 9-11) show low or zero scores for many classes. This is not a minor discrepancy; it is a direct contradiction of the headline result. The authors acknowledge the reduced experimental scope and memory constraints but still conclude superiority, effectively substituting theory for evidence. Additionally, the comparison with black-box XAI from [25] is confounded by different models, different perturbation depths (top-2 vs top-5), and different XAI implementations, so the five-of-six claim is not supported by a controlled experiment. The proposed concrete test — rerunning completeness under a single consistent protocol with full feature perturbation and a softmax-enabled (or equivalent) model — would settle whether the empirical contradiction is real or an artifact of the reduced setup. The reader's verdict of REJECT is appropriate; I see no reason to change it.","tokens_in":33592,"tokens_out":2464,"duration_ms":25128,"concrete_test":"Re-run the completeness experiment on the same DNNs with (a) a softmax-enabled model or an implementation not restricted by iNNvestigate, (b) perturbing all features (or at least the same top-k as [25]), and (c) identical perturbation steps for white-box and black-box methods, using the exact completeness definition from [17]. If white-box methods still yield low class-change rates while black-box methods under the identical protocol perform better, the paper's five-of-six claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that white-box XAI methods \"score high in robustness and completeness\" and \"match or surpass black-box methods in five of six metrics\" (Sections 6.8 and 8) — depends on treating white-box methods as inherently complete. The authors explicitly invoke this theory from [17] in Sections 6.6 and 7, even though their empirical completeness experiment (Tables 9-11) reports many class-level scores below 30%, including 0% for DeepLift on RoEduNet-SIMARGL2021 Normal/PortScan and for IG (Bot 0%, Web Attack 2%) and LRP (PortScan 0%, Infiltration 0%) on CICIDS-2017. The experiment only perturbed the top-2 features, whereas the black-box baseline [25] perturbed top-5, so the cross-paper comparison is not on equal footing. Moreover, Section 7 concedes that omitting the softmax layer (required for iNNvestigate) \"can impact the final accuracy... probably impacts\" metrics including Completeness. The paper's own data therefore contradicts the completeness claim; the theory is used to override empirical measurements rather than being tested. If the theoretical completeness from [17] does not transfer to these DNNs, or if that theory defines completeness differently than the class-change proxy used here, the claim that white-box XAI is superior in completeness collapses, and Tables 9-11 would support the opposite conclusion for several classes.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an end-to-end framework for evaluating three white-box XAI methods (LRP, IG, DeepLift) on DNN-based intrusion detection models, using six metrics (descriptive accuracy, sparsity, stability, efficiency, robustness, completeness) across NSL-KDD, CICIDS-2017, and RoEduNet-SIMARGL2021. It also compares the white-box results with black-box XAI baselines (SHAP, LIME) reported in the authors' earlier work [25]. The authors conclude that white-box methods score high in robustness and completeness and match or surpass black-box methods in five of six metrics.","tokens_in":33891,"tokens_out":6550,"duration_ms":55385,"significance":"If the empirical claims were fully supported, the paper would be a useful step toward deploying white-box XAI in NIDS, and the open-sourced framework would support follow-up evaluations by the community. The paper's strengths are the breadth of its evaluation (three datasets, three white-box methods, six metrics) and the adaptation of the robustness attack from [26] to DNNs and white-box methods. However, the central completeness claim and the cross-method comparison rest on internal inconsistency and protocol mismatch described in the major comments; the current form does not yet substantiate the advertised conclusions.","major_comments":[{"comment":"the paper's central claim that white-box methods 'score high in robustness and completeness' is contradicted by its own empirical completeness results. For example, DeepLift has 0% completeness for RoEduNet Normal and PortScan (Table 9), IG has 0% for CICIDS Bot and 2% for Web Attack (Table 10), and LRP has 0% for CICIDS PortScan and Infiltration (Table 10). The text acknowledges that the reduced top-2 perturbation 'undermined its performance for this metric' and that the softmax omission 'probably impacts' Completeness (Section 7), yet the abstract and Section 8 nevertheless assert high completeness and superiority over black-box methods. The paper cannot use the same experiment as evidence for completeness when it also dismisses the experiment as invalid; the theoretical completeness argument from [17] is invoked without establishing that it applies to the softmax-free DNN models used here.","section":"Section 6.6, Tables 9-11, Section 7, Section 8"},{"comment":"the white-box versus black-box completeness comparison is not protocol-equivalent. The black-box baseline [25] perturbed the top-5 features, whereas the white-box experiment here perturbs only the top-2 features (the paper states this difference explicitly in Section 7). Because perturbing more features makes a class change more likely, the completeness scores are not directly comparable, and the conclusion that white-box methods 'matched or surpassed' black-box methods in completeness is not supported. The authors should either rerun the experiment with the same perturbation depth or restrict the comparison to descriptive, non-quantitative statements.","section":"Section 6.8 compared with Section 6.6"},{"comment":"the descriptive accuracy analysis lacks a random-removal baseline. The authors interpret accuracy drops after removing the top-k features as evidence of 'explainability power' and use these curves to rank LRP, IG, and DeepLift. Without a control condition in which k random features are removed, there is no way to attribute the observed accuracy changes to the quality of the XAI rankings rather than to the mere reduction of input dimensionality. This concern is particularly acute for the CICIDS-2017 LRP curve, where accuracy does not drop (and even increases) as top features are removed; the 'curse of dimensionality' explanation offered in Section 6.1 is post hoc and unfalsifiable without such a baseline.","section":"Section 6.1"}],"minor_comments":[{"comment":"The caption says 'Overall, SHAP provides the best performance compared to LIME for the six metrics analyzed in this work,' but this caption appears to be carried over from the authors' prior black-box paper [25]; the table actually summarizes DeepLift, IG, and LRP, and the caption should be corrected.","section":"Table 12 caption"},{"comment":"Section 5.3 states that for Descriptive Accuracy the removed-feature order is k=0,10,20,40,80, while the step-by-step algorithm in Section 4.3 (C.1) specifies k=(0,5,10,25,50,70); this inconsistency should be resolved.","section":"Section 5.3 vs Section 4.3 (C.1)"},{"comment":"The citation for LRP in the introduction and Section 4.2 is given as [21], but reference [21] is Gunning and Aha's DARPA XAI program, not the Layer-wise Relevance Propagation paper; the correct LRP reference should be supplied.","section":"References [21]"},{"comment":"Several typographical issues remain, including 'white-bpx' in Section 6.2, 'auditory actions' in Section 3.6, and 'thrust' for 'trust' in Section 6.7; a careful proofreading pass is needed.","section":"Throughout"},{"comment":"The GitHub URL contains a space ('XAI Whitebox') and should be URL-encoded to be accessible.","section":"Introduction and Declarations"}],"recommendation":"major_revision","confidential_remarks":"The paper has a material number of self-citations to the authors' own prior work [25]; this is acceptable as a benchmark, but the comparison protocol should be identical. The Section 7 limitations are candid, and the abstract and conclusion should be brought in line with them. There is also a citation error for LRP that should be fixed before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Most useful thing here is the pipeline: six XAI metrics applied to LRP, IG, and DeepLift on three NIDS datasets with DNNs, plus a comparison against the authors' own black-box E-XAI results. That is a legitimate extension of prior work, and the framework plus open code is something the community can reuse. Efficiency and stability results are concrete and believable. The robustness adaptation from Slack et al. to DNNs and white-box explainers is a real effort.\n\nThe soft spot is the completeness claim, and it is load-bearing. The abstract says white-box methods \"score high in robustness and completeness,\" but Tables 9-11 show many class-level completeness scores at 0-30%, including 0% for DeepLift on RoEduNet Normal/PortScan, and for IG on Bot and LRP on PortScan/Infiltration. The authors acknowledge in Section 6.6 that the results show the methods \"would not be complete,\" then pivot to the theoretical argument from [17] that white-box methods are inherently complete because they use model internals. That is not a test of the claim; it is an assumption overriding their own empirical data. The reduced protocol (top-2 features) is also not comparable to the black-box baseline's top-5 perturbation, and the paper admits that the softmax-free DNNs required by iNNvestigate may impact accuracy and completeness. Those are not minor caveats for the central claim.\n\nThe rest is mixed rather than damning. Descriptive accuracy lacks a random-removal baseline, so we cannot tell whether accuracy drops are due to feature importance or simply to removing any features. The robustness comparison uses different models than the black-box baseline, though the white-box methods do appear more resistant in their own setup. The cross-paper \"five of six metrics\" claim is shaky because the experimental protocols differ.\n\nThis paper is for researchers working on XAI evaluation for intrusion detection who want a ready-made benchmark harness. The framework deserves serious referee time, but the completeness claim needs a corrected experiment (more features perturbed, random baseline, softmax-compatible models) or it should be dropped. I would send this to peer review with major revisions required, not desk-reject, because the empirical core is reusable and the main flaw is fixable.","headline":"Useful evaluation pipeline for white-box XAI on NIDS, but the headline completeness claim is not supported by the paper's own tables and needs a major fix.","tokens_in":34446,"tokens_out":2307,"would_cite":true,"duration_ms":24660,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that white-box explainable AI methods (LRP, Integrated Gradients, and DeepLift) match or beat black-box ones on five of six quality metrics for network intrusion detection.","keywords":["white-box XAI","network intrusion detection","Layer-wise Relevance Propagation","Integrated Gradients","DeepLift","XAI evaluation metrics","robustness","completeness"],"falsifier":"For the same DNNs, restore the softmax layer and run the completeness test while perturbing all features (not just the top two) across 100 samples per class; if a substantial share of samples never changes prediction class, the paper's completeness claim is refuted. The RoEduNet-SIMARGL2021 Normal class with DeepLift, which the paper reports as 0 percent complete, would be a decisive place to look.","tokens_in":33354,"feed_emoji":"🛡️","tokens_out":9672,"duration_ms":84011,"temperature":0.7,"pith_summary":"This paper tries to establish that white-box explainable AI methods can be trusted for explaining DNN-based network intrusion detection. It evaluates Layer-wise Relevance Propagation, Integrated Gradients, and DeepLift on three intrusion datasets using six quality metrics, and reports that these methods score high on robustness and completeness and match or beat black-box methods (LIME and SHAP) on five of six metrics, with stability the only exception. If correct, security analysts could rely on explanations drawn from the model's own internals rather than on approximations when investigating alerts. The paper also cautions that all three methods still need improvement before production deployment.","feed_headline":"White-box AI explainers beat black-box on 5 of 6 intrusion metrics","feed_subtitle":"LRP, Integrated Gradients, and DeepLift score high on resilience and completeness across three network datasets.","key_machinery":"The mechanism is attribution by back-propagation. LRP and DeepLift push relevance scores backward through the network under conservation rules, while IG integrates gradients along a path from a baseline to the input. Because these computations use the model's internals, the paper treats them as inherently complete and as harder to fool than black-box approximations, and it evaluates them with six operationalized metrics: accuracy loss when top-ranked features are removed, thresholded sparsity of importance scores, overlap of top features across repeated runs, wall-clock time to generate explanations, detection of a biased feature under a two-model adversarial attack, and prediction-class change under perturbation of the top features.","core_discovery":"The central claim is that white-box XAI methods, because they derive explanations from the DNN's own weights and gradients, are more resilient to adversarial perturbation and more complete than black-box surrogates in the network intrusion setting. Across NSL-KDD, CICIDS-2017, and RoEduNet-SIMARGL2021, the authors report that LRP, IG, and DeepLift outperform or tie LIME and SHAP on descriptive accuracy, sparsity, efficiency, robustness, and completeness, and lose only on stability. The paper further claims that LRP is the best overall white-box method, IG the most robust, and DeepLift the most complete on CICIDS-2017, while cautioning that none of the three is production-ready.","pith_inferences":["Editorial inference: the paper's own completeness tables undercut its theoretical completeness claim, since many cells are 0 to 43 percent; a fuller perturbation test would likely reveal that the reduced top-two-feature experiment was too weak.","Editorial inference: because the DNNs omit the softmax layer to accommodate the explanation software, accuracy and completeness numbers may not transfer to production classifiers; re-running with softmax-enabled models is a direct test.","Editorial inference: the robustness experiment uses a single engineered unrelated feature, so the reported resilience margin is an upper bound; subtler crafted perturbations could fool white-box methods too.","Editorial inference: a natural next step is to apply the same six-metric battery to newer white-box methods and to transformer-based IDS models to see whether the white-box advantage persists."],"forward_implications":["Security analysts can prefer white-box explanations over LIME and SHAP for DNN-based intrusion detectors, because the paper reports white-box methods matching or beating black-box ones on five of six quality metrics.","IG and LRP produce explanations fast enough for practical monitoring (roughly a second or less even at 10,000 samples), whereas DeepLift's runtime grows steeply with sample count.","The open-sourced evaluation framework lets other researchers score additional XAI methods and datasets on the same six metrics, making XAI claims in intrusion detection directly comparable.","White-box XAI should not yet be deployed in production intrusion detection, because the adversarial attack still succeeds often and the completeness experiment was cut short by memory limits.","The best method depends on the dataset: the paper finds IG best for CICIDS-2017 overall, while LRP and DeepLift do better on RoEduNet-SIMARGL2021."],"supporting_citations":[{"why":"Supplies the six XAI evaluation metrics and the white-box/black-box taxonomy, including the claim that white-box methods are inherently complete.","marker":"[17]"},{"why":"The prior black-box XAI evaluation (SHAP and LIME) on the same three datasets that serves as the baseline for the white-box comparison.","marker":"[25]"},{"why":"The adversarial two-model setup (biased model plus perturbed-feature model) adapted here to test robustness of IG, LRP, and DeepLift.","marker":"[26]"},{"why":"Defines Integrated Gradients, one of the three white-box methods evaluated.","marker":"[20]"},{"why":"Cited as the reference for Layer-wise Relevance Propagation, a second evaluated method.","marker":"[21]"},{"why":"Defines DeepLift, the third evaluated white-box method.","marker":"[22]"},{"why":"The NSL-KDD dataset, one of the three intrusion benchmarks used.","marker":"[15]"},{"why":"The RoEduNet-SIMARGL2021 dataset, the real-world traffic benchmark used.","marker":"[27]"},{"why":"The CICIDS-2017 dataset, the third benchmark used.","marker":"[28]"}],"fun_headline_variants":["White-box explainers beat black-box on 5 of 6 NIDS metrics","LRP, IG, DeepLift top LIME, SHAP in 5 of 6 NIDS tests","White-box AI for NIDS: wins 5 of 6 metrics vs black-box","White-box XAI more robust and complete for network intrusion","Study: white-box beats black-box on 5 of 6 XAI metrics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that white-box XAI methods are complete by definition because they access the model's internals, even though the DNNs used here omitted the final softmax layer and the empirical completeness test left many samples unchanged; if that theoretical transfer fails, the claim of high completeness collapses.","fun_headline_variants_meta":{"raw":{"variants":["White-box explainers beat black-box on 5 of 6 NIDS metrics","LRP, IG, DeepLift top LIME, SHAP in 5 of 6 NIDS tests","White-box AI for NIDS: wins 5 of 6 metrics vs black-box","White-box XAI more robust and complete for network intrusion","Study: white-box beats black-box on 5 of 6 XAI metrics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000644,"raw_usage":{"total_tokens":2954,"prompt_tokens":935,"completion_tokens":2019,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":1910}},"tokens_in":551,"tokens_out":2019,"duration_ms":14989,"temperature":1.0,"reasoning_tokens":1910,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:35:29.859493+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For the same DNNs, restore the softmax layer and run the completeness test while perturbing all features (not just the top two) across 100 samples per class; if a substantial share of samples never changes prediction class, the paper's completeness claim is refuted. The RoEduNet-SIMARGL2021 Normal class with DeepLift, which the paper reports as 0 percent complete, would be a decisive place to look.","supporting_citations":[{"cited_title":"In: 2020 IEEE European Symposium on Security and Pri- vacy (EuroS&P), pp","cited_arxiv_id":null,"evidence_quote":"Supplies the six XAI evaluation metrics and the white-box/black-box taxonomy, including the claim that white-box methods are inherently complete."},{"cited_title":"IEEE Access 12, 23954–23988 (2024) https://doi.org/10.1109/ ACCESS.2024.3365140","cited_arxiv_id":null,"evidence_quote":"The prior black-box XAI evaluation (SHAP and LIME) on the same three datasets that serves as the baseline for the white-box comparison."},{"cited_title":"In: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp","cited_arxiv_id":null,"evidence_quote":"The adversarial two-model setup (biased model plus perturbed-feature model) adapted here to test robustness of IG, LRP, and DeepLift."},{"cited_title":"In: Precup, D., Teh, Y.W","cited_arxiv_id":null,"evidence_quote":"Defines DeepLift, the third evaluated white-box method."},{"cited_title":"Interna- tional journal of advanced research in com- puter and communication engineering 4(6), 446–452 (2015)","cited_arxiv_id":null,"evidence_quote":"The NSL-KDD dataset, one of the three intrusion benchmarks used."},{"cited_title":"Sensors 21(13), 4319 (2021)","cited_arxiv_id":null,"evidence_quote":"The RoEduNet-SIMARGL2021 dataset, the real-world traffic benchmark used."}],"review_version":1}