REVIEW 3 major objections 4 minor 35 references
The paper argues that explaining why an event is anomalous—not why a model flagged it—improves both explanation usefulness and detection accuracy, and supports this with a multi-agent LLM investigation framework.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 00:29 UTC pith:AK4GVYHH
load-bearing objection A well-engineered multi-agent framework for event-centric explanation, but the evaluation's reliance on an unvalidated LLM judge and a missing control leaves the strong claims under-supported. the 3 major comments →
(EC)2: Event-Centric Explainability for Cybersecurity Through Multi-Agent LLM Investigations
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On the paper's own terms, the central claim is that the interpretability gap in security operations is not a feature-attribution problem but an investigation problem. (EC)2 operationalizes this by treating each flagged event as a case: agents propose hypotheses, retrieve evidence from a network-topology knowledge graph and a packet/flow traffic database, evaluate each hypothesis, and produce a narrative report documenting supported, refuted, and unresolved hypotheses. The paper reports that this approach outperforms a prompt-augmented, single-inference LLM explainer on all three quality criteria (attack type 3.21 vs 1.49, entity 4.10 vs 3.22, completeness 3.04 vs 1.29), and that feeding the
What carries the argument
The load-bearing mechanism is the hypothesis-driven investigation loop with retrieval-augmented evidence grounding. A contextual question generator expands the event into factual questions; an evidence-gathering agent routes them to retrieval agents that query a network-topology knowledge graph and a traffic database; a planner, hypothesis generator, and hypothesis evaluator iterate over candidate explanations; a flow evaluator decides when to conclude or re-enter the cycle; and an explanation agent writes the final report, which a reviewer checks for coherence. The retrieval-augmented constraint—agents may only reason over retrieved data, not their internal knowledge—is what converts black-
Load-bearing premise
The results stand on the assumption that a large language model prompted as a senior security analyst produces valid, reliable quality scores and verdicts without human validation or inter-rater checks, and that the 1,000-record balanced sample from the uncertain zone is representative enough to attribute the classification gains to the framework's evidence retrieval rather than to the model's prior knowledge.
What would settle it
Re-run the RQ3 reclassification with the judge LLM given only the raw flagged event, bypassing the investigation pipeline; if recall stays near 0.54 and false-positive rate near 0.43, the reported gains come from the evidence-gathering machinery, while materially different numbers would point to the LLM's prior knowledge or the sample's composition.
If this is right
- Analysts receive chronological, evidence-grounded reconstructions of attack stages and entity roles, instead of feature-importance lists.
- A post-hoc investigation layer can be attached to any detector, treating it as a black box, without retraining or internal access.
- Near-threshold detections can be reclassified by routing them through the investigation pipeline, reducing both missed attacks and false alarms.
- Ablation results indicate that database access and the planning/initialization components drive explanation quality, while the reviewer loop adds no measurable improvement.
- The roughly 17-minute runtime per investigation restricts the framework to high-priority or ambiguous alerts rather than full-stream triage.
Where Pith is reading between the lines
- The RQ3 design leaves room for a confound: the judge LLM may rely on its own attack knowledge rather than the evidence the framework retrieved. A control condition feeding the same judge the raw event without investigation would isolate whether the gain comes from the evidence-gathering machinery.
- Explanation quality is scored by an LLM judge without reported human validation; a natural next step is measuring agreement between LLM scores and experienced security analysts, or fact-checking reports against ground-truth databases.
- The runtime economics point toward adaptive investigation depth: stop after the first well-supported hypothesis on clear cases and reserve the full loop for genuinely ambiguous events, yielding a measurable accuracy-versus-latency tradeoff.
- The event-centric framing should transfer beyond security to any domain with relational data and temporal logs—fraud, health monitoring, cloud operations—where the question 'why is this observation anomalous?' benefits from the same hypothesis-testing structure.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes (EC)2, a multi-agent LLM framework that explains anomalous network events through hypothesis-driven investigation grounded in retrieval-augmented generation over topology and traffic data. The evaluation compares (EC)2 against eX-NIDS on 50 CSE-CIC-IDS2018 events using an LLM-based judge (RQ1), ablates framework components (RQ2), and shows that feeding (EC)2 explanations to an LLM improves binary classification of 1,000 records sampled from the uncertain zone of AE-IDS (RQ3). The paper claims that (EC)2 produces operationally meaningful explanations and also enhances detection accuracy.
Significance. If validated, (EC)2 would be a useful contribution: it addresses a real gap in SOC workflows, where feature-level XAI is insufficient, and the multi-agent architecture with explicit evidence retrieval is a reasonable design. The paper also provides reproducibility advantages by using a public dataset and publicly released eX-NIDS explanations. However, the current evaluation does not substantiate the central claims due to the unvalidated LLM-as-judge methodology for explanation quality and the missing control in the detection-improvement experiment. As presented, the evidence is suggestive but not conclusive.
major comments (3)
- [§5.1] All RQ1/RQ2 explanation-quality scores are produced by GPT-5.1 prompted as a senior security analyst, with no human-rated ground truth or inter-rater reliability assessment. The claim that criteria are 'manually reviewed for reliability' (p. 12) is not supported by any described protocol or result. The three-run stability check (Fig. 6) only measures stochastic consistency, not agreement with human judgment. Since the central claim that (EC)2 produces 'operationally meaningful explanations' rests entirely on these scores, the evaluation is self-referential and the conclusion is unsupported. Moreover, the judge may be systematically biased toward longer, narrative outputs (multi-agent reports) over the short eX-NIDS explanations. Please add a human evaluation (e.g., expert ratings on a subset) and report agreement statistics, or at minimum validate the judge against a human-annotated benc
- [§5.3] The RQ3 evaluation lacks a critical control: an LLM classifying the same raw events (or AE-IDS feature inputs) without (EC)2 explanations. The observed improvement (recall 0.54→0.72, precision 0.56→0.73) could be due to the LLM's prior knowledge or to the balanced 50/50 sample rather than to the explanation content. Additionally, the 1,000-record sample is balanced by construction, but the actual class distribution in the uncertain zone is not reported; metrics computed on this balanced sample are not directly comparable to operational conditions where the base rate is ~17% malicious. Please report the size and class distribution of the full uncertain zone, compute metrics on an unbiased sample (or the full zone), and add an LLM-only baseline (e.g., prompt with raw NetFlow features) to allow causal attribution to (EC)2.
- [§5.2] The ablation study performs many paired t-tests (multiple variants × three criteria) without any multiple-comparison correction. Given that the underlying scores are 1–5 discrete ratings derived from an unvalidated LLM judge, the reported p-values should be interpreted cautiously. A non-parametric test (e.g., Wilcoxon) or a correction (e.g., Bonferroni or FDR) would strengthen the ablation conclusions. This is a load-bearing issue for RQ2, as the paper uses these results to claim that database access, planning, and initialization are necessary.
minor comments (4)
- [§5.2] Typographical and grammar issues appear, e.g., 'which their performance is compared' (p. 15) and 'which are evaluated' (p. 16). The paper would benefit from a language edit.
- [§1] The paper describes (EC)2 as 'detector-agnostic,' but the experiments use only one detector (AE-IDS) and one dataset. A brief discussion of generalizability and any potential dependency on the detector's output format would be helpful.
- [§5.1] The evaluator model (GPT-5.1) and the agent model (GPT-5-mini) are from the same family. Although the paper does not use the judge to compare its own outputs, the potential for model-family bias is not addressed. A human validation or a judge from a different family would mitigate this concern.
- [§5.3] The runtime of ~17 minutes per event is mentioned as a limitation, but no systematic analysis of parallel scaling or cost is provided. This is acceptable for an in-depth alert triage use case, but the claim that 'this runtime does not impose fundamental operational limitations' (p. 17) is not fully supported.
Circularity Check
Entity-identification scores and RQ3 gains are partly circular: the knowledge graph is populated with the same 'attacker IP identification' used as ground truth, so the framework retrieves labels it is later credited with explaining/predicting.
specific steps
-
fitted input called prediction
[§4.2 (KG construction); §5.1 (RQ1 ground truth); §3.1 (retrieval agent)]
"From the dataset documentation, additional context is obtained: device types, attacker IP identification, internal-to-public IP address mappings, subnet-to-department assignments, and the overall network hierarchy. / the ground truth, which comprises the attack type, attacker and victim IPs, and the ordered attack stages - constructed from the publicly available dataset documentation. / returns a concise natural language answer based solely on the information stored in the knowledge graph."
Entity Identification is scored against 'attacker and victim IPs' from the dataset documentation, and the very same documentation is used to insert 'attacker IP identification' into the knowledge graph that (EC)2 retrieves from. The Knowledge Graph Agent answers 'based solely on the information stored in the knowledge graph,' so identifying the attacker can be a direct lookup of a pre-inserted label rather than an investigative conclusion. This inflates RQ1's Entity Identification advantage over eX-NIDS (which gets no such store) and can leak into RQ3's explanations, so the reported RQ3 reclassification gain is partly a label-retrieval artifact, not an independent prediction.
full rationale
The main circularity is not the LLM-judge methodology (a validity concern, not a by-construction reduction) but the construction of (EC)2's evidence store. Section 4.2 explicitly builds the knowledge graph using 'attacker IP identification' taken from the dataset documentation, while Section 5.1 uses the same documentation to define the entity ground truth ('attacker and victim IPs'). Because the Knowledge Graph Agent answers queries solely from that store, the framework's entity-identification explanations can reduce to retrieving the ground-truth label that was baked into the input. The comparison against eX-NIDS is therefore asymmetrical for this criterion, and RQ3's reclassification explanations may also carry label-derived content, making some of the reported recall/precision gain a leakage artifact rather than an emergent investigative capability. Attack Type Identification and Explanation Completeness are not directly shown to be leaked by the quoted text, and the system is not wholly circular: the architecture, RAG grounding, external baseline, and McNemar test on external labels give the paper substantial independent content. Still, the label-in-input step undermines a central part of RQ1 and RQ3. No load-bearing self-citation was found, and 'manually reviewed for reliability' in §5.1 is an asserted but unreported support gap, not a circular step.
Axiom & Free-Parameter Ledger
free parameters (6)
- uncertain_zone_size =
0.05
- stratified_sample_size =
1000 (500 malicious, 500 benign)
- evaluator_runs =
3
- max_hypotheses_per_round =
3
- max_hypothesis_rounds =
2
- max_evidence_loop_per_hypothesis =
1
axioms (4)
- domain assumption CSE-CIC-IDS2018 ground-truth labels (attack category, attacker/victim IPs, ordered attack stages) are accurate.
- domain assumption GPT-5.1 prompted as a senior security analyst yields valid, reliable quality scores for explanation correctness.
- domain assumption The manually constructed network topology knowledge graph accurately reflects the real network.
- domain assumption LLM-based retrieval agents generate correct and complete SQL/Cypher queries.
read the original abstract
Security operations centers rely on anomaly detection systems to flag suspicious events. Feature-level explanations for anomaly detectors offer limited value for operational investigations. To effectively handle alerts, analysts need to know contextual relationships and need actionable understanding of the entities involved. This paper introduces an event-centric detector-agnostic approach for explaining cybersecurity alerts in small- to medium-sized enterprise networks. We present (EC)2, a multi-agent framework that performs structured, hypothesis-driven investigation to provide explanations grounded in verifiable evidence. Evaluation results show that the proposed framework improves post-detection analysis by generating operationally meaningful explanations, which also enhance event classification accuracy.
Figures
Reference graph
Works this paper leans on
-
[1]
Scientific Reports15(1), 1726 (2025)
Ahmed, U., Nazir, M., Sarwar, A., Ali, T., Aggoune, E.H.M., Shahzad, T., Khan, M.A.: Signature-based intrusion detection using machine learning and deep learn- ing approaches empowered with fuzzy clustering. Scientific Reports15(1), 1726 (2025)
2025
-
[2]
frontiers in computer science, 6 (2024)
Ali, A., Charfeddine, M., Ammar, B., Hamed, B., Albalwy, F., Alqarafi, A., Hus- sain, A.: Unveiling machine learning strategies and considerations in intrusion de- tection systems: A comprehensive survey. frontiers in computer science, 6 (2024)
2024
-
[3]
arXiv preprint arXiv:2309.16021 (2023)
Ali, T., Kostakos, P.: Huntgpt: Integrating machine learning-based anomaly de- tection and explainable ai with large language models (llms). arXiv preprint arXiv:2309.16021 (2023)
Pith/arXiv arXiv 2023
-
[4]
Information fusion58, 82–115 (2020)
Arrieta, A.B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al.: Explainable artifi- cial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information fusion58, 82–115 (2020)
2020
-
[5]
arXiv preprint arXiv:2509.00069 (2025)
Balasubramanian, P., Kankanamge, D., Gilman, E., Oussalah, M.: Anomalyex- plainer explainable ai for llm-based anomaly detection using bertviz and captum. arXiv preprint arXiv:2509.00069 (2025)
Pith/arXiv arXiv 2025
-
[6]
arXiv preprint arXiv:2503.10814 (2025)
Bandyopadhyay, D., Bhattacharjee, S., Ekbal, A.: Thinking machines: A survey of llm based reasoning strategies. arXiv preprint arXiv:2503.10814 (2025)
Pith/arXiv arXiv 2025
-
[7]
In: 2024 IEEE 10th World Forum on Internet of Things (WF-IoT)
Baral, S., Saha, S., Haque, A.: An adaptive end-to-end iot security framework using explainable ai and llms. In: 2024 IEEE 10th World Forum on Internet of Things (WF-IoT). pp. 469–474. IEEE (2024)
2024
-
[8]
Journal of Cybersecurity and Privacy4(4), 777– 793 (2024)
Chamkar, S.A., Maleh, Y., Gherabi, N.: Security operations centers: Use case best practices, coverage, and gap analysis based on mitre adversarial tactics, tech- niques, and common knowledge. Journal of Cybersecurity and Privacy4(4), 777– 793 (2024)
2024
-
[9]
arXiv preprint arXiv:2507.21193 (2025)
Chatzimiltis, S., Shojafar, M., Mashhadi, M.B., Tafazolli, R.: Interpretable anomaly-based ddos detection in ai-ran with xai and llms. arXiv preprint arXiv:2507.21193 (2025)
Pith/arXiv arXiv 2025
-
[10]
NIST Special Publication800(61), 1–147 (2012)
Cichonski, P., Millar, T., Grance, T., Scarfone, K., et al.: Computer security inci- dent handling guide. NIST Special Publication800(61), 1–147 (2012)
2012
-
[11]
arXiv preprint arXiv:2006.11371 (2020)
Das, A., Rad, P.: Opportunities and challenges in explainable artificial intelligence (xai): A survey. arXiv preprint arXiv:2006.11371 (2020)
Pith/arXiv arXiv 2006
-
[12]
Computers and Electrical Engineering129, 110826 (2026)
Houssel, P.R., Layeghy, S., Singh, P., Portmann, M.: ex-nids: A framework for ex- plainable network intrusion detection leveraging large language models. Computers and Electrical Engineering129, 110826 (2026)
2026
-
[13]
Houssel, P., Singh, P., Layeghy, S., Portmann, M.: Towards explainable net- work intrusion detection using large language models. arxiv 2024. arXiv preprint arXiv:2408.04342 (2024)
Pith/arXiv arXiv 2024
-
[14]
Jetlime: ex-nids: A framework for explainable network intrusion detection leverag- ing large language models (2025), https://github.com/jetlime/eX-NIDS
2025
-
[15]
Jüttner, V., Grimmer, M., Buchmann, E.: Chatids: Advancing explainable cyber- security using generative ai. Int. J. Adv. Secur17(1), 2 (2024)
2024
-
[16]
Computers & Security95, 101851 (2020)
Li, X., Chen, W., Zhang, Q., Wu, L.: Building auto-encoder intrusion detection system based on random forest feature selection. Computers & Security95, 101851 (2020)
2020
-
[17]
arXiv preprint arXiv:2503.20796 (2025) 20 N
Lim, B., Huerta, R., Sotelo, A., Quintela, A., Kumar, P.: Explicate: Enhancing phishing detection through explainable ai and llm-powered interpretability. arXiv preprint arXiv:2503.20796 (2025) 20 N. Kirmayer et al
Pith/arXiv arXiv 2025
-
[18]
Advances in neural information processing systems30(2017)
Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. Advances in neural information processing systems30(2017)
2017
-
[19]
arXiv preprint arXiv:2508.21323 (2025)
Mukherjee, K., Kantarcioglu, M.: Llm-driven provenance forensics for threat in- vestigation and detection. arXiv preprint arXiv:2508.21323 (2025)
arXiv 2025
-
[20]
In: 2024 5th International Conference in Electronic Engineering, Information Technology & Education (EEITE)
Nikolakopoulos, A., Evangelatos, S., Veroni, E., Chasapas, K., Gousetis, N., Apos- tolaras, A., Nikolopoulos, C.D., Korakis, T.: Large language models in modern forensic investigations: Harnessing the power of generative artificial intelligence in crime resolution and suspect identification. In: 2024 5th International Conference in Electronic Engineering,...
2024
-
[21]
Journal of Cybersecurity and Privacy5(3), 55 (2025)
Palma, G., Cecchi, G., Caronna, M., Rizzo, A.: Leveraging large language models for scalable and explainable cybersecurity log analysis. Journal of Cybersecurity and Privacy5(3), 55 (2025)
2025
-
[22]
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al.: Improving lan- guage understanding by generative pre-training (2018)
2018
-
[23]
Research, M.: Autogen — open-source framework for agentic ai (2025), https:// www.microsoft.com/en-us/research/project/autogen/
2025
-
[24]
Cognitive Systems Research86, 101243 (2024)
Retzlaff, C.O., Angerschmid, A., Saranti, A., Schneeberger, D., Roettger, R., Mueller, H., Holzinger, A.: Post-hoc vs ante-hoc explanations: xai design guide- lines for data scientists. Cognitive Systems Research86, 101243 (2024)
2024
-
[25]
why should i trust you?
Ribeiro, M.T., Singh, S., Guestrin, C.: " why should i trust you?" explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD interna- tional conference on knowledge discovery and data mining. pp. 1135–1144 (2016)
2016
-
[26]
ICT express10(4), 935–958 (2024)
Sarker, I.H., Janicke, H., Mohsin, A., Gill, A., Maglaras, L.: Explainable ai for cy- bersecurity automation, intelligence and trustworthiness in digital twin: Methods, taxonomy, challenges and prospects. ICT express10(4), 935–958 (2024)
2024
-
[27]
ICISSp1(2018), 108–116 (2018)
Sharafaldin, I., Lashkari, A.H., Ghorbani, A.A., et al.: Toward generating a new intrusion detection dataset and intrusion traffic characterization. ICISSp1(2018), 108–116 (2018)
2018
-
[28]
Journal of Cybersecurity and Privacy5(4), 95 (2025)
Srinivas, S., Kirk, B., Zendejas, J., Espino, M., Boskovich, M., Bari, A., Dajani, K., Alzahrani, N.: Ai-augmented soc: A survey of llms and agents for security automation. Journal of Cybersecurity and Privacy5(4), 95 (2025)
2025
-
[29]
ACM Computing Surveys 57(9), 1–38 (2025)
Tariq, S., Baruwal Chhetri, M., Nepal, S., Paris, C.: Alert fatigue in security op- erations centres: Research challenges and opportunities. ACM Computing Surveys 57(9), 1–38 (2025)
2025
-
[30]
arXiv preprint arXiv:2312.11805 (2023)
Team, G., Anil, R., Borgeaud, S., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., Millican, K., et al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)
Pith/arXiv arXiv 2023
-
[31]
Ieee Access8, 227756–227779 (2020)
Vielberth, M., Böhm, F., Fichtinger, I., Pernul, G.: Security operations center: A systematic study and open challenges. Ieee Access8, 227756–227779 (2020)
2020
-
[32]
In: 2019 IEEE International Conference on Electro Informa- tion Technology (EIT)
Wafula, K., Wang, Y.: Carve: A scientific method-based threat hunting hypothesis development model. In: 2019 IEEE International Conference on Electro Informa- tion Technology (EIT). pp. 1–6. IEEE (2019)
2019
-
[33]
In: 2024 12th International Symposium on Digital Forensics and Security (ISDFS)
Wickramasekara, A., Scanlon, M.: A framework for integrated digital forensic in- vestigation employing autogen ai agents. In: 2024 12th International Symposium on Digital Forensics and Security (ISDFS). pp. 01–06. IEEE (2024)
2024
-
[34]
Wireshark: tshark(1) manual page (2026), https://www.wireshark.org/docs/ man-pages/tshark.html
2026
-
[35]
4 DLdeep learning
Zhang,Z.,AlHamadi,H.,Damiani,E.,Yeun,C.Y.,Taher,F.:Explainableartificial intelligenceapplicationsincybersecurity:State-of-the-artinresearch.IEEEAccess 10, 93104–93139 (2022) Event-Centric Explainability for Cybersecurity 21 Acronyms AIartificial intelligence............................................ 4 DLdeep learning .......................................
2022
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.