{"id":"d45eb8c4-8f7d-4475-a4ad-22e8e962ceac","arxiv_id":"2607.26201","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"An event-centric, multi-agent LLM framework explains network alerts through hypothesis-driven, retrieval-augmented investigation and claims to improve explanation quality and boundary-case classification.","lead":"(EC)2 is a multi-agent AI system that investigates cybersecurity alerts by forming hypotheses, querying network databases, and writing evidence-based reports. It aims to give security analysts the \"why\" behind a detection instead of just the triggering features.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Explanation-quality scores rest solely on an unvalidated LLM judge; RQ1/RQ2 evidence for 'operationally meaningful explanations' is not established.","rationale":"The paper's central claim is that (EC)2 produces operationally meaningful explanations that also improve classification. The evidence for meaningful explanations rests entirely on RQ1/RQ2 scores from a GPT-5.1 judge. No human validation is reported; the vague 'manually reviewed for reliability' is unsubstantiated. While the scoring rubric is tied to ground truth (attack type, IPs, stages), the completeness criterion requires nuanced temporal/sequential judgment where LLM judges are known to be unreliable. The large reported gaps may reflect a stylistic bias toward longer, more detailed narratives. This is load-bearing because if the judge is biased, the primary contribution is unproven. The RQ3 classification-enhancement experiment is also problematic (no LLM-only baseline, balanced sample), but even a valid RQ3 would not compensate for invalid RQ1/RQ2. Therefore I agree with the reader's rejection. A human-rater validation study would be the definitive test.","tokens_in":13254,"tokens_out":5697,"duration_ms":54479,"concrete_test":"Recruit at least two independent security analysts with SOC experience; blind them to system identity; have them score the same 50 event explanations from (EC)2 and eX-NIDS using the same 1–5 rubric and ground truth. Compute inter-rater reliability (e.g., Cohen's kappa or ICC) among humans and between humans and GPT-5.1. If human raters disagree with the LLM judge or do not reproduce the reported quality gap (e.g., (EC)2 > eX-NIDS on all criteria), then RQ1's conclusion is not supported. Also check whether human scores correlate with explanation length; if so, control for verbosity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All RQ1/RQ2 explanation-quality scores are produced by GPT-5.1 prompted as a senior security analyst (§5.1). The only reliability check is 'three independent evaluation runs' (Fig. 6), which measures run-to-run consistency, not agreement with human judgment. The phrase 'manually reviewed for reliability' in §5.1 is unsupported by any described methodology or result. LLM-as-a-judge is known to exhibit systematic biases (length, verbosity, self-preference). Here (EC)2's outputs are long multi-agent narratives while eX-NIDS outputs are single-pass short explanations; an LLM judge could systematically favor the former without those explanations being more accurate or complete. Since the central claim is that (EC)2 generates 'operationally meaningful explanations,' and RQ1/RQ2 are the primary evidence for this, the absence of human-rated ground truth or inter-rater reliability leaves that conclusion unsupported. Score stability cannot rule out a stable but biased judge.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes (EC)2, a multi-agent LLM framework that explains anomalous network events through hypothesis-driven investigation grounded in retrieval-augmented generation over topology and traffic data. The evaluation compares (EC)2 against eX-NIDS on 50 CSE-CIC-IDS2018 events using an LLM-based judge (RQ1), ablates framework components (RQ2), and shows that feeding (EC)2 explanations to an LLM improves binary classification of 1,000 records sampled from the uncertain zone of AE-IDS (RQ3). The paper claims that (EC)2 produces operationally meaningful explanations and also enhances detection accuracy.","tokens_in":13553,"tokens_out":4785,"duration_ms":46986,"significance":"If validated, (EC)2 would be a useful contribution: it addresses a real gap in SOC workflows, where feature-level XAI is insufficient, and the multi-agent architecture with explicit evidence retrieval is a reasonable design. The paper also provides reproducibility advantages by using a public dataset and publicly released eX-NIDS explanations. However, the current evaluation does not substantiate the central claims due to the unvalidated LLM-as-judge methodology for explanation quality and the missing control in the detection-improvement experiment. As presented, the evidence is suggestive but not conclusive.","major_comments":[{"comment":"All RQ1/RQ2 explanation-quality scores are produced by GPT-5.1 prompted as a senior security analyst, with no human-rated ground truth or inter-rater reliability assessment. The claim that criteria are 'manually reviewed for reliability' (p. 12) is not supported by any described protocol or result. The three-run stability check (Fig. 6) only measures stochastic consistency, not agreement with human judgment. Since the central claim that (EC)2 produces 'operationally meaningful explanations' rests entirely on these scores, the evaluation is self-referential and the conclusion is unsupported. Moreover, the judge may be systematically biased toward longer, narrative outputs (multi-agent reports) over the short eX-NIDS explanations. Please add a human evaluation (e.g., expert ratings on a subset) and report agreement statistics, or at minimum validate the judge against a human-annotated benc","section":"§5.1"},{"comment":"The RQ3 evaluation lacks a critical control: an LLM classifying the same raw events (or AE-IDS feature inputs) without (EC)2 explanations. The observed improvement (recall 0.54→0.72, precision 0.56→0.73) could be due to the LLM's prior knowledge or to the balanced 50/50 sample rather than to the explanation content. Additionally, the 1,000-record sample is balanced by construction, but the actual class distribution in the uncertain zone is not reported; metrics computed on this balanced sample are not directly comparable to operational conditions where the base rate is ~17% malicious. Please report the size and class distribution of the full uncertain zone, compute metrics on an unbiased sample (or the full zone), and add an LLM-only baseline (e.g., prompt with raw NetFlow features) to allow causal attribution to (EC)2.","section":"§5.3"},{"comment":"The ablation study performs many paired t-tests (multiple variants × three criteria) without any multiple-comparison correction. Given that the underlying scores are 1–5 discrete ratings derived from an unvalidated LLM judge, the reported p-values should be interpreted cautiously. A non-parametric test (e.g., Wilcoxon) or a correction (e.g., Bonferroni or FDR) would strengthen the ablation conclusions. This is a load-bearing issue for RQ2, as the paper uses these results to claim that database access, planning, and initialization are necessary.","section":"§5.2"}],"minor_comments":[{"comment":"Typographical and grammar issues appear, e.g., 'which their performance is compared' (p. 15) and 'which are evaluated' (p. 16). The paper would benefit from a language edit.","section":"§5.2"},{"comment":"The paper describes (EC)2 as 'detector-agnostic,' but the experiments use only one detector (AE-IDS) and one dataset. A brief discussion of generalizability and any potential dependency on the detector's output format would be helpful.","section":"§1"},{"comment":"The evaluator model (GPT-5.1) and the agent model (GPT-5-mini) are from the same family. Although the paper does not use the judge to compare its own outputs, the potential for model-family bias is not addressed. A human validation or a judge from a different family would mitigate this concern.","section":"§5.1"},{"comment":"The runtime of ~17 minutes per event is mentioned as a limitation, but no systematic analysis of parallel scaling or cost is provided. This is acceptable for an in-depth alert triage use case, but the claim that 'this runtime does not impose fundamental operational limitations' (p. 17) is not fully supported.","section":"§5.3"}],"recommendation":"major_revision","confidential_remarks":"The central idea is promising, but the evaluation as it stands does not support the paper's main claims. The lack of human validation for the LLM judge is a fundamental validity threat that cannot be patched with more automated scores; the RQ3 experiment needs a control to rule out the LLM's own prior knowledge. These issues are addressable in a revision, but they are substantial. I recommend major revision rather than rejection, as the framework itself is interesting and the evaluation gaps are fixable with additional experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, you should know two things about this paper. First, the framework itself is a genuine piece of engineering: a multi-agent LLM pipeline that generates hypotheses, retrieves evidence from a knowledge graph and a traffic database, and produces a structured investigation report. That is new in network anomaly explanation. Second, the evaluation as written does not support the strong claims. Explanation-quality scores come from an LLM judge with no human validation, and the detection-enhancement experiment lacks the control needed to attribute the gain to (EC)2.\n\nWhat's good: The RAG-grounded agents are a real strength, and the ablation study is a useful attempt to isolate component contributions. Comparing to eX-NIDS's published explanations for the same 50 events is a fair use of prior work. RQ3 is an interesting idea: using explanations to reclassify uncertain boundary cases, with a statistically significant improvement on the sample.\n\nSoft spots, in proportion: The main problem is the LLM judge. The paper says scores were 'manually reviewed for reliability' but no human agreement data are provided. Three independent runs only show the judge is stable, not that it is correct. Since (EC)2 outputs long narratives and eX-NIDS outputs short single-pass explanations, a length or verbosity bias could explain the gap. For RQ3, the improvement is real on the 1,000-record balanced sample, but without an LLM-only baseline you cannot tell whether the gain comes from the investigation structure or just from LLM prior knowledge. The balanced sample from the 5% uncertain zone also limits generalization. Minor: no code or data, and 17 minutes per event is a practical constraint.\n\nNone of this makes the paper a waste. The framework is worth engaging with, and the research questions are well-posed. But the evidence, as presented, is not yet strong enough for the central claims.\n\nThis paper is for researchers building LLM-based SOC tools. It would make a decent reading group topic, and it deserves a serious referee. My recommendation: send it to peer review, but expect the authors to be asked for substantial revisions: human evaluation of explanations, an LLM-only control in RQ3, and ideally a more representative sample.","headline":"A well-engineered multi-agent framework for event-centric explanation, but the evaluation's reliance on an unvalidated LLM judge and a missing control leaves the strong claims under-supported.","tokens_in":14002,"tokens_out":3835,"would_cite":false,"duration_ms":36049,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that explaining why an event is anomalous—not why a model flagged it—improves both explanation usefulness and detection accuracy, and supports this with a multi-agent LLM investigation framework.","keywords":["event-centric explainability","multi-agent LLM","hypothesis-driven investigation","retrieval-augmented generation","network anomaly detection","security operations center","explanation quality","detection enhancement"],"falsifier":"Re-run the RQ3 reclassification with the judge LLM given only the raw flagged event, bypassing the investigation pipeline; if recall stays near 0.54 and false-positive rate near 0.43, the reported gains come from the evidence-gathering machinery, while materially different numbers would point to the LLM's prior knowledge or the sample's composition.","tokens_in":13206,"feed_emoji":"🛡️","tokens_out":8456,"duration_ms":74893,"temperature":0.7,"pith_summary":"This paper tries to establish that cybersecurity explainability should shift from model introspection to event investigation: instead of answering 'why did the detector flag this?', an explainer should answer 'why is this event anomalous in its network context?'. It presents (EC)2, a multi-agent LLM framework that runs a hypothesis-driven investigation—proposing hypotheses, retrieving evidence from a network topology graph and a traffic database, evaluating each hypothesis, and writing a structured report—so explanations are grounded in verifiable data. On 50 benchmark events, this approach scores higher than a state-of-the-art single-call LLM explainer on attack-type identification, entity identification, and completeness. The paper further claims the investigation output, used by a judge model to issue a verdict, can reclassify near-threshold alarms: on 1,000 records sampled from the 5% uncertain zone of an autoencoder detector, recall rose from 0.54 to 0.72, precision from 0.56 to 0.73, and false-positive rate fell from 0.43 to 0.26, with McNemar's test at p<0.001. The stakes are practical: security operations centers drown in alerts that lack causal and temporal context, and this reframes explainability as an investigation layer rather than a feature-attribution afterthought.","feed_headline":"Reclassifying alerts by investigation lifts recall from 0.54 to 0.72","feed_subtitle":"Hypothesis-driven multi-agent LLM investigations cut false positives from 43% to 26% on uncertain detections.","key_machinery":"The load-bearing mechanism is the hypothesis-driven investigation loop with retrieval-augmented evidence grounding. A contextual question generator expands the event into factual questions; an evidence-gathering agent routes them to retrieval agents that query a network-topology knowledge graph and a traffic database; a planner, hypothesis generator, and hypothesis evaluator iterate over candidate explanations; a flow evaluator decides when to conclude or re-enter the cycle; and an explanation agent writes the final report, which a reviewer checks for coherence. The retrieval-augmented constraint—agents may only reason over retrieved data, not their internal knowledge—is what converts black-","core_discovery":"On the paper's own terms, the central claim is that the interpretability gap in security operations is not a feature-attribution problem but an investigation problem. (EC)2 operationalizes this by treating each flagged event as a case: agents propose hypotheses, retrieve evidence from a network-topology knowledge graph and a packet/flow traffic database, evaluate each hypothesis, and produce a narrative report documenting supported, refuted, and unresolved hypotheses. The paper reports that this approach outperforms a prompt-augmented, single-inference LLM explainer on all three quality criteria (attack type 3.21 vs 1.49, entity 4.10 vs 3.22, completeness 3.04 vs 1.29), and that feeding the","pith_inferences":["The RQ3 design leaves room for a confound: the judge LLM may rely on its own attack knowledge rather than the evidence the framework retrieved. A control condition feeding the same judge the raw event without investigation would isolate whether the gain comes from the evidence-gathering machinery.","Explanation quality is scored by an LLM judge without reported human validation; a natural next step is measuring agreement between LLM scores and experienced security analysts, or fact-checking reports against ground-truth databases.","The runtime economics point toward adaptive investigation depth: stop after the first well-supported hypothesis on clear cases and reserve the full loop for genuinely ambiguous events, yielding a measurable accuracy-versus-latency tradeoff.","The event-centric framing should transfer beyond security to any domain with relational data and temporal logs—fraud, health monitoring, cloud operations—where the question 'why is this observation anomalous?' benefits from the same hypothesis-testing structure."],"forward_implications":["Analysts receive chronological, evidence-grounded reconstructions of attack stages and entity roles, instead of feature-importance lists.","A post-hoc investigation layer can be attached to any detector, treating it as a black box, without retraining or internal access.","Near-threshold detections can be reclassified by routing them through the investigation pipeline, reducing both missed attacks and false alarms.","Ablation results indicate that database access and the planning/initialization components drive explanation quality, while the reviewer loop adds no measurable improvement.","The roughly 17-minute runtime per investigation restricts the framework to high-priority or ambiguous alerts rather than full-stream triage."],"fun_headline_variants":["Multi-agent LLM investigation lifts alert recall from 0.54 to 0.72","LLM investigation agents cut false positive rate from 43% to 26%","Event-centric LLM agents beat single-inference in explaining cyber alerts","Alert recall rises to 0.72, false positives fall to 26%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The results stand on the assumption that a large language model prompted as a senior security analyst produces valid, reliable quality scores and verdicts without human validation or inter-rater checks, and that the 1,000-record balanced sample from the uncertain zone is representative enough to attribute the classification gains to the framework's evidence retrieval rather than to the model's prior knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent LLM investigation lifts alert recall from 0.54 to 0.72","LLM investigation agents cut false positive rate from 43% to 26%","Event-centric LLM agents beat single-inference in explaining cyber alerts","Alert recall rises to 0.72, false positives fall to 26%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000739,"raw_usage":{"total_tokens":3088,"prompt_tokens":647,"completion_tokens":2441,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":391,"completion_tokens_details":{"reasoning_tokens":2355}},"tokens_in":391,"tokens_out":2441,"duration_ms":17952,"temperature":1.0,"reasoning_tokens":2355,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T00:29:12.241954+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the RQ3 reclassification with the judge LLM given only the raw flagged event, bypassing the investigation pipeline; if recall stays near 0.54 and false-positive rate near 0.43, the reported gains come from the evidence-gathering machinery, while materially different numbers would point to the LLM's prior knowledge or the sample's composition.","supporting_citations":[],"review_version":1}