{"id":"7f771dca-3f3d-4e2c-bca6-6e774ee2be2e","arxiv_id":"2606.21059","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"DEFENGRAPH integrates a dual-layer static-dynamic KG with LLMs via path retrieval, filtering, and re-ranking, raising reasoning-recall from 61.45% to 73.49% and ticket-action recall from 52.17% to 72.46% on GPT-4o in live red-blue cyber range data.","lead":"DEFENGRAPH augments LLMs with a dual-layer static-dynamic knowledge graph, graph path retrieval, contextual filtering, and re-ranking to ground outputs in both fixed domain knowledge and live event data for cyber defense. If the reported gains hold, it offers a concrete way to reduce hallucinations and improve temporal awareness in high-stakes incident response.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"KG fidelity to domain knowledge and event context remains the unverified link between the dual-layer graph and the reported recall gains.","rationale":"The reader's weakest_assumption directly isolates the same empirical precondition required for the headline numbers to be causally attributable to DEFENGRAPH rather than to ancillary context. Because the original verdict was already UNVERDICTED on the basis of missing methodological detail, confirming or refuting this assumption via the concrete ablation would either leave the verdict unchanged or move it to CONDITIONAL; no stronger internal inconsistency is visible from the given claim.","tokens_in":1903,"tokens_out":363,"duration_ms":19381,"concrete_test":"From the KG-construction subsection, extract the exact procedure for building the dynamic layer and the list of artifacts used; recompute the four-LLM evaluation after replacing the graph retrieval step with flat concatenation of the same artifacts (no paths or re-ranking); if the recall deltas fall below 50% of the reported gains, the graph component is not load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim attributes the SOTA lifts (e.g., reasoning-recall 61.45%→73.49%, ticket-action recall 52.17%→72.46% on GPT-4o) to the Static-Dynamic KG plus path retrieval, LLM filtering, and re-ranking. This requires that the graphs constructed from SIEM alerts, topology, attacker behaviors, and prior actions faithfully encode both long-term knowledge and evolving context without material omissions or noise amplification. The abstract supplies no coverage metrics, expert validation of graph completeness, or ablation isolating graph structure from raw text augmentation, so the observed improvements could arise from simply supplying more structured context rather than graph-specific mechanisms.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces DEFENGRAPH, an LLM-driven assistant for blue-team cyber defense that augments models via a dual-layer Static-Dynamic Knowledge Graph (KG) built from SIEM alerts, system topology, attacker behaviors, and prior actions. Graph path retrieval, LLM contextual filtering, and reasoning-based re-ranking are used to ground outputs in long-term knowledge and evolving context. On data from live Red-vs-Blue cyber-range exercises, the system is reported to raise reasoning-recall from 61.45% to 73.49% and ticket-action recall from 52.17% to 72.46% (precision 24.49% to 29.24%) on GPT-4o, with analogous gains on LLaMA-3, DeepSeek-R1 and QWen-3, while surfacing up to 50 correct actions versus 36 for the next baseline and keeping fault rates steady.","tokens_in":2039,"tokens_out":687,"duration_ms":26254,"significance":"If the empirical claims are substantiated with full methodological detail, the work would offer a concrete demonstration that dual-layer KGs can measurably improve LLM reliability for time-sensitive defensive decision support on realistic, noisy security data. The multi-model evaluation and use of live exercise traces are strengths that increase external validity relative to purely synthetic benchmarks.","major_comments":[{"comment":"Abstract and Evaluation section: the concrete lifts (reasoning-recall 61.45%→73.49%, ticket-action recall 52.17%→72.46% on GPT-4o) are stated without definitions of the recall/precision metrics, descriptions of baseline implementations, statistical significance tests, or data-exclusion rules applied to the cyber-range dataset. These omissions are load-bearing for any claim that the gains are attributable to the proposed mechanisms rather than confounds or metric choices.","section":"Abstract and Evaluation section"},{"comment":"Methodology / KG construction section: the central attribution of performance gains to the Static-Dynamic KG plus path retrieval, filtering, and re-ranking rests on the unverified assumption that the constructed graphs faithfully encode both long-term domain knowledge and evolving event context. No coverage metrics, expert validation of completeness, or ablation isolating graph structure from raw-text augmentation are supplied, leaving open the possibility that improvements arise simply from additional structured context.","section":"Methodology / KG construction section"},{"comment":"Results table (action-surfacing numbers): the claim of surfacing up to 50 correct defense actions versus 36 for the next baseline is presented without variance estimates, per-exercise breakdowns, or confirmation that the same set of ground-truth actions was used for all systems, undermining the cross-baseline comparison.","section":"Results table"}],"minor_comments":[{"comment":"The notation distinguishing static versus dynamic layers in the KG could be made more explicit, ideally with an accompanying diagram that labels edge types and temporal scopes.","section":"Figure 1 / §3"},{"comment":"A small number of recent KG-augmented LLM papers in the security domain are not referenced in the related-work section.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive and detailed feedback. We address each major comment below with clarifications and commitments to revisions that strengthen the manuscript's transparency without altering its core claims.","responses":[{"response":"We agree these details are essential for rigorous interpretation. The revised manuscript will add explicit definitions of reasoning-recall and ticket-action recall (including how ground-truth positives are identified from the annotated cyber-range traces), full descriptions of baseline implementations, results from statistical significance tests such as McNemar's test, and a statement of any data-exclusion rules. These additions will be placed in a new subsection of the Evaluation section.","revision_made":"yes","referee_comment":"[Abstract and Evaluation section] Abstract and Evaluation section: the concrete lifts (reasoning-recall 61.45%→73.49%, ticket-action recall 52.17%→72.46% on GPT-4o) are stated without definitions of the recall/precision metrics, descriptions of baseline implementations, statistical significance tests, or data-exclusion rules applied to the cyber-range dataset. These omissions are load-bearing for any claim that the gains are attributable to the proposed mechanisms rather than confounds or metric choices."},{"response":"The manuscript details KG construction from SIEM alerts, topology, attacker behaviors, and prior actions, with consistent gains across four LLMs supporting the value of the structured dual-layer approach. We will add quantitative coverage metrics for both Static and Dynamic layers. Formal expert validation of completeness was not performed in the original study; we will note this limitation explicitly. A full ablation isolating graph structure from raw-text context was not conducted; we will either add a targeted ablation where feasible or discuss it as future work while maintaining that path retrieval and re-ranking provide benefits beyond unstructured augmentation.","revision_made":"partial","referee_comment":"[Methodology / KG construction section] Methodology / KG construction section: the central attribution of performance gains to the Static-Dynamic KG plus path retrieval, filtering, and re-ranking rests on the unverified assumption that the constructed graphs faithfully encode both long-term domain knowledge and evolving event context. No coverage metrics, expert validation of completeness, or ablation isolating graph structure from raw-text augmentation are supplied, leaving open the possibility that improvements arise simply from additional structured context."},{"response":"The 50 versus 36 figures represent the maximum correct actions surfaced across the set of exercises. The revision will include variance estimates (standard deviation across exercises), per-exercise breakdowns in an appendix, and an explicit statement of the evaluation protocol confirming that identical ground-truth action sets—derived from the same annotated traces—were used for every system and baseline.","revision_made":"yes","referee_comment":"[Results table] Results table (action-surfacing numbers): the claim of surfacing up to 50 correct defense actions versus 36 for the next baseline is presented without variance estimates, per-exercise breakdowns, or confirmation that the same set of ground-truth actions was used for all systems, undermining the cross-baseline comparison."}],"tokens_in":1703,"tokens_out":607,"duration_ms":24074,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main things to know are that this paper reports concrete recall improvements on GPT-4o and three other LLMs when a static-dynamic knowledge graph is added to support defense decisions, and that the evaluation uses data from actual red-blue cyber range exercises on critical infrastructure. The architecture combines graph path retrieval, LLM filtering, and re-ranking to ground outputs in both long-term knowledge and live events.\n\nWhat is new is the explicit dual-layer KG design tailored to SIEM alerts, topology, attacker behaviors, and prior actions, then applied to noisy live exercise logs. The paper does a reasonable job testing the same pipeline across multiple models and showing consistent direction of gains, with more correct actions surfaced while fault rates stay flat. Using real exercise data instead of clean benchmarks is a practical strength for this domain.\n\nThe soft spots sit in the evaluation. The abstract states the percentage lifts but supplies no definitions for reasoning-recall or ticket-action recall, no details on baseline implementations, no statistical tests, and no ablations that isolate the graph mechanisms from simply feeding the LLM more structured text. Without those, the claim that the dual-layer KG is responsible remains unverified, exactly as the stress-test note flags. Graph fidelity to the domain is assumed rather than demonstrated.\n\nThis work is aimed at researchers building grounded LLM assistants for security operations. A reader who needs applied examples of KG augmentation in high-stakes, time-sensitive settings will find usable ideas here. It deserves peer review because the empirical setting is relevant and the architecture is described at a level that can be examined, even though the current methods section will require substantial expansion and controls.","headline":"DEFENGRAPH shows recall gains on cyber range data with a dual-layer KG pipeline, but the abstract gives no evidence that the graph structure itself produces the lift rather than added context.","tokens_in":2562,"tokens_out":409,"would_cite":false,"duration_ms":15748,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A dual-layer static-dynamic knowledge graph grounds LLMs to raise accuracy in cyber defense reasoning and actions.","keywords":["knowledge graph","large language models","cybersecurity","blue team defense","SIEM alerts","cyber range exercises","decision support","graph retrieval"],"falsifier":"Replacing the constructed knowledge graphs with random connections or incomplete security data and checking whether the recall gains on reasoning and actions disappear would settle whether the graph integration drives the results.","tokens_in":2804,"feed_emoji":"🛡️","tokens_out":691,"duration_ms":29006,"temperature":0.7,"pith_summary":"The paper introduces DEFENGRAPH, an LLM assistant for blue team cyber defense that builds a dual-layer static-dynamic knowledge graph from SIEM alerts, system topology, attacker behaviors, and prior defensive actions. Graph-based path retrieval, LLM-driven contextual filtering, and reasoning-based re-ranking then connect long-term domain knowledge with evolving event context. Evaluations on data from live Red versus Blue cyber range exercises show higher reasoning-recall and ticket-action recall across GPT-4o, LLaMA-3, DeepSeek-R1, and QWen-3 while fault rates stay steady and more correct defense actions surface. The approach targets hallucinations and shallow temporal reasoning that limit plain LLMs in high-stakes, time-evolving settings.","feed_headline":"Dual-layer graph raises LLM cyber defense recall","feed_subtitle":"Static-dynamic knowledge integration improves reasoning and action accuracy on data from live attack simulations.","key_machinery":"Dual-layer Static-Dynamic Knowledge Graph together with graph-based path retrieval, LLM-driven contextual filtering, and reasoning-based re-ranking.","core_discovery":"DEFENGRAPH integrates a dual-layer Static-Dynamic Knowledge Graph with graph-based path retrieval, LLM-driven contextual filtering, and reasoning-based re-ranking to ground LLM outputs in both long-term domain knowledge and evolving event context from heterogeneous security artifacts, enabling faithful and temporally aware decision support as measured by improved recall metrics on realistic noisy datasets from cyber range exercises.","pith_inferences":["The same grounding pattern could apply to other high-stakes domains that require tracking evolving states, such as network operations or industrial control.","Structured external memory may allow smaller or less specialized models to reach performance levels otherwise needing larger ones.","Maintaining an accurate dynamic layer in live environments would require automated update mechanisms not tested in the exercises."],"forward_implications":["Reasoning-recall rises from 61.45% to 73.49% on GPT-4o.","Ticket-action recall rises from 52.17% to 72.46% on GPT-4o with precision moving from 24.49% to 29.24%.","Up to 50 correct defense actions surface versus 36 for the next best baseline.","Comparable recall gains appear on LLaMA-3, DeepSeek-R1, and QWen-3.","Fault rates remain steady across the tested models."],"fun_headline_variants":["Dual-layer KG sharpens LLM recall in cyber defense","Static-dynamic graph grounds LLM outputs in incidents","DEFENGRAPH uses dual KG for temporally aware LLM aid","Graph retrieval refines LLM reasoning on live attack data"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Knowledge graphs built from SIEM alerts, system topology, attacker behaviors, and prior defensive actions faithfully represent both long-term domain knowledge and evolving event context in a manner that directly enables the observed improvements.","fun_headline_variants_meta":{"raw":{"variants":["Dual-layer KG sharpens LLM recall in cyber defense","Static-dynamic graph grounds LLM outputs in incidents","DEFENGRAPH uses dual KG for temporally aware LLM aid","Graph retrieval refines LLM reasoning on live attack data"]},"model":"grok-4.3","cost_usd":0.004702,"raw_usage":{"total_tokens":2363,"prompt_tokens":749,"num_sources_used":0,"completion_tokens":54,"cost_in_usd_ticks":47024500,"prompt_tokens_details":{"text_tokens":749,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1560,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":749,"tokens_out":54,"duration_ms":11690,"temperature":1.0,"reasoning_tokens":1560,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T14:09:20.639339+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Replacing the constructed knowledge graphs with random connections or incomplete security data and checking whether the recall gains on reasoning and actions disappear would settle whether the graph integration drives the results.","supporting_citations":[],"review_version":1}