A new benchmark shows frontier LLMs achieve only 3.8% average recall identifying malicious events from raw logs and fail to meet 50% recall thresholds on most tactics.
ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation
3 Pith papers cite this work. Polarity classification is still indexing.
3
Pith papers citing it
abstract
We present ExCyTIn-Bench, the first benchmark to Evaluate an LLM agent X on the task of Cyber Threat Investigation through security questions derived from investigation graphs. Real-world security analysts must sift through a large number of heterogeneous security logs, follow multi-hop chains of evidence to investigate threats. With the developments of LLMs, building LLM-based agents for automatic threat investigation is a promising direction. We construct a benchmark from a controlled Azure tenant including a SQL environment covering 57 log tables from Microsoft Sentinel and related services, and 7542 generated questions. We leverage security logs extracted with expert-crafted detection logic to build threat investigation graphs, and then generate questions with LLMs using paired nodes on the graph, taking the start node as background context and the end node as answer. Anchoring each question to these explicit nodes and edges not only provides automatic, explainable ground truth answers but also makes the pipeline reusable and readily extensible to new logs. Our comprehensive experiments on the test set with different models confirm the difficulty of the task: the best model so far can achieve a reward of 0.606, leaving much headroom for future research. The code is available at https://github.com/microsoft/SecRL
fields
cs.CR 3years
2026 3representative citing papers
DTDA is an LLM agent that produces novel security alerts at 80.1% customer-validated precision and 0.78 F1 on hidden activity while running at production scale inside Microsoft Defender.
Pen-Strategist fine-tunes Qwen-3-14B with RL on a pentesting reasoning dataset and pairs it with a CNN step classifier, reporting 87% better strategy derivation, 47.5% more subtask completions than baselines, and gains on CTFKnow and user studies.
citing papers explorer
-
Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps
A new benchmark shows frontier LLMs achieve only 3.8% average recall identifying malicious events from raw logs and fail to meet 50% recall thresholds on most tactics.
-
GenAI-Driven Threat Detection with Microsoft Security Copilot
DTDA is an LLM agent that produces novel security alerts at 80.1% customer-validated precision and 0.78 F1 on hidden activity while running at production scale inside Microsoft Defender.
-
Pen-Strategist: A Reasoning Framework for Penetration Testing Strategy Formation and Analysis
Pen-Strategist fine-tunes Qwen-3-14B with RL on a pentesting reasoning dataset and pairs it with a CNN step classifier, reporting 87% better strategy derivation, 47.5% more subtask completions than baselines, and gains on CTFKnow and user studies.