A decoupled evaluation framework shows LLM penetration agents reach 90% exploitation success with ground-truth context but only 50% reconnaissance recall due to telemetry parsing failures across 50 vulnerabilities.
CRAKEN: cybersecurity LLM agent with knowledge-based execution.CoRR, abs/2505.17107
11 Pith papers cite this work. Polarity classification is still indexing.
11
Pith papers citing it
citation-role summary
background 1
citation-polarity summary
years
2026 11roles
background 1polarities
support 1representative citing papers
Honeyval evaluates LLM HTTP honeypots with AI attackers and shows they produce longer interactions, lower detection rates, and cost advantages over rule-based baselines.
CyberEvolver introduces a four-layer self-evolving agent architecture with trace-to-diagnosis and population beam search that raises seed agent success rates by 13.6% on CTF, exploitation, and penetration tasks across four LLMs.
CrackMeBench introduces 20 deterministic binary validation tasks and reports GPT-5.5 solving 11/12 generated ones at pass@3 while Claude and Kimi lag, especially on harder tasks.
Dynamic Cyber Ranges with LLM defender agents reduce attacker success to 0-55% and preserve evaluation headroom as models advance by using comparable capabilities on both sides.
HunterAgent combines LLM hypothesis generation with symbolic verification and cost-bounded graph search to reconstruct attack paths under anti-forensics, reporting 86.1% mean F1 on benchmarks with reduced hallucinations.
uGen is the first retrieval-augmented multi-agent LLM framework for generating functionally correct microarchitectural attack PoCs, reporting up to 100% success on Spectre-v1 and 80% on Prime+Probe at low cost.
Veritas detects out-of-bounds vulnerabilities in stripped binaries at 90% recall by grounding LLM reasoning in static witness-backed flows and runtime validation.
Static CTF benchmarks for LLM agents are contamination-prone; CTFusion evaluates agents on live CTFs via an MCP server on CTFd with per-agent isolation.
Claude 4.5 Opus reaches 59% solve rate on offensive cyber CTF tasks, with a Kali Linux environment adding 9.5 percentage points over Ubuntu while prompt engineering often hurts performance in equipped setups.
An LLM-agent framework with RAG generates structured vulnerability analysis reports from source code, achieving 54.21% average quality on 105 NIST-SARD samples evaluated by an LLM judge.
citing papers explorer
-
Decoupling Reconnaissance and Exploitation: Measuring the Capability Boundaries of LLM-Based Web Penetration Testing
A decoupled evaluation framework shows LLM penetration agents reach 90% exploitation success with ground-truth context but only 50% reconnaissance recall due to telemetry parsing failures across 50 vulnerabilities.
-
Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots
Honeyval evaluates LLM HTTP honeypots with AI attackers and shows they produce longer interactions, lower detection rates, and cost advantages over rule-based baselines.
-
CyberEvolver: Structured Self-Evolution for Cybersecurity Agents On the Fly
CyberEvolver introduces a four-layer self-evolving agent architecture with trace-to-diagnosis and population beam search that raises seed agent success rates by 13.6% on CTF, exploitation, and penetration tasks across four LLMs.
-
CrackMeBench: Binary Reverse Engineering for Agents
CrackMeBench introduces 20 deterministic binary validation tasks and reports GPT-5.5 solving 11/12 generated ones at pass@3 while Claude and Kimi lag, especially on harder tasks.
-
Dynamic Cyber Ranges
Dynamic Cyber Ranges with LLM defender agents reduce attacker success to 0-55% and preserve evaluation headroom as models advance by using comparable capabilities on both sides.
-
HunterAgent: Neuro-Symbolic Attack Trace Reconstruction under Anti-Forensics
HunterAgent combines LLM hypothesis generation with symbolic verification and cost-bounded graph search to reconstruct attack paths under anti-forensics, reporting 86.1% mean F1 on benchmarks with reduced hallucinations.
-
uGen: An Agentic Framework for Generating Microarchitectural Attack PoCs
uGen is the first retrieval-augmented multi-agent LLM framework for generating functionally correct microarchitectural attack PoCs, reporting up to 100% success on Spectre-v1 and 80% on Prime+Probe at low cost.
-
Veritas: Grounding LLM Agents for Reliable Vulnerability Reasoning over Stripped Binaries
Veritas detects out-of-bounds vulnerabilities in stripped binaries at 90% recall by grounding LLM reasoning in static witness-backed flows and runtime validation.
-
CTFusion: A CTF-based Benchmark for LLM Agent Evaluation
Static CTF benchmarks for LLM agents are contamination-prone; CTFusion evaluates agents on live CTFs via an MCP server on CTFd with per-agent isolation.
-
Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks
Claude 4.5 Opus reaches 59% solve rate on offensive cyber CTF tasks, with a Kali Linux environment adding 9.5 percentage points over Ubuntu while prompt engineering often hurts performance in equipped setups.
-
RAVEN: Retrieval-Augmented Vulnerability Exploration Network for Memory Corruption Analysis in User Code and Binary Programs
An LLM-agent framework with RAG generates structured vulnerability analysis reports from source code, achieving 54.21% average quality on 105 NIST-SARD samples evaluated by an LLM judge.