Introduces the first open multi-host cyber range benchmark AgentCyberRange with Cage toolchain and evaluates six frontier AI systems on web exploitation and post-exploitation tasks across 110 vulnerabilities.
Cve-bench: A benchmark for ai agents’ ability to exploit real- world web application vulnerabilities.arXiv preprint arXiv:2503.17332, Mar 2025
10 Pith papers cite this work. Polarity classification is still indexing.
10
Pith papers citing it
abstract
Large language model (LLM) agents are increasingly capable of autonomously conducting cyberattacks, posing significant threats to existing applications. This growing risk highlights the urgent need for a real-world benchmark to evaluate the ability of LLM agents to exploit web application vulnerabilities. However, existing benchmarks fall short as they are limited to abstracted Capture the Flag competitions or lack comprehensive coverage. Building a benchmark for real-world vulnerabilities involves both specialized expertise to reproduce exploits and a systematic approach to evaluating unpredictable threats. To address this challenge, we introduce CVE-Bench, a real-world cybersecurity benchmark based on critical-severity Common Vulnerabilities and Exposures. In CVE-Bench, we design a sandbox framework that enables LLM agents to exploit vulnerable web applications in scenarios that mimic real-world conditions, while also providing effective evaluation of their exploits. Our evaluation shows that the state-of-the-art agent framework can resolve up to 13% of vulnerabilities.
citation-role summary
background 1
citation-polarity summary
roles
background 1polarities
background 1representative citing papers
Cybersecurity AI agents invalidate the CRA's premises about vulnerability discovery, exploitation, and remediation speed, making static conformity certificates false and requiring continuous agent-operated defense.
ZERO-APT is a closed-loop framework that integrates an LLM attacker, configurable LLM defender, and judge agent to achieve 79% attack success rate, 0.860 causal consistency, and full decision auditability in penetration testing under intelligent defense.
CyberEvolver introduces a four-layer self-evolving agent architecture with trace-to-diagnosis and population beam search that raises seed agent success rates by 13.6% on CTF, exploitation, and penetration tasks across four LLMs.
LLM agents exhibit persistent attack-selection biases as fixed traits independent of success rates, with a bias momentum effect that resists steering and yields no performance gain.
RealVuln benchmark finds security-specialized scanners outperform general-purpose LLMs and rule-based SAST tools on hand-labeled vulnerable Python code under F3 scoring, with all artifacts released.
An evaluation protocol for AI pentesting agents that scores validated vulnerability discovery using LLM-based semantic matching and bipartite resolution.
A tiered server benchmark with 300 targets shows current LLMs achieve autonomous penetration success rates of 10.7-69.3% using only general cybersecurity tools and no target-specific knowledge.
FORGE deploys a fixed five-agent pipeline on 603 CVEs to achieve 67.8% L1+ exploitation success at $1.50 per CVE while generating detection rules whose grounding improves with deeper exploitation traces.
xOffense automates penetration testing via a fine-tuned Qwen3-32B LLM in a multi-agent setup with specialized agents for reconnaissance, vulnerability scanning, and exploitation, reporting 79.17% sub-task completion on AutoPenBench and AI-Pentest-Benchmark.
citing papers explorer
-
AgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber Ranges
Introduces the first open multi-host cyber range benchmark AgentCyberRange with Cage toolchain and evaluates six frontier AI systems on web exploitation and post-exploitation tasks across 110 vulnerabilities.
-
Certifying Ghosts: How Cybersecurity AI Agents Break the EU Cyber Resilience Act
Cybersecurity AI agents invalidate the CRA's premises about vulnerability discovery, exploitation, and remediation speed, making static conformity certificates false and requiring continuous agent-operated defense.
-
ZERO-APT: A Closed-Loop Adversarial Framework for LLM-Driven Automated Penetration Testing under Intelligent Defense
ZERO-APT is a closed-loop framework that integrates an LLM attacker, configurable LLM defender, and judge agent to achieve 79% attack success rate, 0.860 causal consistency, and full decision auditability in penetration testing under intelligent defense.
-
CyberEvolver: Structured Self-Evolution for Cybersecurity Agents On the Fly
CyberEvolver introduces a four-layer self-evolving agent architecture with trace-to-diagnosis and population beam search that raises seed agent success rates by 13.6% on CTF, exploitation, and penetration tasks across four LLMs.
-
CyBiasBench: Benchmarking Bias in LLM Agents for Cyber-Attack Scenarios
LLM agents exhibit persistent attack-selection biases as fixed traits independent of success rates, with a bias momentum effect that resists steering and yields no performance gain.
-
RealVuln: Benchmarking Rule-Based, General-Purpose LLM, and Security-Specialized Scanners on Real-World Code
RealVuln benchmark finds security-specialized scanners outperform general-purpose LLMs and rule-based SAST tools on hand-labeled vulnerable Python code under F3 scoring, with all artifacts released.
-
From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World
An evaluation protocol for AI pentesting agents that scores validated vulnerability discovery using LLM-based semantic matching and bipartite resolution.
-
The Emergence of Autonomous Penetration Capabilities in Large Language Model-Powered AI Systems
A tiered server benchmark with 300 targets shows current LLMs achieve autonomous penetration success rates of 10.7-69.3% using only general cybersecurity tools and no target-specific knowledge.
-
FORGE: Multi-Agent Graduated Exploitation and Detection Engineering
FORGE deploys a fixed five-agent pipeline on 603 CVEs to achieve 67.8% L1+ exploitation success at $1.50 per CVE while generating detection rules whose grounding improves with deeper exploitation traces.
-
xOffense: An Autonomous Multi-Agent Framework for Penetration Testing with Domain-Adapted Large Language Models
xOffense automates penetration testing via a fine-tuned Qwen3-32B LLM in a multi-agent setup with specialized agents for reconnaissance, vulnerability scanning, and exploitation, reporting 79.17% sub-task completion on AutoPenBench and AI-Pentest-Benchmark.