REVIEW 18 cited by
CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large language model (LLM) agents are increasingly capable of autonomously conducting cyberattacks, posing significant threats to existing applications. This growing risk highlights the urgent need for a real-world benchmark to evaluate the ability of LLM agents to exploit web application vulnerabilities. However, existing benchmarks fall short as they are limited to abstracted Capture the Flag competitions or lack comprehensive coverage. Building a benchmark for real-world vulnerabilities involves both specialized expertise to reproduce exploits and a systematic approach to evaluating unpredictable threats. To address this challenge, we introduce CVE-Bench, a real-world cybersecurity benchmark based on critical-severity Common Vulnerabilities and Exposures. In CVE-Bench, we design a sandbox framework that enables LLM agents to exploit vulnerable web applications in scenarios that mimic real-world conditions, while also providing effective evaluation of their exploits. Our evaluation shows that the state-of-the-art agent framework can resolve up to 13% of vulnerabilities.
Forward citations
Cited by 18 Pith papers
-
The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark
SRE-Bench, a contamination-controlled reverse-engineering benchmark with 262 realistic binary instances, shows the strongest tested AI agent fully solves only 31.5% of instances.
-
From Runnable to Verifiable: An Independent Reproducibility Study of LLM/Agent-Driven Vulnerability Validation Artifacts
A pre-registered audit of 102 cases plus 18 papers found artifact-embedded oracles unreliable: 20 of 30 signals also triggered on patched builds, so runnable artifacts rarely confirm the CVE.
-
AgentSnare: Learning to Delay, Divert, and Defuse Autonomous Penetration Agents
A trajectory-adaptive honeypot system, AgentSnare, achieves a 0/45 verified exploit rate against LLM-based penetration testers across 15 vulnerable web apps and three attacker models.
-
SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response
No frontier LLM agent fully detects and remediates any of 10 real post-compromise host ranges; alert-driven findings work, silent intrusion and verified cleanup do not.
-
Certifying Ghosts: How Cybersecurity AI Agents Break the EU Cyber Resilience Act
Cybersecurity AI agents invalidate the CRA's premises about vulnerability discovery, exploitation, and remediation speed, making static conformity certificates false and requiring continuous agent-operated defense.
-
AXE: Grey-Box Exploitability Confirmation for Localized Vulnerability Reports
Grey-box metadata (CWE + code location) plus a multi-agent LLM workflow raises automated web-exploit confirmation from ~10% to 30% on CVE-Bench, with actionable PoC output.
-
FaultLine: Automated Proof-of-Vulnerability Generation Using LLM Agents
A staged LLM agent that reasons about data flow and branch conditions generated valid proof-of-vulnerability tests for 16 of 100 Java, C, and C++ projects, versus 9 for the CodeAct 2.1 baseline.
-
LLMs unlock new paths to monetizing exploits
LLMs can already mine personal data for blackmail, find real vulnerabilities in obscure software, and automate in-browser and server-side attacks, changing the economics of cybercrime.
-
Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction
Tencent WorkBuddy Bench is a 260-task, four-domain coding-agent benchmark whose colloquial prompts are reverse-engineered from real commits, CVEs, and business scenarios, released openly with a dual-harness cross-mode...
-
Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents
Security-agent success changes differently with budget: offensive CTF tasks improve with more compute, while defensive SOC work depends more on tool discipline than spend.
-
From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World
A practical evaluation protocol for AI pentesting agents that uses validated vulnerability discovery, LLM semantic matching, and bipartite scoring to assess performance in realistic, complex targets.
-
Reliable Weak-to-Strong Monitoring of LLM Agents
Monitor scaffolding, not monitor awareness or omniscience, drives detection reliability, and a hybrid chunked monitor lets weak models supervise strong LLM agents.
-
Establishing Best Practices for Building Rigorous Agentic Benchmarks
Agentic benchmarks frequently mis-grade agents, and the new ABC checklist helps identify and correct such errors in ten popular benchmarks.
-
Eradicating the Unseen: Detecting, Exploiting, and Remediating a Path Traversal Vulnerability across GitHub
A single vulnerable Node.js path traversal pattern was found in 1,756 GitHub projects, most rated critical, and the authors' automated pipeline produced patches, disclosures, and evidence that LLMs have learned the pattern.
-
Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Response
A structured review organizes cyber-capable-agent risks into five vulnerability classes and argues that evaluation environments must be treated as operational security systems rather than background.
-
Running in CIRCLE? A Simple Benchmark for LLM Code Interpreter Security
CIRCLE is a 1,260-prompt benchmark that measures how often commercial LLM code interpreters refuse, execute, or time out on resource-exhaustion tasks, revealing large and inconsistent safety gaps.
-
Agent Identity Evals: Measuring Agentic Identity
Introduces Agent Identity Evals (AIE), five similarity-based metrics for LMA identity stability, with pilot experiments showing identifiability always at zero and no statistical support.
-
Rethinking Agent Security as a Networking Problem
The authors propose treating AI agent security as a networking problem, with centralized policy control and per-agent sidecars that enforce both deterministic and context-aware rules.
Discussion (0). Continue with ORCID to comment.