REVIEW 27 cited by
Teams of LLM Agents can Exploit Zero-Day Vulnerabilities
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
LLM agents have become increasingly sophisticated, especially in the realm of cybersecurity. Researchers have shown that LLM agents can exploit real-world vulnerabilities when given a description of the vulnerability and toy capture-the-flag problems. However, these agents still perform poorly on real-world vulnerabilities that are unknown to the agent ahead of time (zero-day vulnerabilities). In this work, we show that teams of LLM agents can exploit real-world, zero-day vulnerabilities. Prior agents struggle with exploring many different vulnerabilities and long-range planning when used alone. To resolve this, we introduce HPTSA, a system of agents with a planning agent that can launch subagents. The planning agent explores the system and determines which subagents to call, resolving long-term planning issues when trying different vulnerabilities. We construct a benchmark of 14 real-world vulnerabilities and show that our team of agents improve over prior agent frameworks by up to 4.3X.
Forward citations
Cited by 27 Pith papers
-
PTSan: A Practical Memory Safety Sanitizer for C/C++ with Pointer-Object Authority
A pointer-tagging sanitizer brings object-authority memory checking to about AddressSanitizer-class speed with near-native memory overhead.
-
Certifying Ghosts: How Cybersecurity AI Agents Break the EU Cyber Resilience Act
Cybersecurity AI agents invalidate the CRA's premises about vulnerability discovery, exploitation, and remediation speed, making static conformity certificates false and requiring continuous agent-operated defense.
-
AXE: Grey-Box Exploitability Confirmation for Localized Vulnerability Reports
Grey-box metadata (CWE + code location) plus a multi-agent LLM workflow raises automated web-exploit confirmation from ~10% to 30% on CVE-Bench, with actionable PoC output.
-
FaultLine: Automated Proof-of-Vulnerability Generation Using LLM Agents
A staged LLM agent that reasons about data flow and branch conditions generated valid proof-of-vulnerability tests for 16 of 100 Java, C, and C++ projects, versus 9 for the CodeAct 2.1 baseline.
-
LLMs unlock new paths to monetizing exploits
LLMs can already mine personal data for blackmail, find real vulnerabilities in obscure software, and automate in-browser and server-side attacks, changing the economics of cybercrime.
-
Evaluating Large Language Models' Capability to Launch Fully Automated Spear Phishing Campaigns: Validated on Human Subjects
Fully automated AI spear phishing achieved a 54% click-through rate on 101 human participants, matching human experts and far exceeding a 12% control rate.
-
SkillSieve: A Hierarchical Triage Framework for Detecting Malicious AI Agent Skills
SkillSieve is a hierarchical triage framework combining regex/AST/XGBoost filtering, parallel LLM subtasks, and multi-LLM jury voting to detect malicious AI agent skills, reaching 0.800 F1 on a 400-skill benchmark at ...
-
Agentic Discovery and Validation of Android App Vulnerabilities
An LLM-agent pipeline discovers Android app vulnerabilities and validates many of them with automatically generated proof-of-concept exploits.
-
Can We End the Cat-and-Mouse Game? Simulating Self-Evolving Phishing Attacks with LLMs and Genetic Algorithms
A closed-loop LLM simulation with genetic algorithms suggests that phishing strategies can evolve to bypass simulated victims' defenses, but the result has not been validated against real humans.
-
Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges
A benchmark and agent for CTF solving, but the agent's retrieval database appears to contain the answers to the test challenges, undermining the reported improvements.
-
Improving LLM Agents with Reinforcement Learning on Cryptographic CTF Challenges
A tool-augmented 8B LLM fine-tuned with GRPO on a new procedurally generated crypto CTF dataset reaches 0.88 Pass@8 on unseen easy tasks, up from 0.10 in the body's tables.
-
Eradicating the Unseen: Detecting, Exploiting, and Remediating a Path Traversal Vulnerability across GitHub
A single vulnerable Node.js path traversal pattern was found in 1,756 GitHub projects, most rated critical, and the authors' automated pipeline produced patches, disclosures, and evidence that LLMs have learned the pattern.
-
Can LLMs Hack Enterprise Networks? Autonomous Assumed Breach Penetration-Testing Active Directory Networks
An autonomous LLM-driven agent can compromise accounts in a realistic Active Directory testbed, with reasoning models outperforming non-reasoning ones at competitive cost.
-
The AI Agent Index
The AI Agent Index catalogs 67 deployed agentic AI systems and shows that most developers publicly disclose little about safety policies and evaluations.
-
Code Change Intention, Development Artifact and History Vulnerability: Putting Them Together for Vulnerability Fix Detection by LLM
LLM4VFD combines LLM-generated summaries of commit intent, issue/PR context, and retrieved historical fixes to detect vulnerability-fixing commits, reporting F1 gains of 68.1% to 145.4% over prior PLM baselines.
-
VulnBot: Autonomous Penetration Testing for A Multi-Agent Collaborative Framework
VulnBot, a three-role LLM agent team with a penetration task graph and summarizer, raises penetration testing completion rates over raw GPT-4o and Llama3.1 on public benchmarks, with one end-to-end real-machine succes...
-
Safety case template for frontier AI: A cyber inability argument
A proof-of-concept safety case template formalizes an inability argument for offensive cyber risk using risk models, proxy tasks, and evaluation results.
-
Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Response
A structured review organizes cyber-capable-agent risks into five vulnerability classes and argues that evaluation environments must be treated as operational security systems rather than background.
-
A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
The paper surveys security risks of LLM agents, organizes them into a five-level autonomy taxonomy, and proposes an untested CMDP-based architecture called R2A2.
-
Seven Security Challenges in Cross-domain Multi-agent LLM Systems
The paper proposes that cross-domain multi-agent LLM systems face seven novel security challenge categories that existing single-domain defenses do not cover.
-
Forewarned is Forearmed: A Survey on Large Language Model-based Agents in Autonomous Cyberattacks
A review of LLM-based agents as autonomous cyberattackers, arguing that they lower attack costs, scale up threats, and outpace existing defenses.
-
AutoPentest: Enhancing Vulnerability Management With Autonomous LLM Agents
AutoPentest, an autonomous GPT-4o multi-agent pentesting tool, completes 15-26% of subtasks on three Hack The Box machines, mildly outperforming manual ChatGPT-4o at API cost of $96.20.
-
LLM Cyber Evaluations Don't Capture Real-World Risk
The paper argues and demonstrates with a 100-prompt case study that LLM cyber risk evaluations need to include threat actor adoption and impact, not just model capability.
-
Position Paper: Model Access should be a Key Concern in AI Governance
Model access decisions should be studied and coordinated through a dedicated research field, with recommendations for evaluators, companies, governments, and international bodies.
-
What AI evaluations for preventing catastrophic risks can and cannot do
AI evaluations can establish lower bounds on capabilities but cannot establish upper bounds, forecast future capabilities robustly, or assess misalignment risk, so they should not be the primary basis for AI safety decisions.
-
On the Surprising Efficacy of LLMs for Penetration-Testing
A critical review arguing that LLMs are surprisingly effective for penetration testing because the task is largely pattern-matching, while noting serious reliability, safety, and cost barriers to autonomous use.
-
From Turing to Tomorrow: The UK's Approach to AI Regulation
The UK should establish a flexible, principles-based regulator for frontier AI development, plus defensive measures against biological risks and updated legal frameworks for copyright, discrimination, and AI agents.
Discussion (0). Continue with ORCID to comment.