Cybersecurity AI agents invalidate the CRA's premises about vulnerability discovery, exploitation, and remediation speed, making static conformity certificates false and requiring continuous agent-operated defense.
hub Canonical reference
Pentestgpt: An llm-empowered automatic penetration testing tool
Canonical reference. 80% of citing Pith papers cite this work as background.
19
Pith papers citing it
30
external citations · Pith
Background
80% of classified citations
abstract
Penetration testing, a crucial industrial practice for ensuring system security, has traditionally resisted automation due to the extensive expertise required by human professionals. Large Language Models (LLMs) have shown significant advancements in various domains, and their emergent abilities suggest their potential to revolutionize industries. In this research, we evaluate the performance of LLMs on real-world penetration testing tasks using a robust benchmark created from test machines with platforms. Our findings reveal that while LLMs demonstrate proficiency in specific sub-tasks within the penetration testing process, such as using testing tools, interpreting outputs, and proposing subsequent actions, they also encounter difficulties maintaining an integrated understanding of the overall testing scenario. In response to these insights, we introduce PentestGPT, an LLM-empowered automatic penetration testing tool that leverages the abundant domain knowledge inherent in LLMs. PentestGPT is meticulously designed with three self-interacting modules, each addressing individual sub-tasks of penetration testing, to mitigate the challenges related to context loss. Our evaluation shows that PentestGPT not only outperforms LLMs with a task-completion increase of 228.6\% compared to the \gptthree model among the benchmark targets but also proves effective in tackling real-world penetration testing challenges. Having been open-sourced on GitHub, PentestGPT has garnered over 4,700 stars and fostered active community engagement, attesting to its value and impact in both the academic and industrial spheres.
hub tools
citation-role summary
background 4
dataset 1
citation-polarity summary
representative citing papers
Honeyval evaluates LLM HTTP honeypots with AI attackers and shows they produce longer interactions, lower detection rates, and cost advantages over rule-based baselines.
CyberEvolver introduces a four-layer self-evolving agent architecture with trace-to-diagnosis and population beam search that raises seed agent success rates by 13.6% on CTF, exploitation, and penetration tasks across four LLMs.
SCAgent automates side-channel leakage discovery via LLM agents for target identification and few-shot foundation models for scalable analysis on iOS.
PHANTOM raises honeytoken believability from 0.576 to 0.778 by adding organization-specific mimicry, lifting human acceptance to 100% and detection resistance to 0.870.
Trace fingerprints AI penetration testing agents from terminal command sequences to identify model families and extracts their system prompts via targeted defensive prompt injection.
Pre-execution scope gating works only when the judge sees the user's request, with GLM-5.2 achieving F1=0.66 against an expert reference of 0.78.
HunterAgent combines LLM hypothesis generation with symbolic verification and cost-bounded graph search to reconstruct attack paths under anti-forensics, reporting 86.1% mean F1 on benchmarks with reduced hallucinations.
Open-world agents suffer from an Authorization-Execution Gap arising from delegation incompleteness, channel corruption, and composition fragmentation, requiring dynamic runtime integrity checks instead of only upfront filters or post-hoc audits.
An agentic pipeline localizes the security-relevant function in 10 of 20 Ubuntu binary security updates and produces an accepted root-cause classification in 11 of 20, limited mainly by binary differencing coverage.
SAGE uses sparse autoencoders to boost vulnerability signals in LLMs, raising internal SNR 12.7x and delivering up to 318% MCC gains on vulnerability detection benchmarks.
PoC-Adapt improves automated PoC exploit generation reliability by 25% and lowers cost using semantic state validation and RL adaptive policies, verifying 12 PoCs from 80 recent CVE attempts at $0.42 each.
A survey of LLM-based autonomous agents that proposes a unified framework for their construction and reviews applications in social science, natural science, and engineering along with evaluation methods and future directions.
The paper provides a lifecycle-based systematization of LLM vulnerabilities across data collection, pretraining, alignment, packaging, retrieval, prompting, tool execution, and deployment, mapping them to security objectives and identifying open problems.
AI-emulated APTs compromise enterprise hosts and weaponize defender tools in 8/10 trials while military ranges resist, indicating TTP attribution fails when agents can be scaffolded to mimic threat actors.
The paper proposes the Cybersecurity AI Scientist as a modular multi-agent architecture for automating cybersecurity research, distinguished by its focus on non-stationary threats and anchored in a four-zeros risk-trust-incident-energy frame.
A benchmarking experiment finds low rediscovery rates for three models on six Mythos-linked bug tasks, with only six target matches across 54 attempts under controlled prompting.
citing papers explorer
-
Certifying Ghosts: How Cybersecurity AI Agents Break the EU Cyber Resilience Act
Cybersecurity AI agents invalidate the CRA's premises about vulnerability discovery, exploitation, and remediation speed, making static conformity certificates false and requiring continuous agent-operated defense.
-
Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots
Honeyval evaluates LLM HTTP honeypots with AI attackers and shows they produce longer interactions, lower detection rates, and cost advantages over rule-based baselines.
-
CyberEvolver: Structured Self-Evolution for Cybersecurity Agents On the Fly
CyberEvolver introduces a four-layer self-evolving agent architecture with trace-to-diagnosis and population beam search that raises seed agent success rates by 13.6% on CTF, exploitation, and penetration tasks across four LLMs.
-
Rethinking Side-Channel Analysis: Automated Discovery and Analysis of Side-Channel Leakage with LLM-Assisted Agents
SCAgent automates side-channel leakage discovery via LLM agents for target identification and few-shot foundation models for scalable analysis on iOS.
-
PHANTOM: Polymorphic Honeytoken Adaptation with Narrative-Tailored Organisational Mimicry
PHANTOM raises honeytoken believability from 0.576 to 0.778 by adding organization-specific mimicry, lifting human acceptance to 100% and detection resistance to 0.870.
-
Trace: Unmasking AI Attack Agents Through Terminal Behavior Fingerprinting
Trace fingerprints AI penetration testing agents from terminal command sequences to identify model families and extracts their system prompts via targeted defensive prompt injection.
-
ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents
Pre-execution scope gating works only when the judge sees the user's request, with GLM-5.2 achieving F1=0.66 against an expert reference of 0.78.
-
HunterAgent: Neuro-Symbolic Attack Trace Reconstruction under Anti-Forensics
HunterAgent combines LLM hypothesis generation with symbolic verification and cost-bounded graph search to reconstruct attack paths under anti-forensics, reporting 86.1% mean F1 on benchmarks with reduced hallucinations.
-
The Authorization-Execution Gap Is a Major Safety and Security Problem in Open-World Agents
Open-world agents suffer from an Authorization-Execution Gap arising from delegation incompleteness, channel corruption, and composition fragmentation, requiring dynamic runtime integrity checks instead of only upfront filters or post-hoc audits.
-
Patch2Vuln: Agentic Reconstruction of Vulnerabilities from Linux Distribution Binary Patches
An agentic pipeline localizes the security-relevant function in 10 of 20 Ubuntu binary security updates and produces an accepted root-cause classification in 11 of 20, limited mainly by binary differencing coverage.
-
SAGE: Signal-Amplified Guided Embeddings for LLM-based Vulnerability Detection
SAGE uses sparse autoencoders to boost vulnerability signals in LLMs, raising internal SNR 12.7x and delivering up to 318% MCC gains on vulnerability detection benchmarks.
-
PoC-Adapt: Semantic-Aware Automated Vulnerability Reproduction with LLM Multi-Agents and Reinforcement Learning-Driven Adaptive Policy
PoC-Adapt improves automated PoC exploit generation reliability by 25% and lowers cost using semantic state validation and RL adaptive policies, verifying 12 PoCs from 80 recent CVE attempts at $0.42 each.
-
A Survey on Large Language Model based Autonomous Agents
A survey of LLM-based autonomous agents that proposes a unified framework for their construction and reviews applications in social science, natural science, and engineering along with evaluation methods and future directions.
-
A Lifecycle and Application-Stack Survey of Large Language Model Vulnerabilities: Attacks, Risks, Defenses, and Open Problems
The paper provides a lifecycle-based systematization of LLM vulnerabilities across data collection, pretraining, alignment, packaging, retrieval, prompting, tool execution, and deployment, mapping them to security objectives and identifying open problems.
-
Synthetic APTs: the Collapse of TTP-Based Attribution
AI-emulated APTs compromise enterprise hosts and weaponize defender tools in 8/10 trials while military ranges resist, indicating TTP attribution fails when agents can be scaffolded to mimic threat actors.
-
Hephaestus: Toward a Cybersecurity AI Scientist
The paper proposes the Cybersecurity AI Scientist as a modular multi-agent architecture for automating cybersecurity research, distinguished by its focus on non-stationary threats and anchored in a four-zeros risk-trust-incident-energy frame.
-
Benchmarking Mythos-Linked Bug Rediscovery
A benchmarking experiment finds low rediscovery rates for three models on six Mythos-linked bug tasks, with only six target matches across 54 attempts under controlled prompting.
- CornerCase: Automated Extremal Testing of Protocol Implementations using LLMs
- Enhancing Linux Privilege Escalation Attack Capabilities of Local LLM Agents