REVIEW 26 cited by
PentestGPT: An LLM-empowered Automatic Penetration Testing Tool
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Penetration testing, a crucial industrial practice for ensuring system security, has traditionally resisted automation due to the extensive expertise required by human professionals. Large Language Models (LLMs) have shown significant advancements in various domains, and their emergent abilities suggest their potential to revolutionize industries. In this research, we evaluate the performance of LLMs on real-world penetration testing tasks using a robust benchmark created from test machines with platforms. Our findings reveal that while LLMs demonstrate proficiency in specific sub-tasks within the penetration testing process, such as using testing tools, interpreting outputs, and proposing subsequent actions, they also encounter difficulties maintaining an integrated understanding of the overall testing scenario. In response to these insights, we introduce PentestGPT, an LLM-empowered automatic penetration testing tool that leverages the abundant domain knowledge inherent in LLMs. PentestGPT is meticulously designed with three self-interacting modules, each addressing individual sub-tasks of penetration testing, to mitigate the challenges related to context loss. Our evaluation shows that PentestGPT not only outperforms LLMs with a task-completion increase of 228.6\% compared to the \gptthree model among the benchmark targets but also proves effective in tackling real-world penetration testing challenges. Having been open-sourced on GitHub, PentestGPT has garnered over 4,700 stars and fostered active community engagement, attesting to its value and impact in both the academic and industrial spheres.
Forward citations
Cited by 26 Pith papers
-
Certifying Ghosts: How Cybersecurity AI Agents Break the EU Cyber Resilience Act
Cybersecurity AI agents invalidate the CRA's premises about vulnerability discovery, exploitation, and remediation speed, making static conformity certificates false and requiring continuous agent-operated defense.
-
Geometric quantification for nonlinear deformation in knitted fabrics
A geometric quantification framework reconstructs yarn centerlines and fabric surfaces from sparse knit data and partitions large deformation into stitch reorientation, loop bending, surface bending, and dilation.
-
AXE: Grey-Box Exploitability Confirmation for Localized Vulnerability Reports
Grey-box metadata (CWE + code location) plus a multi-agent LLM workflow raises automated web-exploit confirmation from ~10% to 30% on CVE-Bench, with actionable PoC output.
-
CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks
A 5,352-pair benchmark shows thinking LLMs outperform non-thinking ones as code judges, but position bias and formatting sensitivity remain strong.
-
LLMs unlock new paths to monetizing exploits
LLMs can already mine personal data for blackmail, find real vulnerabilities in obscure software, and automate in-browser and server-side attacks, changing the economics of cybercrime.
-
Evaluating Large Language Models' Capability to Launch Fully Automated Spear Phishing Campaigns: Validated on Human Subjects
Fully automated AI spear phishing achieved a 54% click-through rate on 101 human participants, matching human experts and far exceeding a 12% control rate.
-
ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents
Static-policy judges achieve near-zero recall on scope violations; request-conditioned pre-execution judges reach F1 0.66 (open-weight best) against an expert reference of 0.78 on a 4,897-call labeled benchmark.
-
CTIConnect: A Benchmark for Retrieval-Augmented LLMs over Heterogeneous Cyber Threat Intelligence
A 691-question benchmark (CTIARENA) shows LLMs need retrieval over heterogeneous cyber-threat-intelligence sources and that domain-specific retrieval beats generic RAG; the attached abstract describes a different 1,86...
-
Benchmarking LLM-Assisted Blue Teaming via Standardized Threat Hunting
Standardized modular threat-hunting workflows (CyberTeam) improve LLM performance on blue team tasks compared to open-ended ICL, CoT, and ToT prompting across 30 tasks and 452k samples.
-
iPanda: An LLM-based Agent for Automated Conformance Testing of Communication Protocols
An LLM-based agent that parses protocol RFCs into test cases, synthesizes executable test programs with RAG and chain-of-thought, and iteratively debugs them, reports 4.7x to 10.8x Pass@1 improvements over a pure-LLM ...
-
Improving LLM Agents with Reinforcement Learning on Cryptographic CTF Challenges
A tool-augmented 8B LLM fine-tuned with GRPO on a new procedurally generated crypto CTF dataset reaches 0.88 Pass@8 on unseen easy tasks, up from 0.10 in the body's tables.
-
CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution
CRAKEN, an LLM agent combining Self-RAG and Graph-RAG over a CTF writeup database, solves 22% of NYU CTF Bench challenges, three percentage points above the prior D-CIPHER baseline.
-
Benchmarking LLMs in an Embodied Environment for Blue Team Threat Hunting
CYBERTEAM is a 30-task, 452,293-sample benchmark in which guided function-calling pipelines improve LLM performance on blue team threat hunting relative to ICL, CoT, and ToT prompting.
-
Can LLMs Hack Enterprise Networks? Autonomous Assumed Breach Penetration-Testing Active Directory Networks
An autonomous LLM-driven agent can compromise accounts in a realistic Active Directory testbed, with reasoning models outperforming non-reasoning ones at competitive cost.
-
AEGIS: An Agent-based Framework for General Bug Reproduction from Issue Descriptions
AEGIS combines concise context extraction with finite-state-machine feedback control to make LLM agents reproduce more software bugs from issue descriptions than existing baselines.
-
Tiny Enough to Break In: Agentic Remote Access Trojans Powered by Small Language Models
A locally hosted 8B language model closed an autonomous observe-decide-act attack loop against a vulnerable target but completed only 10.9% of tasks, showing architectural feasibility without operational reliability.
-
Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Response
A structured review organizes cyber-capable-agent risks into five vulnerability classes and argues that evaluation environments must be treated as operational security systems rather than background.
-
Breaking Android with AI: A Deep Dive into LLM-Powered Exploitation
LLM-generated scripts can automate several Android rooting and exploitation tasks in an emulator, but kernel, bootloader, and A/B-partition attacks are beyond current capability.
-
Enabling Cyber Security Education through Digital Twins and Generative AI
A position paper outlining a digital twin plus LLM plus penetration testing toolkit framework for cybersecurity education, with no experimental validation.
-
Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques
A survey that maps LLM applications, vulnerabilities, and defenses across eight cybersecurity domains, but with significant citation and rigor problems.
-
AutoPentest: Enhancing Vulnerability Management With Autonomous LLM Agents
AutoPentest, an autonomous GPT-4o multi-agent pentesting tool, completes 15-26% of subtasks on three Hack The Box machines, mildly outperforming manual ChatGPT-4o at API cost of $96.20.
-
Generative AI for Internet of Things Security: Challenges and Opportunities
A survey that catalogs 33 GenAI-for-IoT-security works through the MITRE ICS mitigations lens, with three small case studies on adapting LLMs to IoT incident response and security question answering.
-
LLM Cyber Evaluations Don't Capture Real-World Risk
The paper argues and demonstrates with a 100-prompt case study that LLM cyber risk evaluations need to include threat actor adoption and impact, not just model capability.
-
HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing
A two-module LLM agent with a planner and summarizer solves 41 of 120 PicoCTF and 32 of 80 OverTheWire challenges using GPT-4o, and the authors release both the agent and the benchmarks.
-
AI-Augmented Ethical Hacking: A Practical Examination of Manual Exploitation and Privilege Escalation in Linux Environments
In a virtual lab, ChatGPT-4o successfully guided manual exploitation and privilege escalation to root access on two Linux machines, though human expertise was needed to steer it.
-
Cybersecurity AI: The Dangerous Gap Between Automation and Autonomy
A roboticist adapts driving-automation levels to cybersecurity, arguing that today's 'autonomous' penetration testers run at Level 3-4 and still require human oversight.
Discussion (0). Continue with ORCID to comment.