Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

Can LLMs Hack Enterprise Networks? Autonomous Assumed Breach Penetration-Testing Active Directory Networks

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Pith's one-line read No human in the loop: LLM agents autonomously compromise accounts inside a live Microsoft Active Directory network, with reasoning models succeeding several times more often at costs competitive with human testers.

desk verdict An honest, well-documented feasibility study that credibly demonstrates the first autonomous LLM compromise of an Active Directory testbed, but the evaluation is too small and too GOAD-flavored to support the broader cost and generalization claims. read the letter →

arxiv 2502.04227 v3 pith:X6S4UDRT submitted 2025-02-06 cs.CR

classification cs.CR
keywords LLMagentspenetrationtestingActiveDirectoryassumedbreachsimulationreasoningLLMsautonomouscybersecurityGOADtestbedagentplanning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a fully autonomous, LLM-driven system can perform assumed-breach penetration testing against a realistic Microsoft Active Directory network: the prototype cochise, given command-line access to a Kali machine inside the target lab, plans attacks, executes commands, reads output, and compromises user accounts with no human in the loop. The authors report the first such autonomous account compromises in the GOAD testbed, which they chose deliberately because its live, multi-domain, defender-equipped setup is closer to real enterprise conditions than synthetic benchmarks. Their central comparison is economic as well as technical: the strongest configuration compromised about 1.8 accounts per two-hour run at roughly $23, while a human engagement at quoted industry rates would cost thousands of dollars. The paper also documents clear failure modes, including one small open-weight model that never compromised an account because its planner could not integrate executor findings into the plan, and argues the underlying planner/executor design transfers to other software-engineering domains.

What carries the argument

The carrying mechanism is the two-level agent architecture. The Planner maintains a Pentest-Task-Tree (PTT), a hierarchical text plan that accumulates findings, credentials, and leads, updating it after every executor report and selecting the next task with all relevant context attached. The Executor, built as a ReAct loop, issues shell commands on a Kali virtual machine inside the target network, reads the results, and self-corrects until the task is done or ten rounds elapse; it deliberately keeps no memory, so every fact the system knows must be written into the PTT. This division of labor is what lets the agent span long multi-step attack chains while staying grounded in real tool output, and it is also the documented source of the system's main failures: hashes and credentials are sometimes redacted or replaced with placeholders when handed from Planner to Executor, and the planner can fixate on a single avenue, 'going down a rabbit hole.'

What would settle it

Build a private Active Directory lab whose domains, users, and password choices were invented after the models' training cutoff, run the same prototype for the same two-hour budget, and compare accounts compromised per dollar; if the compromise rate collapses or the cost per account climbs past a human tester's, the GOAD-based feasibility and cost claims are lab artifacts. A cheaper probe is to ask each model directly for GOAD usernames and passwords, since memorization would surface immediately.

Watch

Extended reading notes

Core claim

The paper's central claim is that off-the-shelf LLMs can autonomously run the consolidation phase of an enterprise intrusion: enumerating the Active Directory forest, harvesting and cracking Kerberos AS-REP hashes, generating scenario-specific password candidates, reading network shares for embedded credentials, and chaining these steps into verified account compromises inside a live lab. On GOAD, the reasoning configurations (o1 as planner with GPT-4o as executor, and Gemini-2.5-Flash in both roles) compromised an average of 0.83 to 1.83 accounts per two-hour run and logged roughly twice as many leads as the non-reasoning models, while Qwen3, a small open-weight reasoning model, compromised none because its planner loop kept re-issuing the same enumeration tasks. The claimed cost advantage is stark: the most expensive configuration ran at $11.64 per hour, about $17.56 per compromised account, against the $180-per-hour rate and roughly $10,000 projected cost of a human assumed-breach engagement.

Load-bearing premise

That GOAD, a public deliberately vulnerable training lab, is representative enough of real enterprise Active Directory networks, and that the agents' success does not come from having memorized GOAD's users and passwords during training.

Editorial extensions

If this is right

  • Organizations that cannot afford human assumed-breach testing could run continuous, repeatable LLM-driven assessments: the paper's cheapest working configuration cost about $0.10–$2.42 per hour.
  • Reasoning models are the effective direction for autonomous security agents: the o1+GPT-4o and Gemini-2.5-Flash configurations compromised 2.5 to 5.5 times more accounts than GPT-4o or DeepSeek-V3 and produced double the leads.
  • LLM attackers behave unlike any scanner: they install missing tools, switch from network attacks to web-application audits, craft phishing emails, and read unstructured files for credential hints, so defenders must treat unstructured data and social engineering as surfaces an automated attacker will use.
  • The planner/executor pattern with a task tree as the single state store is a viable template for long-horizon LLM agents beyond security; the paper's own failure modes point to concrete fixes, such as a circuit breaker for rabbit holes and a lossless shared fact store at the handover.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The results are measured against a public vulnerable lab that may be present in the models' training data; the paper's contamination check looked only for non-causal shortcut flows, so a private, freshly built AD lab with post-cutoff passwords is the test that would confirm whether the compromise rates transfer to unseen networks.
  • The inter-context credential finds (passwords embedded in narrative text or PowerShell scripts) suggest the automated-attacker threat model has changed qualitatively; defenders can exploit this by seeding honey-token credentials in shares, which a scanner would ignore but an LLM agent would likely pick up and act on.
  • The Planner-to-Executor handover failures (truncated hashes, placeholder tokens) generalize to any LLM agent that must act over long horizons: external lossless memory is likely cheaper than re-discovering a fact, a conclusion the paper's own token-cost data supports even though the authors do not draw it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces cochise, a prototype that combines a high-level Planner (maintaining a Pentest-Task-Tree) with a low-level Executor (a ReAct-style command loop) to perform autonomous Assumed Breach penetration testing against a Microsoft Active Directory testbed. The authors evaluate five LLM configurations — GPT-4o, DeepSeek-V3, Qwen3, Gemini-2.5-Flash, and an o1+GPT-4o combination — on the publicly available GOADv3 lab, with six two-hour runs per configuration. They report quantitative metrics (planner/executor rounds, commands, token costs, compromised accounts) and qualitative expert analyses, claiming that reasoning LLMs compromise substantially more accounts, that inter-context attacks such as web audits, social engineering, and unstructured data analysis occur, and that costs are competitive with human penetration testers. The paper also documents limitations including rabbit-holing, Planner/Executor information-transfer failures, and safety concerns requiring human oversight.

Significance. If the central claims hold, this is a noteworthy contribution to software engineering and security automation: it would be among the first demonstrations of an LLM-driven agent chaining multiple Active Directory attacks in a realistic, multi-host environment, with open-source code, logs, and commit-pinned artifacts that support replication. The expert triangulation and explicit threats-to-validity discussion are strengths, as is the honest documentation of failure modes. However, the significance is contingent on external validity: the evaluation uses one public, widely used vulnerable testbed, and the quantitative evidence base is thin (six runs per model, no confidence intervals or significance tests). The cost and generalization claims therefore remain suggestive rather than established.

major comments (4)
  1. [§3.2.2, §3.2.4, Appendix A.1] The dataset-contamination check is too weak to support the paper's central generalization claim to 'real-world AD enterprise networks.' The check in §3.2.2 only searches for non-causal attack flows (shortcuts that skip expected stages), but a model that memorized GOAD walkthroughs would produce a perfectly causal-looking sequence — enumerate, AS-REP roast, crack, pivot — because that is exactly the documented GOAD path. The scenario prompt itself leaks GOAD cues: Appendix A.1 states 'the target environment and all users were setup in 2022, the current year is 2022,' which matches GOAD's known context and invites passwords like 'Winter2022'; §3.2.4 says the OSINT user list was inspired by a GOAD walkthrough; and the Game-of-Thrones usernames (j.snow, d.targaryen) are recognizable from public GOAD documentation. A model with GOAD-specific knowledge would pass the paper's test while still reflecting memorization rather than general enterprise-hacking capability. This directly affects the abstract's claim that autonomous LLMs can effectively conduct Assumed Breach simulations in 'real-world AD enterprise networks.' A concrete remedy is to evaluate on a private or modified GOAD variant with renamed users, domains, and passwords (including changed setup year), or on a second independently themed AD lab, and to show that performance is not driven by GOAD-specific cues.
  2. [§5, Tables 3–7] The headline comparative claims rest on very small samples without statistical support. Each configuration was run only six times, and Tables 3–7 report no confidence intervals, effect sizes, or significance tests. The variance is high: GPT-4o's average of 0.33 compromised accounts per run is driven by a single run with two compromises; o1+GPT-4o averages 1.83, while Gemini-2.5-Flash averages 0.83, and the per-run ranges overlap considerably. The statement in §5.3.1 and the Conclusion that 'reasoning LLMs compromised substantially more accounts' is therefore not established by the data as presented. The paper should either report appropriate uncertainty quantification (e.g., bootstrap confidence intervals or Bayesian intervals) and justify the comparison, or explicitly frame the quantitative differences as exploratory and in need of larger-sample confirmation.
  3. [§5.5, §6.5, §7.1] The cost comparison omits the required human oversight and the fixed setup costs, which undermines the claim that LLM-driven testing is 'competitive with professional penetration-testers.' The paper itself states that safety necessitates human oversight (§6.5, §7.1) and that expert penetration testers were needed to analyze traces and classify outcomes (§3.5.2). The reported $11.64/hour for the o1+GPT-4o configuration excludes: the human monitor's time during the two-hour runs, the time spent preparing the Kali VM, crafting the OSINT user list and scenario prompt, and any report-generation or verification work. A fair comparison should include a total-cost-of-ownership model that accounts for these human-in-the-loop components, or should explicitly scope the claim to 'LLM API and hosting costs' rather than end-to-end testing cost.
  4. [§2.5, §6.3.1] The claim that the approach is 'more dynamic than traditional security scanner[s]' and 'surpass[es] traditional security tooling' is not supported because no non-LLM automated baseline was evaluated on the same testbed. GOAD is a deliberately vulnerable lab with publicly documented attack paths; a scripted or rule-based attacker (e.g., a fixed sequence of nmap, GetNPUsers, hashcat, and nxc) might achieve comparable account compromises on GOAD. Without such a baseline, it is impossible to attribute the observed successes to LLM reasoning rather than to the testbed's intrinsic easiness. Adding a Caldera adversary profile or a simple shell-scripted attack chain as a baseline would strengthen the central feasibility claim substantially.
minor comments (6)
  1. [§3.2.2] The criterion 'non-causal attack flows' is not operationalized; the paper should specify how the experts defined and detected such flows and provide at least one example of a flow that was checked and rejected.
  2. [§5.1.2] The subsection heading '5.1.2 Attack Vector Coverage' is repeated within the same section, and the text refers to 'Section 5.1.2' in a way that is ambiguous because the subsection itself is numbered 5.1.2; consider renumbering the later subsection or removing the self-reference.
  3. [Figure 2 and Appendix A.5] There are several typos and incomplete sentences: 'differnet' in Figure 2, 'informaiton' in Appendix A.1 line 18, 'Plese provide a containing findings' in Appendix A.5, and 'erogenous calls' in §5.6.1 should be 'erroneous calls.'
  4. [§5.5.2, Figure 10(b)] The axis label 'T otal T oken Count of Query' has inconsistent spacing/capitalization; the phrase 'over evaluated LLMs' in the Figure 10 caption should be 'the evaluated LLMs.'
  5. [§6.2.3] The observation that GPT-4o missed the plain-text password of samwell.tarly in the summary but o1 recovered it from the full history is interesting, but the paper should clarify whether this occurred in one run or multiple runs, since the current phrasing makes the prevalence of the problem hard to assess.
  6. [§3.6] The threats-to-validity section mentions 'opaque behavior of used LLMs' but does not mention the risk that the scenario prompt itself provides GOAD-specific information; adding that as an explicit threat would improve transparency.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the empirical evaluation against an external testbed is self-contained; the GOAD-contamination and OSINT-scaffolding concerns are validity threats, not definitional loops.

full rationale

The paper's central claims are empirical observations about LLM-driven account compromise in a live third-party testbed (GOAD), plus token-cost arithmetic. No model parameter is fitted to a subset of outcomes and then renamed a prediction; no equation makes the output equal to the input by construction. The prototype builds on the authors' own prior hackingBuddyGPT executor loop and PTT planning, but those citations are architectural lineage, not load-bearing proofs of the present results: the GOAD runs are independently executed and logged, and the measured compromises are contingent outcomes, not consequences of the self-citations. The strongest circularity-adjacent issues are external-validity threats, not circularity: Section 3.2.2's contamination check only searches for 'non-causal attack flows', so causal-looking memorized GOAD paths would pass, and the scenario prompt leaks GOAD-specific cues (Section A.1 tells the model 'the current year is 2022'; Section 3.2.4 says the OSINT user list was inspired by a GOAD walkthrough). These could inflate the apparent 'enterprise' capability and weaken the 'real-life enterprise network' generalization, but they do not make the observed account compromises definitionally equivalent to the prompt inputs. The cost comparison is straightforward multiplication of logged tokens by public API prices and does not reduce to a fitted value. Overall, the derivation chain is not circular; the paper honestly discusses environmental representativeness and contamination as limitations.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on a small number of hand-chosen experimental inputs: the OSINT user list, the scenario prompt with domain and tool hints, the 10-minute command timeout, the 10-round executor cap, and the saturation-derived sample count of 6 runs per configuration. The main domain assumptions are that GOAD generalizes to real enterprise AD networks, that the LLMs do not have GOAD-specific training data, and that the three experts' coding is reliable. No new physical or theoretical entities are introduced.

free parameters (5)
  • OSINT user list (/root/osint_users.txt)
    Hand-provided list of likely usernames, inspired by an older GOAD walkthrough, given to the VM before runs. It directly enables AS-REP roasting and password spraying, so it is a researcher-chosen input that boosts success rates.
  • Scenario prompt constraints
    Includes 'current year is 2022', target IP range, exclusion of management hosts, netexec instead of crackmapexec, impacket naming, and prohibition of OpenVAS. These hand-authored instructions shape model behavior and success.
  • Command timeout = 10 minutes
    Chosen to align with 5-minute periodic GOAD background activities; affects whether sniffing tasks capture data.
  • Executor round limit = 10 rounds
    Chosen by the authors; affects task completion and costs.
  • Sample count per configuration = 6 runs
    Saturation-based; o1+GPT-4o needed 6 runs, and the maximum was applied to all configs. Post-hoc determination of sample size.
assumptions (4)
  • domain assumption GOAD is a realistic and representative stand-in for real enterprise AD networks.
    Section 3.2.3 argues synthetic testbeds are inadequate and GOAD captures real-world complexity. If GOAD does not generalize to production AD environments, the central feasibility claim is weakened.
  • domain assumption The evaluated LLMs have no GOAD-specific knowledge in their training data.
    Section 3.2.2 searches for non-causal attack flows as a check. This is a weak test; GOAD is public and may appear in training corpora, so this assumption carries risk.
  • domain assumption Human expert coding of traces is reliable.
    Section 3.5.2 relies on three experts' thematic analysis and categorization of almost-theres, leads, and MITRE techniques; consensus discussions mitigate but do not eliminate subjectivity.
  • domain assumption LLM provider APIs behave consistently and pricing and token counts are accurate.
    Section 3.5.1 uses provider-reported token usage for cost calculations. Any reporting errors would propagate into the cost claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can LLMs Hack Enterprise Networks? Autonomous Assumed Breach Penetration-Testing Active Directory Networks." pith.science (2026). https://pith.science/paper/X6S4UDRT

@misc{pith2026250204227,
  author       = {Pith},
  title        = {Pith review of: Can LLMs Hack Enterprise Networks? Autonomous Assumed Breach Penetration-Testing Active Directory Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X6S4UDRT}},
  note         = {Machine review of arXiv:2502.04227}
}
read the original abstract

Enterprise penetration-testing is often limited by high operational costs and the scarcity of human expertise. This paper investigates the feasibility and effectiveness of using Large Language Model (LLM)-driven autonomous systems to address these challenges in real-world Active Directory (AD) enterprise networks. We introduce a novel prototype designed to employ LLMs to autonomously perform Assumed Breach penetration-testing against enterprise networks. Our system represents the first demonstration of a fully autonomous, LLM-driven framework capable of compromising accounts within a real-life Microsoft Active Directory testbed, GOAD. We perform our empirical evaluation using five LLMs, comparing reasoning to non-reasoning models as well as including open-weight models. Through quantitative and qualitative analysis, incorporating insights from cybersecurity experts, we demonstrate that autonomous LLMs can effectively conduct Assumed Breach simulations. Key findings highlight their ability to dynamically adapt attack strategies, perform inter-context attacks (e.g., web-app audits, social engineering, and unstructured data analysis for credentials), and generate scenario-specific attack parameters like realistic password candidates. The prototype exhibits robust self-correction mechanisms, installing missing tools and rectifying invalid command generations. We find that the associated costs are competitive with, and often significantly lower than, those incurred by professional human pen-testers, suggesting a path toward democratizing access to essential security testing for organizations with budgetary constraints. However, our research also illuminates existing limitations, including instances of LLM ``going down rabbit holes'', challenges in comprehensive information transfer between planning and execution modules, and critical safety concerns that necessitate human oversight.

Figures

Figures reproduced from arXiv: 2502.04227 by the authors.

Figure 1
Figure 1. Our prototype combines two Active Directory attacks (AS-REP Kerberos Roasting, following up by password-cracking) to [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. System-Diagram of our Experiment Environment. Our Prototype ( [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Simplified System-Diagram of the used “A Game of Active Directory” (GOAD) Testbed highlighting attack paths and [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: High-Level Architecture Diagram of our Prototype [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]
Figure 5
Figure 5. Figure 5: Example of an initially generated Planner State (Pentest-Task-Tree) by OpenAI’s o1-GPT-4o. As the LLM does only possess limited information about the target environment, the generated PTT only consists of initial network enumeration tasks. The prototype has a Executor …
Figure 6
Figure 6. Figure 6: Excerpt of the Planner state (PTT) generated by OpenAI’s o1-GPT-4o after 10 update-strategy rounds (see Section B.2 for full listing). The Planner was able to integrate findings, e.g., the retrieved and cracked hash of missandei, within the PTT. Please note, that origi…
Figure 7
Figure 7. Figure 7: Example Task and Context generated by the [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Example Task and Context generated by the [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: Attack Vectors pursued by the different LLMs. For each attack vector we detail the percentage of runs in which the respective [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]
Figure 10
Figure 10. Figure 10: Our sampling runs were time-capped at two hours, making the efficacy of time spent by the LLMs of high importance. The [PITH_FULL_IMAGE:figures/full_fig_p029_10.png]
Figure 11
Figure 11. Figure 11: The Pentest-Task-Tree (PTT) includes all current knowledge of the prototype about the target environment. Its size impacts [PITH_FULL_IMAGE:figures/full_fig_p030_11.png]
Figure 12
Figure 12. Figure 12: The Executor is given a task by the Planner and has up to 10 rounds to successfully finish this task. During each round, it can select new command line tool invocations to execute and analyzes the gathered result. Each round has all messages of previous rounds prefixe…
Figure 13
Figure 13. Figure 13: Inclusion of Tools within OpenAI’s o1+GPT-4o experiment runs. Please note, that this is an exact count. A tool that is [PITH_FULL_IMAGE:figures/full_fig_p031_13.png]
Figure 14
Figure 14. Figure 14: Phishing Email suggested by GPT-4o as part of a Social Engineering attack. No mail servers were configured within our [PITH_FULL_IMAGE:figures/full_fig_p038_14.png]
Figure 15
Figure 15. Figure 15: Message that from arya.stark to jon.snow containing the password Needle that can be found on a publicly accessible SMB network file share within the testbed. LLMs performing Web-Application Audits against discovered Web Applications. LLMs, esp. GPT-4o, tried to perfor…
Figure 16
Figure 16. Figure 16: Content of PowerShell script script.ps1 containing credentials. It is stored on one of the testbed’s domain controllers within SYSVOL and accessible by all AD users, representative for typical insecure configuration scripts often deployed by system administrators. 1 #…
Figure 17
Figure 17. Figure 17: Content of PowerShell script secret.ps1 containing credentials. It is stored on one of the testbed’s domain controllers within SYSVOL and accessible by all AD users. An attacker shoudl be able to reverse-engineer the encryption and retrieve the plain-text secret. from…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Ethics of Autonomous AI Agents for Offensive Security

    cs.CR 2026-07 conditional novelty 6.0 of 10

    Autonomous AI hacking tools combine three kinds of indeterminacy—action, impact, and users—making moral responsibility diffuse and giving attackers a short-term advantage under current cost asymmetries.

  2. Recognition Without Mitigation: Ethical Frameworks in Autonomous Offensive-LLM Agent Research

    cs.CR 2025-06 reject novelty 6.0 of 10

    In offensive-LLM agent papers, dual-use risk is acknowledged in 39% of papers but concrete mitigations appear in only 7%.

  3. On the Surprising Efficacy of LLMs for Penetration-Testing

    cs.CR 2025-07 conditional novelty 3.0 of 10

    A critical review arguing that LLMs are surprisingly effective for penetration testing because the task is largely pattern-matching, while noting serious reliability, safety, and cost barriers to autonomous use.

Reference graph

Works this paper leans on

83 extracted references · 40 canonical work pages · cited by 3 Pith papers

  1. [1]

    Abdulrahman Alamri and Lexie Mooney. 2025. Dragos Industrial Ransomware Analysis: Q1 2025. https://www.dragos.com/blog/dragos-industrial- ransomware-analysis-q1-2025/. Accessed: 2025-06-02

  2. [2]

    Ron Alford, Dean Lawrence, and Michael Kouremetis. 2022. Caldera: A red-blue cyber operations automation platform.MITRE: Bedford, MA, USA (2022)

  3. [3]

    Afnan Binduf, Hanan Othman Alamoudi, Hanan Balahmar, Shatha Alshamrani, Haifa Al-Omar, and Naya Nagy. 2018. Active Directory and Related Aspects of Security. In2018 21st Saudi Computer Society National Computer Conference (NCC). 4474–4479. doi:10.1109/NCG.2018.8593188

  4. [4]

    Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology.Qualitative research in psychology3, 2 (2006), 77–101

  5. [5]

    2006.Constructing grounded theory: A practical guide through qualitative analysis

    Kathy Charmaz. 2006.Constructing grounded theory: A practical guide through qualitative analysis. Sage

  6. [6]

    dair ai. 2025. Reasoning LLMs Guide. https://www.promptingguide.ai/guides/reasoning-llms. Accessed: 2025-06-11

  7. [7]

    Gelei Deng, Yi Liu, Víctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. 2024. PentestGPT: An LLM-empowered Automatic Penetration Testing Tool. arXiv:2308.06782 [cs.SE] https://arxiv.org/abs/2308.06782

  8. [8]

    2017.Sociological methods: A sourcebook

    Norman K Denzin. 2017.Sociological methods: A sourcebook. routledge. Manuscript submitted to ACM 46 Andreas Happe and Jürgen Cito

Show all 83 references
  1. [9]

    Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. 2024. LLM Agents can Autonomously Exploit One-day Vulnerabilities. arXiv:2404.08144 [cs.CR] https://arxiv.org/abs/2404.08144

  2. [10]

    Richard Fang, Rohan Bindu, Akul Gupta, Qiusi Zhan, and Daniel Kang. 2024. LLM Agents can Autonomously Hack Websites. arXiv:2402.06664 [cs.CR] https://arxiv.org/abs/2402.06664

  3. [11]

    Richard Fang, Rohan Bindu, Akul Gupta, Qiusi Zhan, and Daniel Kang. 2024. Teams of LLM Agents can Exploit Zero-Day Vulnerabilities. arXiv:2406.01637 [cs.MA] https://arxiv.org/abs/2406.01637

  4. [12]

    Luca Gioacchini, Marco Mellia, Idilio Drago, Alexander Delsanto, Giuseppe Siracusano, and Roberto Bifulco. 2024. AutoPenBench: Benchmarking Generative Agents for Penetration Testing. arXiv:2410.03225 [cs.CR] https://arxiv.org/abs/2410.03225

  5. [13]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948(2025)

  6. [14]

    Andreas Happe and Jürgen Cito. 2023. Getting pwn’d by AI: Penetration Testing with Large Language Models. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’23). ACM, 2082–2086. doi:1...

  7. [15]

    Andreas Happe and Jürgen Cito. 2023. Understanding Hackers’ Work: An Empirical Study of Offensive Security Practitioners. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’23). ACM, ...

  8. [16]

    Andreas Happe and Jürgen Cito. 2025. Benchmarking Practices in LLM-driven Offensive Security: Testbeds, Metrics, and Experiment Design. arXiv:2504.10112 [cs.CR] https://arxiv.org/abs/2504.10112

  9. [17]

    Andreas Happe and Jürgen Cito. 2025. On the Ethics of Using LLMs for Offensive Security. arXiv:2506.08693 [cs.CR] https://arxiv.org/abs/2506.08693

  10. [18]

    Andreas Happe, Aaron Kaplan, and Juergen Cito. 2024. Llms as hackers: Autonomous linux privilege escalation attacks.arXiv preprint arXiv:2310.11409 (2024)

  11. [20]

    Fred Heiding, Simon Lermen, Andrew Kao, Bruce Schneier, and Arun Vishwanath. 2024. Evaluating Large Language Models’ Capability to Launch Fully Automated Spear Phishing Campaigns: Validated on Human Subjects. arXiv:2412.00586 [cs.CR] https://arxiv.org/abs/2412.00586

  12. [21]

    Monique Hennink and Bonnie N Kaiser. 2022. Sample sizes for saturation in qualitative research: A systematic review of empirical tests.Social science & medicine292 (2022), 114523

  13. [22]

    Junjie Huang and Quanyan Zhu. 2023. Penheal: a two-stage llm framework for automated pentesting and optimal remediation. InProceedings of the Workshop on Autonomous Cybersecurity. 11–22

  14. [23]

    Isamu Isozaki, Manil Shrestha, Rick Console, and Edward Kim. 2024. Towards automated penetration testing: Introducing llm benchmark, analysis, and improvements.arXiv preprint arXiv:2410.17141(2024)

  15. [24]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card.arXiv preprint arXiv:2412.16720(2024)

  16. [25]

    Samar Kamil, Huda Sheikh Abdullah Siti Norul, Ahmad Firdaus, and Opeyemi Lateef Usman. 2022. The Rise of Ransomware: A Review of Attacks, Detection Techniques, and Future Challenges. In2022 International Conference on Business Analytics for Technology and Security (ICBATS). 1–...

  17. [26]

    Ilker Kara and Murat Aydos. 2022. The rise of ransomware: Forensic analysis for windows based ransomware attacks.Expert Systems with Applications190 (2022), 116198. doi:10.1016/j.eswa.2021.116198

  18. [27]

    Harpreet Kaur, Dharani Sanjaiy SL, Tirtharaj Paul, Rohit Kumar Thakur, K Vijay Kumar Reddy, Jay Mahato, and Kaviti Naveen. 2024. Evolution of endpoint detection and response (edr) in cyber security: A comprehensive review. InE3S Web of Conferences, Vol. 556. EDP Sciences, 01006

  19. [28]

    2006.Mastering Active directory for Windows server 2003

    Robert R King. 2006.Mastering Active directory for Windows server 2003. John Wiley & Sons

  20. [29]

    Hoaglin, Khaled El Emam, and Jarrett Rosenberg

    Barbara A Kitchenham, Shari Lawrence Pfleeger, Lesley M Pickard, Peter W Jones, David C. Hoaglin, Khaled El Emam, and Jarrett Rosenberg. 2002. Preliminary guidelines for empirical research in software engineering.IEEE Transactions on software engineering28, 8 (2002), 721–734

  21. [30]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems35 (2022), 22199–22213

  22. [31]

    He Kong, Die Hu, Jingguo Ge, Liangxiong Li, Tong Li, and Bingzhen Wu. 2025. VulnBot: Autonomous Penetration Testing for A Multi-Agent Collaborative Framework.arXiv preprint arXiv:2501.13411(2025)

  23. [32]

    Swetha Krishnamoorthi and Jarad Carleton. 2020. Active Directory Holds the Keys to your Kingdom, but is it Secure? https://www.frost.com/growth- opportunity-news/active-directory-holds-the-keys-to-your-kingdom-but-is-it-secure. Accessed: 2025-06-02

  24. [33]

    Xiaomin Li, Zhou Yu, Zhiwei Zhang, Xupeng Chen, Ziji Zhang, Yingying Zhuang, Narayanan Sadagopan, and Anurag Beniwal. 2025. When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs. arXiv:2505.11423 [cs.CL] https://arxiv.org/abs/2505.11423

  25. [34]

    Kamil˙e Lukoši¯ut˙e and Adam Swanda. 2025. LLM Cyber Evaluations Don’t Capture Real-World Risk. arXiv:2502.00072 [cs.CR] https://arxiv.org/abs/ 2502.00072

  26. [35]

    Steve Morgan. 2025. Global Ransomware Damage Costs Predicted To Exceed $275 Billion By 2031. https://cybersecurityventures.com/global- ransomware-damage-costs-predicted-to-reach-250-billion-usd-by-2031/. Accessed: 2025-06-02. Manuscript submitted to ACM Can LLMs Hack Enterpris...

  27. [36]

    Lajos Muzsai, David Imolai, and András Lukács. 2024. HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing. arXiv:2412.01778 [cs.CR] https://arxiv.org/abs/2412.01778

  28. [37]

    Nitin Naik, Paul Jenkins, Paul Grace, and Jingping Song. 2022. Comparing attack models for it systems: Lockheed martin’s cyber kill chain, mitre att&ck framework and diamond model. In2022 IEEE International Symposium on Systems Engineering (ISSE). IEEE, 1–7

  29. [38]

    Sho Nakatani. 2025. RapidPen: Fully Automated IP-to-Shell Penetration Testing with LLM-based Agents. arXiv:2502.16730 [cs.CR] https: //arxiv.org/abs/2502.16730

  30. [39]

    part of the National Cyber Security Centre (NCSC) New Zealand’s CERT (Computer Emergency Response Team). 2023. How ransomware happens and how to stop it. https://www.cert.govt.nz/information-and-advice/guides/how-ransomware-happens-and-how-to-stop-it/. Accessed: 2025-06-02

  31. [40]

    Jakob Nielsen and Rolf Molich. 1990. Heuristic evaluation of user interfaces. InProceedings of the SIGCHI conference on Human factors in computing systems. 249–256

  32. [41]

    OpenAI. 2024. Introducing OpenAI o1-preview. https://openai.com/index/introducing-openai-o1-preview/. Accessed: 2025-02-5

  33. [42]

    OpenAI. 2024. Learning to reason with LLMs. https://openai.com/index/learning-to-reason-with-llms/. Accessed: 2025-06-06

  34. [43]

    boomer prompts

    OpenAI. 2025. As some of you have noticed, avoid “boomer prompts” with o-series models. Instead, be simple and direct, with specific guidelines. https://x.com/OpenAIDevs/status/1890147300493914437. Accessed: 2025-06-11

  35. [44]

    OpenAI. 2025. Reasoning best practices. https://platform.openai.com/docs/guides/reasoning-best-practices. Accessed: 2025-06-10

  36. [45]

    Giulio De Pasquale, Ilya Grishchenko, Riccardo Iesari, Gabriel Pizarro, Lorenzo Cavallaro, Christopher Kruegel, and Giovanni Vigna. 2024. ChainReactor: Automated Privilege Escalation Chain Discovery via AI Planning. In33rd USENIX Security Symposium (USENIX Security 24). USENIX...

  37. [46]

    Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislav Balunović, Nikola Jovanović, and Martin Vechev

  38. [47]

    Boomer Prompts. 2025. BoomerPrompts. https://boomerprompts.com/. Accessed: 2025-06-11

  39. [48]

    Pat Rondon, Renyao Wei, José Cambronero, Jürgen Cito, Aaron Sun, Siddhant Sanyam, Michele Tufano, and Satish Chandra. 2025. Evaluating Agent-based Program Repair at Google.arXiv preprint arXiv:2501.07531(2025)

  40. [49]

    Shanto Roy, Emmanouil Panaousis, Cameron Noakes, Aron Laszka, Sakshyam Panda, and George Loukas. 2023. SoK: The MITRE ATT&CK Framework in Research and Practice. arXiv:2304.07411 [cs.CR] https://arxiv.org/abs/2304.07411

  41. [50]

    Carlos Sarraute, Olivier Buffet, and Jörg Hoffmann. 2012. POMDPs make better hackers: Accounting for uncertainty in penetration testing. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 26. 1816–1824

  42. [51]

    Carlos Sarraute, Olivier Buffet, and Jörg Hoffmann. 2013. Penetration testing== POMDP solving?arXiv preprint arXiv:1306.4714(2013)

  43. [52]

    Minghao Shao, Boyuan Chen, Sofija Jancheska, Brendan Dolan-Gavitt, Siddharth Garg, Ramesh Karri, and Muhammad Shafique. 2024. An Empirical Evaluation of LLMs for Solving Offensive Security Challenges. arXiv:2402.11814 [cs.CR] https://arxiv.org/abs/2402.11814

  44. [53]

    Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, and Muhammad Shafique. 2024. NYU CTF Dataset: A Scalable Open-Source Benchmark Dataset ...

  45. [54]

    Venu Shastri. 2022. Attackers Set Sights on Active Directory: Understanding Your Identity Exposure. https://www.crowdstrike.com/en-us/blog/ attackers-set-sights-on-active-directory-understanding-your-identity-exposure/. Accessed: 2025-06-02

  46. [55]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems36 (2024)

  47. [56]

    Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. 2025. The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity. arXiv:2506.06941 [cs.AI] https://arxiv.org/...

  48. [57]

    Brian Singer, Keane Lucas, Lakshmi Adiga, Meghna Jain, Lujo Bauer, and Vyas Sekar. 2025. On the Feasibility of Using LLMs to Execute Multistage Network Attacks.arXiv preprint arXiv:2501.16466(2025)

  49. [58]

    Robin Sommer and Vern Paxson. 2010. Outside the closed world: On using machine learning for network intrusion detection. In2010 IEEE symposium on security and privacy. IEEE, 305–316

  50. [59]

    V Stafford. 2020. Zero trust architecture.NIST special publication800, 207 (2020), 800–207

  51. [60]

    Jai Vijayan. 2025. 25 Years On, Active Directory Is Still a Prime Attack Target. https://www.darkreading.com/identity-access-management- security/25-years-active-directory-prime-attack-target. Accessed: 2025-06-02

  52. [61]

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models.arXiv preprint arXiv:2305.04091(2023)

  53. [62]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems35 (2022), 24824–24837

  54. [63]

    Benlong Wu, Guoqiang Chen, Kejiang Chen, Xiuwei Shang, Jiapeng Han, Yanru He, Weiming Zhang, and Nenghai Yu. 2024. AutoPT: How Far Are We from the End2End Automated Web Penetration Testing? arXiv:2411.01236 [cs.CR] https://arxiv.org/abs/2411.01236

  55. [64]

    Siwei Wu, Zhongyuan Peng, Xinrun Du, Tuney Zheng, Minghao Liu, Jialong Wu, Jiachen Ma, Yizhi Li, Jian Yang, Wangchunshu Zhou, Qunshu Lin, Junbo Zhao, Zhaoxiang Zhang, Wenhao Huang, Ge Zhang, Chenghua Lin, and J. H. Liu. 2024. A Comparative Study on Reasoning Patterns of OpenAI...

  56. [65]

    Jiacen Xu, Jack W Stokes, Geoff McDonald, Xuesong Bai, David Marshall, Siyue Wang, Adith Swaminathan, and Zhou Li. 2024. Autoattacker: A large language model guided system to implement automatic cyber-attacks.arXiv preprint arXiv:2403.01038(2024)

  57. [66]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025)

  58. [67]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering.arXiv preprint arXiv:2405.15793(2024)

  59. [68]

    Liping Yang, Lidong QI, and Bo Zhang. 2022. Concepts and evaluation of saturation in qualitative research.Advances in Psychological Science30, 3 (2022), 511

  60. [69]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629(2022)

  61. [70]

    Andy K Zhang, Neil Perry, Riya Dulepet, Joey Ji, Justin W Lin, Eliot Jones, Celeste Menders, Gashon Hussein, Samantha Liu, Donovan Jasper, et al

  62. [71]

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompting in large language models.arXiv preprint arXiv:2210.03493(2022)

  63. [72]

    Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, et al . 2024. Evaluation of openai o1: Opportunities and challenges of agi.arXiv preprint arXiv:2409.18486(2024). A Used Prompts A.1 Scenario Description...

  64. [78]

    Identify which hosts are Windows systems and potential domain controllers 3

    Perform basic network enumeration on 192.168.56.0/24 , excluding 192.168.56.1 and 192.168.56.107 21.1. Identify which hosts are Windows systems and potential domain controllers 3

  65. [79]

    Enumerate domain accounts using netexec , focusing on discovered Windows hosts that appear to be domain controllers or file servers B.2 State/Pentest-Task-Tree after 10 Rounds

  66. [80]

    Identify which hosts are Windows systems and potential domain controllers 31.1.1

    Perform basic network enumeration on 192.168.56.0/24 , excluding 192.168.56.1 and 192.168.56.107 21.1. Identify which hosts are Windows systems and potential domain controllers 31.1.1. Findings : Hosts 192.168.56.10 , 192.168.56.11 , and 192.168.56.12 show Kerberos (88) and LD...

  67. [81]

    Attempt domain user enumeration on 192.168.56.10 , 192.168.56.11 , and 192.168.56.12 82.1.1

    Enumerate domain accounts using netexec , focusing on discovered Windows hosts that appear to be domain controllers or file servers 72.1. Attempt domain user enumeration on 192.168.56.10 , 192.168.56.11 , and 192.168.56.12 82.1.1. Findings : 9- 192.168.56.10 ( KINGSLANDING ) d...

  68. [82]

    fr3edom

    Perform offline password cracking on discovered Kerberos hash 263.1. Use a cracking tool ( john or hashcat ) with / usr / share / wordlists / rockyou . txt to attempt cracking missandei@ESSOS . LOCAL's hash . 273.2. If successful , validate the credentials with netexec to conf...

  69. [111]

    12 131.1

    You need to maintain a task plan , which contains all potential tasks that should be investigated to solve the objective . 12 131.1. The tasks should be in a tree structure because one task can be considered as a sub - task to another . Manuscript submitted to ACM 50 Andreas H...

  70. [162]

    Initially , create an minimal plan based upon the provided information . 172.1. The plan should contain the inital 2 -3 tasks that could be delegated to the worker . 182.2. You will evolve the plan over time based upon the workers'feedback . 192.3. Don't over - engineer the in...

  71. [253]

    Analyze the results and identify information that might be relevant for solving your objective through future steps

    Each time you receive results from the worker you should 26 273.1. Analyze the results and identify information that might be relevant for solving your objective through future steps . 283.2. Add new tasks or update existing task information according to the findings . 293.2.1...

  72. [414]

    sevenkingdoms

    Evaluate potential password - spraying approach for newly discovered domain accounts on north . sevenkingdoms . local 424.1. Use caution to avoid lockouts and consider a small custom password list for minimal attempts . 434.2. If any credentials are found , attempt domain auth...

  73. [455]

    essos . local

    Use valid credentials essos . local \ missandei : fr3edom to pivot within essos . local 465.1. Enumerate domain users , groups , and privileges ( e . g . , netexec or Impacket - based tools ) 475.1.1. Findings : 48- Domain users discovered : Administrator , daenerys . targarye...

  74. [2024]

    Cybench: A framework for evaluating cybersecurity capabilities and risks of language models.arXiv preprint arXiv:2408.08926(2024)

  75. [2025]

    arXiv:2503.21934 [cs.CL] https://arxiv.org/abs/2503.21934

    Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad. arXiv:2503.21934 [cs.CL] https://arxiv.org/abs/2503.21934

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.