Pith. sign in

REVIEW 4 major objections 5 minor 8 cited by

HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read HackSynth, an LLM agent using only a planner and summarizer, autonomously solves 73 of 200 CTF challenges, with GPT-4o best.

desk verdict The dynamic solver benchmark is the real contribution; the agent numbers are plausible but the unvalidated solver scripts make them provisional. read the letter →

arxiv 2412.01778 v1 pith:VVBYJTCU submitted 2024-12-02 cs.CR cs.AI

Lajos Muzsai , David Imolai , András Lukács This is my paper · ORCID
classification cs.CRcs.AI
keywords LLMagentautonomouspenetrationtestingCTFbenchmarkPicoOverTheWireplannersummarizerarchitectureGPT-4ocybersecurityautomation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces HackSynth, an LLM-based agent with just two modules—a Planner that emits one shell command per step and a Summarizer that condenses command output into a running history—and claims this simple loop is enough to solve a substantial fraction of entry-level penetration-testing tasks. On the authors' new benchmark of 200 capture-the-flag (CTF) challenges drawn from PicoCTF and OverTheWire, the best configuration (GPT-4o) solved 41 of 120 and 32 of 80 challenges respectively, which the authors say exceeds what GPT-4o's system card would predict. The paper also contributes standardized benchmarks with dynamic flag-checking solver scripts, and an analysis of temperature, top-p, observation-window size, token use, and safety failures such as hallucinated target IPs. The reason this matters: if the claim holds, a minimal planning-plus-memory architecture rather than elaborate tool integration is sufficient to automate meaningful pentesting work with current LLMs.

What carries the argument

The load-bearing object is the two-module feedback loop. The Planner prompt forces a single executable command, in a strict tag format, that must make progress using the summarized history; the Summarizer merges each new command output into a running summary, bounded by a new-observation window size. This loop runs in a containerized Kali environment with a firewall whitelist that pre-emptively overrides rules before each command, plus an iteration cap of 20 steps. The benchmarks' machinery is the set of per-challenge solver functions that re-derive flags from live platforms, making ground truth dynamic.

What would settle it

Re-run the GPT-4o configuration on the released PicoCTF benchmark while independently verifying each of the 41 claimed flags against the platform, or audit the 200 solver scripts by hand; finding even a handful of solvers that return wrong flags on current PicoCTF or OverTheWire instances would invalidate the corresponding solve counts.

Watch

Extended reading notes

Core claim

In the paper's own terms, the central discovery is that an autonomous penetration-testing agent does not need complex scaffolding: a Planner that proposes a single terminal command wrapped in <CMD></CMD> tags, paired with a Summarizer that keeps a compressed, evolving summary of past commands and outputs, can complete 41 out of 120 PicoCTF challenges and 32 out of 80 OverTheWire challenges when driven by GPT-4o. The same architecture, run with eight different base LLMs, shows a consistent ranking across categories, and GPT-4o's result is described as better than expected from the GPT-4o system card. The paper further claims that sampling improves performance by 38%, that temperature above 1 degrades both completion and safety, and that moderate observation windows (250 characters for PicoCTF, 500 for OverTheWire) balance context and distraction. The benchmarks are built so flags are fetched dynamically by hand-written solver functions, which the authors argue prevents LLMs from memorizing fixed answers.

Load-bearing premise

The reported solve rates assume every hand-written solver function returns the correct flag for its challenge; if any solver is wrong or the live platform changes in a way the solver does not handle, a 'solved' challenge may be miscounted.

Editorial extensions

If this is right

  • A minimal planner/summarizer loop, not a multi-agent scaffold, is enough for meaningful CTF solving, so future agents can focus on improving memory and search rather than basic autonomy.
  • GPT-4o-class models can serve as a reference point, while smaller local models like Llama-3.1-70B approximate it, suggesting local or fine-tuned agents are within reach.
  • Sampling should be enabled and temperature should stay at or below 1 for both performance and safety, because higher temperatures make agents unusable and risk environment damage.
  • Observation-window size is a real hyperparameter: too little context loses key output and too much distracts the summarizer, with optimal values differing by environment.
  • Autonomous hacking agents need firewall whitelists and containerization because models hallucinate IPs and can destabilize their own environment, and these protections shape benchmark design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test the paper leaves implicit: run the same benchmark with flags intentionally rotated or with solvers hidden, to separate genuine agent reasoning from memorized tool-call patterns.
  • The architecture's simplicity hints that performance gains may come more from the base model's cybersecurity knowledge than from the agent loop; an ablation with a fixed oracle summarizer would test this.
  • Because the benchmarks use live platforms, they will drift over time; a version that snapshots challenge files while keeping dynamic flag retrieval would improve reproducibility.
  • The observed 'rabbit hole' persistence implies that adding a cheap strategy-diversity signal to the summarizer might reduce wasted steps without another LLM call.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces HackSynth, an autonomous penetration-testing agent built from two LLM modules: a Planner that outputs shell commands and a Summarizer that maintains a compressed history of observations. The authors also contribute two CTF benchmarks, one with 120 PicoCTF challenges and one with 80 OverTheWire challenges, each accompanied by a hand-written solver script intended to programmatically retrieve the live flag. Experiments explore temperature, top-p, observation-window size, and sampling, and then compare eight LLM backends on the benchmarks. The headline results are that GPT-4o solves 41/120 PicoCTF and 32/80 OverTheWire challenges, which the authors state is better than expected from the GPT-4o system card. The paper also discusses safety mechanisms and unexpected agent behaviors.

Significance. If the benchmarks are reliable and the reported numbers are reproducible, the paper would provide a useful baseline agent and a dynamic-flag benchmark infrastructure for the community. The public release of the code and benchmarks is a concrete strength, and the idea of using per-challenge solver scripts to handle user- and time-varying flags is valuable. However, the current evidence does not yet support the quantitative claims: the solver scripts that define ground truth are unvalidated, the model comparison rests on a single stochastic run per configuration, and the hyperparameters are tuned on the same benchmarks used for the final scores. These issues directly affect the headline solve counts and the model ranking, so the central claims are not yet established to the standard expected for a published benchmark-plus-agent paper.

major comments (4)
  1. [§3.2 and Appendix Listings 9–10] The benchmark's ground truth for all 200 challenges is a set of hand-written solver functions, yet the paper shows only two examples and reports no validation that every solver retrieves the correct current flag from PicoCTF or OverTheWire. Because §3.2.1 states that flags vary per user and can change over time, a stale or account-specific solver would both mislabel the agent's successful flag retrievals and corrupt the benchmark as a reusable evaluation artifact. Please provide a validation harness that runs each solver against the live platforms and report per-solver success rates, or otherwise document how each solver was verified against the current platform state.
  2. [§4.2, Table 1 and Figure 8] All headline solve counts, including GPT-4o's 41/120 and 32/80, appear to come from a single run per model at temperature 1. LLM command generation is stochastic, so without repeated seeds or confidence intervals the model ranking (for example, GPT-4o at 34.2% versus Llama-3.1-70B at 22.5% on PicoCTF) cannot be separated from run-to-run variation. Please report repeated trials with at least three to five seeds, giving per-model means and variances, or provide a statistical test for the differences that are claimed.
  3. [§4.1 and §4.2] The observation-window size (250 for PicoCTF, 500 for OverTheWire), temperature 1, and top-p 0.9 were selected using the same two benchmarks on which the final model comparison is reported, and the tuning was performed with Llama-3.1-8B and Phi-3-mini before being applied unchanged to all eight models. This introduces selection bias: the final numbers are not an unbiased estimate of performance under a fixed configuration, and the transferability of these parameters to larger and proprietary models is an untested assumption. Please add a validation split or a sensitivity analysis showing that the ranking is robust to the chosen parameter values.
  4. [§4.2, OverTheWire paragraph] The statement that GPT-4o's performance is 'better than expected based on the GPT-4o system card' is not substantiated. The cited system card [59] is primarily a safety and capability document and, as far as the manuscript reports, does not contain a comparable CTF solve-rate protocol; no concrete system-card number is quoted for comparison. Please either provide the exact system-card metric and matching evaluation conditions, or remove the claim from the abstract and body.
minor comments (5)
  1. [§1, Contributions] The contribution list includes 'human evaluation of HackSynth's hacking process,' but no human evaluation study or results appear in Sections 4 or 5; please add this analysis or remove the bullet.
  2. [§4.1] The claims that 'using sampling increases performance by 38%' and 'prompt-chaining decreased performance by 5%' are not accompanied by a definition of the baseline, the number of runs, or the benchmark subset used; these numbers need context to be interpretable.
  3. [Table 1] Table 1 reports only percentages, while the text reports raw counts (41/120, 32/80); please include raw counts alongside percentages in the table or systematically in the text to avoid confusion.
  4. [Figure 7] The term 'rare commands' is defined only in the caption (as outside the top 10 most frequently used commands); please define it in the main text where the figure is discussed.
  5. [§3.1.4 and §5.2] The safety description in §3.1.4 says firewall rules are overridden before each command, while §5.2 says a whitelist firewall restricts network interactions; please reconcile these details so the actual containment mechanism is clear.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the evaluation is empirical and grounded in external live flags; solver scripts are a measurement tool, not an input that forces the reported outcomes.

full rationale

The central claim—that HackSynth with GPT-4o solves 41/120 PicoCTF and 32/80 OverTheWire challenges—is an empirical measurement against flags obtained from external platforms, not a quantity derived from the paper's own definitions. The solver functions in Section 3.2 are a benchmark-instrumentation choice: they are intended to retrieve the current live flag, and the reported success counts are not defined as 'matches the authors' solver script' in a way that would make the result true by construction. If some solver scripts are stale or incorrect, the counts could be mislabeled, but that is a validity and reproducibility risk, not circular reasoning. Similarly, the hyperparameter search (observation window, temperature, top-p) is performed on the same benchmarks before the final runs, but the final results are presented as evaluations of a fixed configuration, not as predictions validated against held-out data; this is overfitting or selection-bias territory, not circularity. The paper contains no self-citations that bear the load of the argument, and no known external result is renamed or imported from the authors' prior work. The comparison with the GPT-4o system card is an external benchmark reference. Therefore no step in the paper reduces by construction to its own inputs, and the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The benchmark and agent performance rest on several unverified premises: the 200 solver scripts are correct, the live CTF flags remain compatible with those scripts, and hyperparameters chosen on two small LLMs transfer to all eight models. None of these are proven in the paper. The main empirical claims also depend on the hand-picked challenge set being representative, and on the container plus firewall containment being adequate, which the paper itself notes can be bypassed by routing through the target host.

free parameters (4)
  • temperature = 1.0
    Chosen from the Figure 4 sweep on Llama-3.1-8B and Phi-3-mini; applied to all eight models in Table 1.
  • top-p = 0.9
    Chosen from the Figure 6 sweep; higher values slightly improved performance and were applied to all models.
  • new observation window size = 250 (PicoCTF), 500 (OverTheWire)
    Selected from the Figure 3 sweep; applied to all models without per-model tuning.
  • maximum steps = 20
    Predefined iteration limit for benchmark runs; not optimized in the paper.
assumptions (4)
  • domain assumption Hand-written solver functions correctly and stably recover flags for all 200 challenges.
    Section 3.2 states that solver functions dynamically return flags; if any solver is wrong, the completion labels are wrong.
  • domain assumption The selected challenges do not require personalized instances and are solvable through non-interactive bash.
    Section 3.2.1 excludes instanced challenges; the non-interactive constraint shapes the Planner prompt in Appendix A.
  • ad hoc to paper Hyperparameters tuned on Llama-3.1-8B and Phi-3-mini transfer to larger and proprietary models.
    Section 4.1 optimization runs use two small models; Section 4.2 applies the same temperature, top-p, and window sizes to all models without per-model validation.
  • domain assumption Containerization and the firewall whitelist are sufficient to keep agent actions within the intended target.
    Section 3.1.4 describes containment, but also notes the agent could route attacks through the target machine or reset firewall rules, so the assumption is acknowledged as imperfect.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing." pith.science (2026). https://pith.science/paper/VVBYJTCU

@misc{pith2026241201778,
  author       = {Pith},
  title        = {Pith review of: HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VVBYJTCU}},
  note         = {Machine review of arXiv:2412.01778}
}
read the original abstract

We introduce HackSynth, a novel Large Language Model (LLM)-based agent capable of autonomous penetration testing. HackSynth's dual-module architecture includes a Planner and a Summarizer, which enable it to generate commands and process feedback iteratively. To benchmark HackSynth, we propose two new Capture The Flag (CTF)-based benchmark sets utilizing the popular platforms PicoCTF and OverTheWire. These benchmarks include two hundred challenges across diverse domains and difficulties, providing a standardized framework for evaluating LLM-based penetration testing agents. Based on these benchmarks, extensive experiments are presented, analyzing the core parameters of HackSynth, including creativity (temperature and top-p) and token utilization. Multiple open source and proprietary LLMs were used to measure the agent's capabilities. The experiments show that the agent performed best with the GPT-4o model, better than what the GPT-4o's system card suggests. We also discuss the safety and predictability of HackSynth's actions. Our findings indicate the potential of LLM-based agents in advancing autonomous penetration testing and the importance of robust safeguards. HackSynth and the benchmarks are publicly available to foster research on autonomous cybersecurity solutions.

Figures

Figures reproduced from arXiv: 2412.01778 by the authors.

Figure 1
Figure 1. High level overview of the architecture of HackSynth. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Distribution of benchmark challenges across cat [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Effect of new observation window size on com [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Impact of the Temperature Parameter on Error [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 4
Figure 4. Figure 4: Impact of the Temperature Parameter on Challenge [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 6
Figure 6. Figure 6: Effect of the top-p parameter on the number of [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Impact of the top-p Parameter on Rare Command [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Models demonstrate varying levels of benefit from [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Overall token usage for HackSynth when utilizing [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. StealthBench: Measuring Operational Stealth in Autonomous Offensive-Security Agents

    cs.CR 2026-07 conditional novelty 6.0 of 10

    StealthBench's LLM-judge panel finds no AI agent solves offensive-security tasks stealthily more than 54% of the time.

  2. From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    A practical evaluation protocol for AI pentesting agents that uses validated vulnerability discovery, LLM semantic matching, and bipartite scoring to assess performance in realistic, complex targets.

  3. Benchmarking LLM-Assisted Blue Teaming via Standardized Threat Hunting

    cs.CR 2025-09 conditional novelty 6.0 of 10

    Standardized modular threat-hunting workflows (CyberTeam) improve LLM performance on blue team tasks compared to open-ended ICL, CoT, and ToT prompting across 30 tasks and 452k samples.

  4. Improving LLM Agents with Reinforcement Learning on Cryptographic CTF Challenges

    cs.CR 2025-06 conditional novelty 6.0 of 10

    A tool-augmented 8B LLM fine-tuned with GRPO on a new procedurally generated crypto CTF dataset reaches 0.88 Pass@8 on unseen easy tasks, up from 0.10 in the body's tables.

  5. Can LLMs Hack Enterprise Networks? Autonomous Assumed Breach Penetration-Testing Active Directory Networks

    cs.CR 2025-02 conditional novelty 6.0 of 10

    An autonomous LLM-driven agent can compromise accounts in a realistic Active Directory testbed, with reasoning models outperforming non-reasoning ones at competitive cost.

  6. VulnBot: Autonomous Penetration Testing for A Multi-Agent Collaborative Framework

    cs.SE 2025-01 conditional novelty 6.0 of 10

    VulnBot, a three-role LLM agent team with a penetration task graph and summarizer, raises penetration testing completion rates over raw GPT-4o and Llama3.1 on public benchmarks, with one end-to-end real-machine succes...

  7. A Survey of LLM-Driven Penetration Testing: Taxonomy, Co-Evolution, and Open Challenges

    cs.SE 2026-07 accept novelty 5.5 of 10

    LLM pentest agents co-evolved through four bottleneck-driven phases into RLVR systems, while CTF platforms became dual evaluation/training infrastructure and three linked reliability gaps remain.

  8. Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Response

    cs.AI 2026-07 conditional novelty 5.0 of 10

    A structured review organizes cyber-capable-agent risks into five vulnerability classes and argues that evaluation environments must be treated as operational security systems rather than background.

Reference graph

Works this paper leans on

62 extracted references · 33 canonical work pages · cited by 8 Pith papers

  1. [59]

    Qwen2 technical report,

    A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang et al. , “Qwen2 technical report,” arXiv preprint arXiv:2407.10671, 2024

    arXiv 2024

  2. [1]

    (2024, February) 2024 Global Threat Report

    CrowdStrike. (2024, February) 2024 Global Threat Report. Accessed: November 1, 2024. [Online]. Available: https://www.crowdstrike.co m/global-threat-report/

    work page 2024

  3. [2]

    (2024) Nessus

    Tenable, Inc. (2024) Nessus. Accessed: November 1, 2024. [Online]. Available: https://www.tenable.com/products/nessus

    work page 2024

  4. [3]

    (2024) Snyk

    Snyk Limited. (2024) Snyk. Accessed: November 1, 2024. [Online]. Available: https://snyk.io/

    work page 2024

  5. [4]

    (2024) OpenVAS

    Greenbone Networks. (2024) OpenVAS. Accessed: November 1,

    work page 2024

  6. [5]

    Large language models: A survey,

    S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Ama- triain, and J. Gao, “Large language models: A survey,” arXiv preprint arXiv:2402.06196, 2024

    arXiv 2024

  7. [6]

    A survey on large language model (LLM) security and privacy: The good, the bad, and the ugly,

    Y . Yao, J. Duan, K. Xu, Y . Cai, Z. Sun, and Y . Zhang, “A survey on large language model (LLM) security and privacy: The good, the bad, and the ugly,” High-Confidence Computing, p. 100211, 2024

    2024

  8. [7]

    DARPA AIxCC

    DARPA, “DARPA AIxCC.”

    work page

Show all 62 references
  1. [8]

    PentestGPT: An ŁŁM-empowered automatic penetration testing tool,

    G. Deng, Y . Liu, V . Mayoral-Vilches, P. Liu, Y . Li, Y . Xu, T. Zhang, Y . Liu, M. Pinzger, and S. Rass, “PentestGPT: An ŁŁM-empowered automatic penetration testing tool,” arXiv preprint arXiv:2308.06782, 2023

    2023 arXiv

  2. [9]

    Getting pwn’d by AI: Penetration testing with large language models,

    A. Happe and J. Cito, “Getting pwn’d by AI: Penetration testing with large language models,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ser. ESEC/FSE ’23. ACM, Nov. 2023. [Online]. Av...

    2023

  3. [10]

    AutoAttacker: A large language model guided system to implement automatic cyber-attacks,

    J. Xu, J. W. Stokes, G. McDonald, X. Bai, D. Marshall, S. Wang, A. Swaminathan, and Z. Li, “AutoAttacker: A large language model guided system to implement automatic cyber-attacks,” 2024. [Online]. Available: https://arxiv.org/abs/2403.01038

    2024 arXiv

  4. [11]

    EniGMA: Enhanced interactive generative model agent for CTF challenges,

    T. Abramovich, M. Udeshi, M. Shao, K. Lieret, H. Xi, K. Milner, S. Jancheska, J. Yang, C. E. Jimenez, F. Khorrami et al., “EniGMA: Enhanced interactive generative model agent for CTF challenges,” arXiv preprint arXiv:2409.16165 , 2024

    2024 arXiv

  5. [12]

    Using capture-the-flag to enhance the effectiveness of cybersecurity education,

    K. Leune and S. J. Petrilli Jr, “Using capture-the-flag to enhance the effectiveness of cybersecurity education,” in Proceedings of the 18th annual conference on information technology education , 2017, pp. 47–52

    2017

  6. [13]

    (2024) PicoCTF

    Carnegie Mellon University. (2024) PicoCTF. Accessed: November 1, 2024. [Online]. Available: https://picoctf.org/

    2024

  7. [14]

    (2024) OverThewire wargames

    OverTheWire. (2024) OverThewire wargames. Accessed: November 1, 2024. [Online]. Available: https://overthewire.org/wargames/

    2024

  8. [15]

    Intercode: Stan- dardizing and benchmarking interactive coding with execution feed- back,

    J. Yang, A. Prabhakar, K. Narasimhan, and S. Yao, “Intercode: Stan- dardizing and benchmarking interactive coding with execution feed- back,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  9. [16]

    Hack The Box - Hacking Training Platform,

    “Hack The Box - Hacking Training Platform,” https://www.hacktheb ox.com, accessed: 2024-10-02

    2024

  10. [17]

    TryHackMe - Cyber Security Training Platform,

    “TryHackMe - Cyber Security Training Platform,” https://www.tryh ackme.com, accessed: 2024-10-02

    2024

  11. [18]

    Root Me - Hacking and Cybersecurity Challenges,

    “Root Me - Hacking and Cybersecurity Challenges,” https://www.ro ot-me.org, accessed: 2024-10-02

    2024

  12. [19]

    DEF CON Hacking Conference,

    “DEF CON Hacking Conference,” https://www.defcon.org, accessed: 2024-10-02

    2024

  13. [20]

    Cybercompetitions: A survey of com- petitions, tools, and systems to support cybersecurity education,

    T. Balon and I. Baggili, “Cybercompetitions: A survey of com- petitions, tools, and systems to support cybersecurity education,” Education and Information Technologies , vol. 28, no. 9, pp. 11 759– 11 791, 2023

    2023

  14. [21]

    CTFtime - Capture the Flag Competition Tracker,

    “CTFtime - Capture the Flag Competition Tracker,” https://ctftime. org, accessed: 2024-10-02

    2024

  15. [22]

    Katana - Automatic CTF Challenge Solver in Python3,

    “Katana - Automatic CTF Challenge Solver in Python3,” https://gith ub.com/JohnHammond/katana, accessed: 2024-11-01

    2024

  16. [23]

    Remenissions - An autopwner for simple CTF PWN challenges,

    “Remenissions - An autopwner for simple CTF PWN challenges,” https://github.com/guyinatuxedo/remenissions, accessed: 2024-11-01

    2024

  17. [24]

    Large language models in cybersecurity: State-of-the- art,

    F. N. Motlagh, M. Hajizadeh, M. Majd, P. Najafi, F. Cheng, and C. Meinel, “Large language models in cybersecurity: State-of-the- art,” arXiv preprint arXiv:2402.00891 , 2024

    2024 arXiv

  18. [25]

    Lost at c: A user study on the security implications of large language model code assistants,

    G. Sandoval, H. Pearce, T. Nys, R. Karri, S. Garg, and B. Dolan- Gavitt, “Lost at c: A user study on the security implications of large language model code assistants,” in 32nd USENIX Security Symposium (USENIX Security 23) , 2023, pp. 2205–2222

    2023

  19. [26]

    How well does LLM generate security tests?

    Y . Zhang, W. Song, Z. Ji, N. Meng et al. , “How well does LLM generate security tests?” arXiv preprint arXiv:2310.00710 , 2023

    2023

  20. [27]

    Can large language models find and fix vulnerable software?

    D. Noever, “Can large language models find and fix vulnerable software?” arXiv preprint arXiv:2308.10345 , 2023

    2023 arXiv

  21. [28]

    LLM-Assisted Malware Review: AI and Humans Join Forces to Combat Malware,

    Endor Labs, “LLM-Assisted Malware Review: AI and Humans Join Forces to Combat Malware,” 2023, accessed: 2024-11-13. [Online]. Available: https://www.endorlabs.com/learn/llm-assisted-malware-r eview-ai-and-humans-join-forces-to-combat-malware

    2023

  22. [29]

    Impact of code language models on automated program repair,

    N. Jiang, K. Liu, T. Lutellier, and L. Tan, “Impact of code language models on automated program repair,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 1430–1442

    2023

  23. [30]

    Yaman, Agent SCA: Advanced Physical Side Channel Analysis Agent with LLMs

    F. Yaman, Agent SCA: Advanced Physical Side Channel Analysis Agent with LLMs . North Carolina State University, 2023

    2023

  24. [31]

    An attacker’s dream? exploring the capabilities of ChatGPT for developing malware,

    Y . M. Pa Pa, S. Tanizaki, T. Kou, M. Van Eeten, K. Yoshioka, and T. Matsumoto, “An attacker’s dream? exploring the capabilities of ChatGPT for developing malware,” in Proceedings of the 16th Cyber Security Experimentation and Test Workshop , 2023, pp. 10–18

    2023

  25. [32]

    Devising and detecting phishing: Large language models vs. smaller human models,

    F. Heiding, B. Schneier, A. Vishwanath, J. Bernstein, and P. S. Park, “Devising and detecting phishing: Large language models vs. smaller human models,” arXiv preprint arXiv:2308.12287 , 2023

    2023 arXiv

  26. [33]

    Fake News in Sheep’s Clothing: Robust Fake News Detection Against LLM-Empowered Style Attacks,

    J. Wu, J. Guo, and B. Hooi, “Fake News in Sheep’s Clothing: Robust Fake News Detection Against LLM-Empowered Style Attacks,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2024, pp. 3367–3378

    2024

  27. [34]

    Decoding the Threat Landscape : ChatGPT, FraudGPT, and WormGPT in Social Engineering Attacks,

    P. V . Falade, “Decoding the Threat Landscape : ChatGPT, FraudGPT, and WormGPT in Social Engineering Attacks,” arXiv preprint arXiv:2310.05595, 2023

    2023 arXiv

  28. [35]

    The rise and potential of large language model based agents: A survey,

    Z. Xi, W. Chen, X. Guo, W. He, Y . Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou et al. , “The rise and potential of large language model based agents: A survey,” arXiv preprint arXiv:2309.07864, 2023

    2023 arXiv

  29. [36]

    Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security,

    Y . Li, H. Wen, W. Wang, X. Li, Y . Yuan, G. Liu, J. Liu, W. Xu, X. Wang, Y . Sun, R. Kong, Y . Wang, H. Geng, J. Luan, X. Jin, Z. Ye, G. Xiong, F. Zhang, X. Li, M. Xu, Z. Li, P. Li, Y . Liu, Y .-Q. Zhang, and Y . Liu, “Personal LLM Agents: Insights and Survey about the Capabi...

    2024 arXiv

  30. [37]

    (MLAgentbench: Evaluating language agents on machine learning experimentation

    “(MLAgentbench: Evaluating language agents on machine learning experimentation.”

  31. [38]

    Generative agents: Interactive simulacra of human behavior,

    J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Generative agents: Interactive simulacra of human behavior,” in Proceedings of the 36th annual acm symposium on user interface software and technology , 2023, pp. 1–22

    2023

  32. [39]

    OpenHands: An open platform for ai software developers as generalist agents,

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y . Shao, N. Muennighoff, Y . Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig, “OpenHands: An open platform for ai software d...

    2024 arXiv

  33. [40]

    Auto Dev,

    U. Mesh, “Auto Dev,” 2024, accessed: 2024-10-31. [Online]. Available: https://github.com/unit-mesh/auto-dev

    2024

  34. [41]

    Research, “Devon,” 2024, accessed: 2024-10-31

    E. Research, “Devon,” 2024, accessed: 2024-10-31. [Online]. Available: https://github.com/entropy-research/Devon

    2024

  35. [42]

    Plandex,

    P. AI, “Plandex,” 2024, accessed: 2024-10-31. [Online]. Available: https://github.com/plandex-ai/plandex

    2024

  36. [43]

    [Online]

    StitionAI, “Devika,” 2024, accessed: 2024-10-31. [Online]. Available: https://github.com/stitionai/devika

    2024

  37. [44]

    Swe-agent: Agent-computer interfaces enable auto- mated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable auto- mated software engineering,” arXiv preprint arXiv:2405.15793, 2024

    2024 arXiv

  38. [45]

    MetaGPT: Meta programming for a multi-agent collaborative framework,

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber, “MetaGPT: Meta programming for a multi-agent collaborative framework,” in The Twelfth International Conference on Learning Re...

    2024

  39. [46]

    crewAI Inc., “crewAI: Cutting-edge framework for orchestrating role- playing, autonomous AI agents. By fostering collaborative intelli- gence, crewAI empowers agents to work together seamlessly, tackling complex tasks.” https://github.com/crewAIInc/crewAI, 2024, accessed: 2024-11-12

    2024

  40. [47]

    LLM agents can autonomously hack websites,

    R. Fang, R. Bindu, A. Gupta, Q. Zhan, and D. Kang, “LLM agents can autonomously hack websites,” arXiv preprint arXiv:2402.06664 , 2024

    2024 arXiv

  41. [48]

    Cybench: A frame- work for evaluating cybersecurity capabilities and risk of language models,

    A. K. Zhang, N. Perry, R. Dulepet, E. Jones, J. W. Lin, J. Ji, C. Menders, G. Hussein, S. Liu, D. Jasper et al., “Cybench: A frame- work for evaluating cybersecurity capabilities and risk of language models,” arXiv preprint arXiv:2408.08926 , 2024

    2024 arXiv

  42. [49]

    NYU CTF Dataset: A scalable open-source benchmark dataset for evaluating llms in offensive security,

    M. Shao, S. Jancheska, M. Udeshi, B. Dolan-Gavitt, H. Xi, K. Milner, B. Chen, M. Yin, S. Garg, P. Krishnamurthy et al. , “NYU CTF Dataset: A scalable open-source benchmark dataset for evaluating llms in offensive security,” arXiv preprint arXiv:2406.05590 , 2024

    2024 arXiv

  43. [50]

    The Llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al., “The Llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  44. [51]

    Phi-3 technical report: A highly capable language model locally on your phone,

    M. Abdin, S. A. Jacobs, A. A. Awan, J. Aneja, A. Awadallah, H. Awadalla, N. Bach, A. Bahree, A. Bakhtiari, H. Behl et al., “Phi-3 technical report: A highly capable language model locally on your phone,” arXiv preprint arXiv:2404.14219 , 2024

    2024 arXiv

  45. [52]

    Probing the creativity of large language models: Can models produce divergent semantic association?

    H. Chen and N. Ding, “Probing the creativity of large language models: Can models produce divergent semantic association?” arXiv preprint arXiv:2310.11158, 2023

    2023 arXiv

  46. [53]

    Is Temperature the Creativity Parameter of Large Language Models?

    M. Peeperkorn, T. Kouwenhoven, D. Brown, and A. Jordanous, “Is Temperature the Creativity Parameter of Large Language Models?”

  47. [54]

    The curious case of neural text degeneration,

    A. Holtzman, J. Buys, L. Du, M. Forbes, and Y . Choi, “The curious case of neural text degeneration,” arXiv preprint arXiv:1904.09751 , 2019

    1904 arXiv

  48. [55]

    Available: https://arxiv.org/abs/2405.00492

    [Online]. Available: https://arxiv.org/abs/2405.00492

    arXiv

  49. [56]

    GPT-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “GPT-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  50. [57]

    A Survey of Large Language Models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong, Y . Du, C. Yang, Y . Chen, Z. Chen, J. Jiang, R. Ren, Y . Li, X. Tang, Z. Liu, P. Liu, J.-Y . Nie, and J.-R. Wen, “A Survey of Large Language Models,” 2024. [Online]. Available: https:...

    2024 arXiv

  51. [58]

    Mixtral of experts,

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand et al., “Mixtral of experts,” arXiv preprint arXiv:2401.04088 , 2024

    2024 arXiv

  52. [60]

    Retrieval- augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschel et al. , “Retrieval- augmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems , vol. 33, pp. 9459–9474, 202...

    2020

  53. [61]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radfordet al., “Gpt-4o system card,” arXiv preprint arXiv:2410.21276 , 2024

    2024 arXiv

  54. [2024]

    Available: https://www.openvas.org/

    [Online]. Available: https://www.openvas.org/

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.