Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Good News for Script Kiddies? Evaluating Large Language Models for Automated Exploit Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read No evaluated LLM succeeded in exploiting refactored vulnerable programs.

desk verdict A useful first systematic study on LLM-driven exploit generation with a plausible but not yet airtight central negative result; the paper deserves peer review but needs a more rigorous evaluation protocol. read the letter →

arxiv 2505.01065 v1 pith:YXQTCTDL submitted 2025-05-02 cs.CR cs.AI

David Jin , Qian Fu , Yuekang Li This is my paper · ORCID
classification cs.CRcs.AI
keywords automatedexploitgenerationlargelanguagemodelsSEEDLabsLLMsecurityexploitabilityassessmentjailbreakcoderefactoringbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether current large language models can automate exploit generation well enough to be a practical threat. It builds a benchmark from five software-security labs, each provided in its original form and in a refactored form where variable and function names are replaced with meaningless tokens to defeat memorized answers. An automated attacker pipeline iteratively prompts each target model up to fifteen times. The central negative result is that no model produced a working exploit for any refactored lab, while GPT-4o came closest with only six blocking errors across all five. The finding matters because it suggests today's models are not yet a reliable tool for script-kiddie-level exploitation, but the gap to success is small.

What carries the argument

The central mechanism is the LLM-based attacker, an automated script in which one LLM (here GPT-4o) acts as a prompter that reads the target model's latest exploit attempt and issues a refined instruction, cycling for up to fifteen iterations or until it decides no more improvement is possible. The other load-bearing component is the refactored benchmark: the original vulnerable programs have every variable and function renamed to varX and functionX, so a model cannot succeed by recalling a known SEED Lab solution. The evaluation metric is a manual count of errors in the model's final code that would prevent the exploit from running, compared against the authors' own reference solutions.

What would settle it

Run the five refactored labs repeatedly with automated verification that actually executes each generated payload and checks for a shell or code execution. If any model's exploit succeeds on any run, the paper's central negative claim is false; conversely, many repeated failures across different random seeds would strengthen it.

Watch

Extended reading notes

Core claim

The authors report the first systematic evaluation of LLMs for automated exploit generation (AEG). Across five refactored SEED Labs programs covering buffer overflow, return-to-libc, format string, race condition, and dirty COW attacks, none of the five tested models (GPT-4o, GPT-4o-mini, Llama3, Dolphin-Mistral, Dolphin-Phi) produced an exploit that worked. The metric was the number of mistakes in each model's final code that would prevent the exploit from functioning; GPT-4o made the fewest with six mistakes total, followed by GPT-4o-mini with eight, Dolphin-Mistral with twelve, Llama3 with fifteen, and Dolphin-Phi with seventeen. The authors also find that GPT-4 and GPT-4o cooperate with exploit requests almost as readily as uncensored open-source models, while Llama3 resists most often. They interpret the performance drop between original and refactored labs as evidence that some models may have memorized SEED Lab solutions from training data.

Load-bearing premise

The headline result rests on a single manual pass: one run of the iterative attacker per lab, with the errors in each model's final output counted by hand, so a lucky run or a miscounted mistake could change whether the 'none succeeded' claim holds.

Editorial extensions

If this is right

  • Current LLMs are not yet capable of fully automated, general exploit generation: none of the tested models solved any refactored lab.
  • GPT-4o's error counts (one or two mistakes per attempt) suggest that a small improvement in reasoning or debugging could push LLM-driven AEG over the threshold to success.
  • The cooperation data indicate that alignment does not strongly protect GPT-series models against exploit-request prompting, so safety cannot rely on refusal behavior alone.
  • Because performance drops sharply when code symbols are obfuscated, some models likely succeed on public benchmarks by memorization rather than by understanding the vulnerability.
  • The released benchmark and attacker pipeline give future work a reproducible starting point for measuring progress in LLM-based AEG.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The single-run, manually counted protocol is likely high-variance; if the authors were to sample each model over several runs and average error counts, the ordering could shift even if the all-fail result stands.
  • Reasoning-focused models released after this study (such as GPT-o1 and DeepSeek-r1) might succeed on the refactored labs because the failures appear to be about forming connections between program structure and payload mechanics, which inference-time reasoning directly targets.
  • The refactoring only renames symbols; a future benchmark that changes control flow or adds unrelated code would be a stronger test of genuine exploitation ability, and the current result may overstate or understate the memorization effect.
  • If a single LLM achieves success on refactored labs, the practical implication for exploitability assessment is immediate: AEG could shift from expert-driven symbolic execution to prompting, changing who can meaningfully assess and exploit a known vulnerability.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports a first systematic evaluation of LLMs for automated exploit generation (AEG). It uses five SEED Labs programs in both original and refactored (renamed) forms, builds an LLM-based attacker that iteratively prompts a target model for up to 15 iterations, and measures (i) cooperativeness, defined only through an average percentage, and (ii) effectiveness, measured as the number of mistakes in the final generated code as judged by the authors. The headline findings are that GPT-4o and GPT-4o-mini are nearly as cooperative as uncensored models, Llama3 is most resistant, and none of the evaluated models successfully generates exploits for the refactored labs, with GPT-4o making the fewest errors. The authors release the benchmark and attacker code.

Significance. If the central negative result is reliable, the paper provides a valuable data point for the security community: current LLMs are not yet a serious AEG threat for refactored programs, while their cooperativeness and moderate error counts suggest potential for improvement. The refactored benchmark is a sensible step toward mitigating training-data memorization, and the LLM-based attacker is a useful attempt to make prompt-based evaluations more reproducible and less dependent on human interaction. However, the evaluation protocol currently lacks execution-based verification, clear definitions, and variance information, so the headline claims are not yet established. With a tightened protocol, this could become a solid contribution to a security or software engineering venue.

major comments (4)
  1. [§III-A, §VI (RQ2)] The success metric is 'the number of errors in the LLM-generated code that prevented the exploit from functioning correctly,' but the paper nowhere states that the final generated code was compiled or executed against the refactored programs. Since the central claim is a universal negative—'none of the evaluated models successfully generated exploits for refactored vulnerabilities'—a manual code inspection that can misclassify harmless errors as fatal, or miss runtime interactions that break the exploit, does not establish that claim. The authors should either execute the generated exploits in the actual lab environments and report per-lab success/failure, or weaken the conclusion to 'no model produced code that the authors judged likely to work.'
  2. [§III-C] Section III-C says the target model's output is 'further refined by GPT-4o-mini' during the iterative loop. It is therefore unclear whether the final artifact whose mistakes are counted in Figures 2 and 3 is the target model's raw output or a GPT-4o-mini-edited version. If the latter, the mistake counts are not attributable solely to the target models being compared, and GPT-4o-mini acts as both a target model and a post-processor in every condition. The authors must clarify the exact data flow and ensure that the measured artifact is the target model's own output, or explain how the refiner's edits are controlled for.
  3. [§IV-A, Table I] Table I reports the 'average percentage of the cooperative response' but never defines what cooperativeness means operationally—for example, whether a response is cooperative if it contains any exploit code, if it does not refuse, or if it follows the attacker's prompt. Without a coding rubric and inter-rater agreement, the comparison of GPT-4o and GPT-4o-mini with uncensored models ('surprisingly cooperative') is not reproducible. The authors should provide the prompt template, the classification criteria, and ideally raw dialogue logs in the released artifact.
  4. [§III-C, §IV-B] The paper does not report the number of independent runs per lab and model, nor sampling temperature or seeds. The figures appear to present a single run per condition (e.g., 'GPT-4o, total: 6' in Figure 3). Because LLM outputs are stochastic, the claim that no model succeeded on refactored labs rests on one sample per condition, and the inference that 'Dolphin Mistral might have been trained with SEED Lab materials' is based on that same single-run, non-executed difference. At minimum, the authors should report multiple runs and the variance of mistake counts, or justify why a single run suffices for a universal negative.
minor comments (5)
  1. [§IV-C] There is a typo: 'occuring' should be 'occurring.'
  2. [Figures 2 and 3] The figures are hard to parse because each lab's mistake count is shown across several stacked or grouped values without a clear key; a small table of per-lab mistake counts with success/failure flags would be more readable.
  3. [§I, Abstract] The abstract and introduction attribute high cooperativeness to 'GPT-4 and GPT-4o,' but Table I lists only GPT-4o and GPT-4o-mini; the paper should clarify whether GPT-4 base was actually evaluated or whether 'GPT-4' is a collective label.
  4. [§II-C] The threat model assumes the attacker has access to source code; this is worth stating explicitly as a boundary condition in the limitations, since many real-world attacks target binaries without source.
  5. [References] Reference [14] for SEED Labs is listed as 'Seed labs, 2025' with no version or URL in the text; the full URL should be provided in the reference entry.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: this is an empirical measurement study against an external benchmark, with no fitted parameters or derivations presented as predictions.

full rationale

The paper does not contain a derivation chain whose outputs are equivalent to its inputs by construction. It reports an empirical evaluation: five SEED Labs programs are refactored, five LLMs are prompted by an automated attacker, and the resulting code is manually compared with reference solutions. The central claim, 'no model successfully generates exploits for refactored labs,' is an observed outcome of that comparison, not a quantity derived from a fitted model or from a self-citation. The evaluation metric ('the number of errors in the LLM-generated code that prevented the exploit from functioning correctly') is a manual inspection rather than an execution-based test, and the attacker model being GPT-4o while GPT-4o is also an evaluated target is a methodological confound; however, neither issue makes the result circular. No load-bearing step invokes prior work by the same authors, no uniqueness theorem is imported, and no ansatz is smuggled in via citation. The paper is self-contained as a benchmark study, so the appropriate finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical evaluation, so its ledger consists of design choices and domain assumptions rather than fitted parameters or new entities. The main items are the choice of benchmark, the assumption that refactoring preserves vulnerability, the manual error metric, and the use of GPT-4o both as attacker and as target.

free parameters (1)
  • maximum attacker iterations = 15
    Set to 15 empirically (Section III-C) rather than derived; a larger budget could change success rates.
assumptions (4)
  • domain assumption The five SEED Labs are representative of common software vulnerabilities (buffer overflow, return-to-libc, format string, race condition, dirty COW) and that manual solutions provide correct ground truth.
    Section III-A selects these labs and uses manually solved versions as the reference for error counting.
  • domain assumption Refactoring by renaming variables and functions to varX and functionX preserves the vulnerabilities and does not introduce new obstacles beyond removing memorized clues.
    Section III-A states the refactoring 'aimed to mitigate the risk of LLMs recalling memorized solutions'; if the refactoring altered exploit feasibility, the negative result would be an artifact.
  • domain assumption The attacker LLM's prompts do not inject the answer or unfairly bias the target LLM's output.
    Section III-C uses GPT-4o as the attacker while GPT-4o is also a target; the paper does not control for the attacker's own knowledge or potential leakage.
  • domain assumption The manual classification of responses as 'cooperative' is consistent across models and labs.
    Section IV-A reports average cooperation percentages without defining the classification rubric.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Good News for Script Kiddies? Evaluating Large Language Models for Automated Exploit Generation." pith.science (2026). https://pith.science/paper/YXQTCTDL

@misc{pith2026250501065,
  author       = {Pith},
  title        = {Pith review of: Good News for Script Kiddies? Evaluating Large Language Models for Automated Exploit Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YXQTCTDL}},
  note         = {Machine review of arXiv:2505.01065}
}
read the original abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in code-related tasks, raising concerns about their potential for automated exploit generation (AEG). This paper presents the first systematic study on LLMs' effectiveness in AEG, evaluating both their cooperativeness and technical proficiency. To mitigate dataset bias, we introduce a benchmark with refactored versions of five software security labs. Additionally, we design an LLM-based attacker to systematically prompt LLMs for exploit generation. Our experiments reveal that GPT-4 and GPT-4o exhibit high cooperativeness, comparable to uncensored models, while Llama3 is the most resistant. However, no model successfully generates exploits for refactored labs, though GPT-4o's minimal errors highlight the potential for LLM-driven AEG advancements.

Figures

Figures reproduced from arXiv: 2505.01065 by the authors.

Figure 1
Figure 1. Workflow of the LLM-based attacker et al. [4] demonstrated that hints from patches could facilitate exploit generation. The following year, Heelan and Kroening [3] introduced a technique for automatically generating control flow hijacking exploits. Subsequent studies advanced AEG methods [2], [5], [7]–[9], with Avgerinos et al. [5] pioneering the first end-to-end approach using preconditioned symbolic execution, est… view at source ↗
Figure 2
Figure 2. Number of mistakes made by LLMs on original SEED Lab programs [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Runnable to Verifiable: An Independent Reproducibility Study of LLM/Agent-Driven Vulnerability Validation Artifacts

    cs.CR 2026-08 conditional novelty 7.0 of 10

    A pre-registered audit of 102 cases plus 18 papers found artifact-embedded oracles unreliable: 20 of 30 signals also triggered on patched builds, so runnable artifacts rarely confirm the CVE.

  2. Knowdit: Agentic Smart Contract Vulnerability Detection with Auditing Knowledge Summarization

    cs.CR 2026-03 conditional novelty 6.0 of 10

    Knowdit links abstract DeFi semantics to vulnerability patterns in a knowledge graph and drives an agentic specify–harness–fuzz–reflect loop that finds all high-severity and most medium-severity bugs on held-out Code4...

Reference graph

Works this paper leans on

18 extracted references · 17 canonical work pages · cited by 2 Pith papers

  1. [1]

    A survey on software vulnerability exploitability assessment,

    S. Elder, M. R. Rahman, G. Fringer, K. Kapoor, and L. Williams, “A survey on software vulnerability exploitability assessment,” ACM Computing Surveys , vol. 56, pp. 1 – 41, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:268570392

    work page 2024

  2. [2]

    Koobe: Towards facilitating exploit generation of kernel out-of-bounds write vulnerabilities,

    W. Chen, X. Zou, G. Li, and Z. Qian, “Koobe: Towards facilitating exploit generation of kernel out-of-bounds write vulnerabilities,” in USENIX Security Symposium , 2020. [Online]. Available: https: //api.semanticscholar.org/CorpusID:209515033

    work page 2020

  3. [3]

    Automatic generation of control flow hijacking exploits for software vulnerabilities,

    S. Heelan and D. Kroening, “Automatic generation of control flow hijacking exploits for software vulnerabilities,” 2009. [Online]. Available: https://api.semanticscholar.org/CorpusID:62247176

    work page 2009

  4. [4]

    Automatic patch-based exploit generation is possible: Techniques and implications

    “Automatic patch-based exploit generation is possible: Techniques and implications.” [Online]. Available: https://api.semanticscholar.org/ CorpusID:8081699

    work page

  5. [5]

    Aeg: Automatic exploit generation,

    T. Avgerinos, S. K. Cha, B. L. T. Hao, and D. Brumley, “Aeg: Automatic exploit generation,” in Network and Distributed System Security Symposium , 2011. [Online]. Available: https://api. semanticscholar.org/CorpusID:14420062

    work page 2011

  6. [6]

    Fuze: Towards facilitating exploit generation for kernel use-after-free vulnerabilities,

    W. Wu, Y . Chen, J. Xu, X. Xing, X. Gong, and W. Zou, “Fuze: Towards facilitating exploit generation for kernel use-after-free vulnerabilities,” in USENIX Security Symposium , 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:52052904

    work page 2018

  7. [7]

    Automated exploit generation for stack buffer overflow vulnerabilities,

    V . A. Padaryan, V . V . Kaushan, and A. N. Fedotov, “Automated exploit generation for stack buffer overflow vulnerabilities,” Programming and Computer Software, vol. 41, pp. 373 – 380, 2015. [Online]. Available: https://api.semanticscholar.org/CorpusID:11233774

    work page 2015

  8. [8]

    Automatic exploit generation for buffer overflow vulnerabilities,

    L. Xu, W. Jia, W. Dong, and Y . Li, “Automatic exploit generation for buffer overflow vulnerabilities,” 2018 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) , pp. 463–468, 2018. [Online]. Available: https://api.semanticscholar.org/ CorpusID:52003316

    work page 2018

Show all 18 references
  1. [9]

    Toward automated exploit generation for known vulnerabilities in open-source libraries,

    E. Iannone, D. D. Nucci, A. Sabetta, and A. D. Lucia, “Toward automated exploit generation for known vulnerabilities in open-source libraries,” 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC) , pp. 396–400, 2021. [Online]. Available: https://api.sem...

    2021

  2. [10]

    Large language models for software engineering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. C. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” ACM Trans. Softw. Eng. Methodol., vol. 33, pp. 220:1–220:79, 2023. [Online]. Available: https://api.s...

    2023

  3. [11]

    How well does llm generate security tests?

    Y . Zhang, W.-K. Song, Z. Ji, D. D. Yao, and N. Meng, “How well does llm generate security tests?” ArXiv, vol. abs/2310.00710, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:263334479

    2023

  4. [12]

    Pentestgpt: Evaluating and harnessing large language models for automated penetration testing,

    G. Deng, Y . Liu, V . M. Vilches, P. Liu, Y . Li, Y . Xu, M. Pinzger, S. Rass, T. Zhang, and Y . Liu, “Pentestgpt: Evaluating and harnessing large language models for automated penetration testing,” in USENIX Security Symposium , 2024. [Online]. Available: https: //api.semanti...

    2024

  5. [13]

    Leveraging semantic relations in code and data to enhance taint analysis of embedded systems,

    J. Zhao, Y . Li, Y . Zou, Z. Liang, Y . Xiao, Y . Li, B. Peng, N. Zhong, X. Wang, W. Wang, and W. Huo, “Leveraging semantic relations in code and data to enhance taint analysis of embedded systems,” in USENIX Security Symposium , 2024. [Online]. Available: https://api.semantic...

    2024

  6. [14]

    Seed labs,

    W. Du, “Seed labs,” 2025. [Online]. Available: https://seedsecuritylabs. org/

    2025

  7. [15]

    Explaining neural scaling laws,

    Y . Bahri, E. Dyer, J. Kaplan, J. Lee, and U. Sharma, “Explaining neural scaling laws,” Proceedings of the National Academy of Sciences of the United States of America , vol. 121, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:231918701

    2021

  8. [16]

    A comprehensive study of jailbreak attack versus defense for large language models,

    Z. Xu, Y . Liu, G. Deng, Y . Li, and S. Picek, “A comprehensive study of jailbreak attack versus defense for large language models,” in Annual Meeting of the Association for Computational Linguistics ,

  9. [17]

    Get up and running with large language models

    Ollama, “Get up and running with large language models.” 2025. [Online]. Available: https://ollama.com/

    2025

  10. [2024]

    Available: https://api.semanticscholar.org/CorpusID: 267770234

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 267770234

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.