Pith. sign in

REVIEW 4 major objections 6 minor 4 cited by

FaultLine: Automated Proof-of-Vulnerability Generation Using LLM Agents

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A staged LLM workflow writes correct proof-of-vulnerability tests for 16 of 100 known vulnerabilities.

desk verdict A genuinely new staged-agent pipeline for PoV tests plus a reusable 100-vuln benchmark, but the 16-vs-9 headline is only as strong as a PoV criterion that never checks the fixed state; worth a serious referee. read the letter →

arxiv 2507.15241 v1 pith:LDYL3RYL submitted 2025-07-21 cs.SE

Vikram Nitin , Baishakhi Ray , Roshanak Zilouchian Moghaddam This is my paper · ORCID
classification cs.SE
keywords proof-of-vulnerabilitytestsLLMagentstestgenerationdataflowreasoningcontrolsoftwarevulnerabilitiesagenticworkflow
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vulnerability reports rarely include a proof-of-vulnerability (PoV) test: a failing test that demonstrates the exploit, validates a fix, and guards against regressions. Writing such a test is hard because it requires tracing attacker-controlled data from an external entry point to the vulnerability sink, and constructing an input that satisfies every branch condition along the way. FaultLine claims that an LLM agent can do this substantially better by making those two reasoning steps explicit, first extracting a source-to-sink data flow, then deriving the input constraints implied by the branch conditions, before generating and repairing the test in a feedback loop. On a curated benchmark of 100 known vulnerabilities in Java, C, and C++, it reports correct PoV tests for 16 projects versus 9 for the CodeAct 2.1 baseline, a 77% relative improvement. The paper reads the result as evidence that hierarchical reasoning helps LLM agents on this task, while noting that the problem remains largely unsolved.

What carries the argument

The machinery is a three-stage hierarchical reasoning workflow, with the intermediate artifacts, not the final test, doing the work. Stage 1 (data flow reasoning) produces a labeled source-to-sink path from the vulnerability report and repository exploration tools such as find and grep. Stage 2 (control flow reasoning) converts that path into a sequence of branch conditions (if, switch, try-except) and then into a distilled set of input conditions, so the generated test input can walk the tightrope through the branches. Stage 3 generates a test that must actually execute the vulnerable code according to category-specific success criteria (for example, a CWE-94 test must inject code that is executed), runs it in Docker, and repairs it from the build or run output in a loop. The workflow deliberately uses no language-specific static or dynamic analysis, which is what lets the same prompts run across Java, C, and C++.

What would settle it

Run each of the 16 accepted FaultLine tests against the corresponding fixed commit (the commit that patches the vulnerability) and count how many build successfully and exit with code 0. If most of them still fail on the fixed version, the claim that these are genuine proof-of-vulnerability tests, and with it the 16-versus-9 headline, is weakened.

Watch

Extended reading notes

Core claim

The central claim is that decomposing PoV test generation into explicit reasoning stages lets an LLM agent outperform a general-purpose coding agent on real-world vulnerabilities. FaultLine first prompts the model to reconstruct the vulnerable flow from source to sink, labeling each program point with its file, variable, and role. It then prompts the model to enumerate every branch condition along that flow and to compose them into a list of concrete constraints an input must satisfy. Finally, a test-generation prompt, seeded with the flow and the input constraints, writes a test in a Docker-contained project and repairs it iteratively from build and run feedback. The paper reports 16 correct PoV tests on 100 vulnerabilities, versus 9 for the CodeAct baseline, and 31 tests that reach the vulnerable function, versus 19; ablations show that removing either the flow or the branch reasoning component drops the success count to between 9 and 11.

Load-bearing premise

The counting rule for a correct test is that it builds at the vulnerable commit, exits non-zero when run, and calls a method touched by the fix, but there is no check that the same test passes once the vulnerability is patched, so a test that fails for unrelated reasons is still counted as a success.

Editorial extensions

If this is right

  • Correct PoV test generation becomes a staged reasoning problem: future agent designs should expose flow paths and branch-derived input constraints as first-class artifacts rather than expecting end-to-end generation to work.
  • The same prompt structure, without language-specific tools, transfers across Java, C, and C++, so the workflow can be pointed at other CWE categories and languages with minimal changes.
  • Reaching the vulnerable function (31 cases) is much easier than building a correct exploit test (16 cases), so improving payload construction and semantic verification is the next bottleneck to attack.
  • The released 100-vulnerability benchmark gives the community a reproducible way to compare PoV-generation agents on a harder task than general bug reproduction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the reported 16-vs-9 gap is an upper bound on genuine PoV quality: since the evaluation never runs the tests against the patched code, some of the 16 may be failing for build, dependency, or environment reasons rather than because they trigger the vulnerability; the paper's own limitation discussion grants this possibility.
  • We infer that the biggest wins from the workflow come from the branch-condition stage, since the motivating failure of the baseline (an input with the wrong number of cron fields) is exactly what the input-constraint step catches; a testable extension would be to replace that step with a lightweight constraint solver or symbolic execution and measure whether the gap widens or shrinks.
  • We infer that the same decomposition could be applied to other security tasks, such as regression-test generation for patched code or fuzzing seed construction, where the value of an input depends on controlling a long path through branches.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents FaultLine, an LLM-agent workflow for automatically generating proof-of-vulnerability (PoV) tests for known vulnerabilities. The workflow has three stages: data-flow reasoning from a source to a sink, control-flow or branch reasoning that derives input constraints, and test generation plus feedback-driven repair inside Docker. The authors evaluate FaultLine on a newly collated dataset of 100 vulnerabilities from CWE-Bench-Java (70 Java projects) and PrimeVul (30 C/C++ projects), comparing against the CodeAct 2.1 agent in OpenHands with the same underlying model (Claude 3.7 Sonnet) and the same budget. They report 16 correct PoV tests for FaultLine versus 9 for CodeAct, and 31 versus 19 tests that reach the vulnerable functions. They also report ablation results suggesting that the flow and branch reasoning components contribute to performance. The code, prompts, and dataset are released publicly.

Significance. If the reported results hold, this is a useful step toward automated PoV test generation: the multi-stage reasoning workflow is language-agnostic, the benchmark is a public resource, and the comparison uses the same model and budget for both systems. The paper also ships reproducible Docker-based builds, detailed prompts in the appendix, and explicit failure-stage accounting, which are strengths. The headline improvement over CodeAct is meaningful and the problem is clearly hard: only 16 of 100 vulnerabilities receive a PoV test. However, the evaluation has load-bearing gaps: correctness is never checked in the fixed state, each agent is run only once despite LLM stochasticity, and the ablation study is restricted to the 16 successful projects. These gaps mean the quantitative claims are plausible but not yet established. The benchmark and agent design are valuable regardless, and the limitations are partly acknowledged in Section 7, but the paper needs further evaluation work before the central claims can be accepted.

major comments (4)
  1. [Section 4.4 and Section 7] The correctness definition omits the fixed-state pass check, which is a necessary property of a PoV test. Section 4.4 requires only that the test builds at the vulnerable commit, exits non-zero, reaches a method touched by the fix, and passes a manual CWE-specific inspection. The Introduction defines a PoV test as failing when the vulnerability exists and succeeding when it is fixed, and Appendix Listing 5 explicitly instructs the agent that the test should PASS when the vulnerability is fixed, yet the evaluation never verifies this. Section 7 concedes this absence. Consequently, a test that fails at the vulnerable commit because of a wrong API assumption, a missing dependency, or an environment issue is indistinguishable from a test that actually demonstrates the vulnerability. The CWE-Bench-Java subset was already filtered using fixed-commit builds, so adding a fixed-state run is feasible for at least the Java portion. The authors should either add this check and report the fixed-state pass rate, or relabel the results as 'vulnerable-state failing tests' and temper the abstract and contribution claims accordingly.
  2. [Section 5.3, Table 2] The ablation study is run only on the 16 projects for which the full FaultLine configuration succeeded, which is selection on the dependent variable. This design cannot support the claim that both flow and branch reasoning are essential: the configurations without flow or branch reasoning are never given the opportunity to succeed on the other 84 projects, so the comparison only shows what happens on the already-solved subset. Moreover, the counts 9, 11, and 16 differ by only a few examples and are based on a single run per configuration, so the differences may be within stochastic noise. Please run the ablations on a representative sample of the full 100-project benchmark, or at least on a fixed random subset that is not defined by the full system's successes, and report repeated runs.
  3. [Section 5.1] The headline result of 16 versus 9 correct PoV tests rests on a single run of each stochastic LLM agent on each project. No temperature, seed, or randomization control is reported, and no statistical significance test or confidence interval accompanies the 7-project difference. With 100 paired instances and small counts, the observed gap could easily shift under repeated sampling. Please report multiple runs (e.g., 3 to 5 seeds) and a paired significance test, or otherwise justify that the stochasticity does not affect the comparison.
  4. [Section 3.3 and Section 4.4] The manual inspection criteria are the same CWE-specific criteria that are inserted verbatim into the test-generation prompt (Figure 4 and the corresponding appendix prompt). As a result, the manual check is not an independent oracle: it verifies that the agent followed the instructions it was given, rather than independently validating that the generated test demonstrates an exploit. The automated checks alone are too weak, because any non-zero exit at the vulnerable commit is accepted, so the manual inspection is the only guard against shallow tests. The authors should make the inspection more independent, report who performed it, and state whether the evaluators were blind to which system generated each test; this is especially important because the fixed-state check is absent.
minor comments (6)
  1. [Table 1] The Command Injection row lists a total of 16 even though the two columns contain 6 and 4, which sum to 10; with the printed row totals, the overall sum becomes 106 instead of 100. Please correct the row and verify all totals.
  2. [Abstract and Section 5.1] The vulnerable-function coverage counts are inconsistent: the Abstract reports 31 versus 21, the Section 5.1 summary reports 31 versus 19, and the in-text discussion reports 28 versus 17 for CWE-Bench-Java. The values derived from Figure 5's stage counts are again different, so the authors should audit these numbers and use one consistent set.
  3. [Section 2.1] The text says 'CWE-94 corresponds to Code Injection vulnerabilities, such as SQL injection,' but SQL injection is a distinct CWE (CWE-89). A CWE-94 example such as expression-language injection or template injection would be more accurate.
  4. [Section 4.1] The authors say they selected 30 PrimeVul vulnerabilities at random, but they do not report the random seed or the selection procedure. A reproducible selection process is needed for the benchmark to be reusable.
  5. [Figure 5] The numeric labels in the funnel diagrams are hard to parse and do not obviously match the prose in Section 5.1 (for example, the 'Succeeded 70 9 63 19 16' label). Please redraw the figure so each stage count and the transition from total to successful tests is explicit.
  6. [Section 4.4] The manual inspection is described as 'manual' but the paper does not report how many annotators performed it, what their agreement was, or whether they were blinded to the tool that generated each test. Even without a full inter-rater reliability study, reporting these details would strengthen the evaluation.

Circularity Check

1 steps flagged · score 2.0 of 10

No derivation-level circularity; a mild self-referential evaluation loop (the prompt supplies the same success criteria used to judge) and a missing fixed-commit check keep the headline claim under-supported but not circular.

  1. other [Section 3.3 (Figure 4) / Section 4.4 (Metrics); Section 7]
    "This is a Code Injection vulnerability (CWE-94). The test case must call a public API of the project with an input that contains embedded code, and this code must be executed. [Figure 4] ... manually inspect it to evaluate whether it satisfies the category-specific criteria listed in Section 3.3. [Section 4.4]"

    The final manual-inspection rubric in Section 4.4 is the same category-specific criterion that the test-generation prompt in Figure 4 already tells the model to satisfy. The judge therefore checks the instruction given to the generator, rather than an independent property of the generated artifact. Additionally, Section 7 admits the evaluation omits the fixed-commit pass check, so the paper's own definition of a PoV test (fail when vulnerable, succeed when fixed) is not the property being verified. This is a mild self-referential evaluation loop rather than an equation-level reduction.

full rationale

There is no equation-level circularity and no fitted parameter renamed as a prediction. The headline 16-vs-9 comparison is an empirical measurement: both FaultLine and CodeAct 2.1 use Claude-3.7-Sonnet with identical budget and time constraints, and the evaluation relies on build outcome, non-zero exit, fix-touched method coverage, and a manual CWE-specific check. The flow and branch reasoning components are ablated on the 16 successful projects, which is a biased but not circular design choice. The main concern is construct validity, not circularity: the CWE-specific success criteria are embedded in the test-generation prompt and also used as the manual inspection rubric, creating a mild self-referential loop, and Section 7 explicitly concedes that no check verifies the test passes on the fixed commit. A test that fails at the vulnerable commit for unrelated reasons could therefore be labeled correct, but this is an evaluation-objectivity threat rather than a derivation that reduces to its own inputs. One non-load-bearing self-citation exists (PrimeVul, reference [11], with co-author Ray), but it is only a dataset source and does not support the central claim. Overall, the paper's central contribution is not circular; the score reflects the minor self-referential evaluation loop and the acknowledged missing fixed-state validation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on evaluation assumptions rather than mathematical axioms or fitted parameters. The only constraints like the 5 USD budget, 40 minute time limit, and iteration cap are fixed experiment settings, not fitted values. The most load-bearing assumptions are the sufficiency of the non-zero-exit plus coverage proxy, the omission of a fixed-state check, and the objectivity of the manually applied CWE-specific criteria.

assumptions (4)
  • domain assumption A PoV test need only fail in the vulnerable state; passing in the fixed state is not required for evaluation.
    Section 4.4 defines success as build success plus non-zero exit plus reaching changed methods; Section 7 explicitly defers the fixed-state check.
  • domain assumption Reaching a method whose code changed in the fix commit is a valid proxy for reaching the vulnerable sink.
    Section 4.4 item (3) instruments functions to print their names and considers the coverage check satisfied if any changed method is called; this may not correspond to executing the actual vulnerable path.
  • domain assumption The CWE-specific semantic criteria defined by the authors are an objective ground truth for a successful exploit.
    Section 3.3 defines per-CWE criteria (e.g., CWE-94 requires embedded code actually be executed) and Section 4.4 relies on manual inspection against these criteria; Section 7 acknowledges possible errors in human judgment.
  • domain assumption The LLM's training-data exposure to the benchmark projects does not invalidate the comparison because the baseline uses the same model.
    Section 7 discusses data leakage and argues the baseline comparison neutralizes the advantage; this mitigates but does not eliminate leakage bias in absolute success rates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FaultLine: Automated Proof-of-Vulnerability Generation Using LLM Agents." pith.science (2026). https://pith.science/paper/LDYL3RYL

@misc{pith2026250715241,
  author       = {Pith},
  title        = {Pith review of: FaultLine: Automated Proof-of-Vulnerability Generation Using LLM Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LDYL3RYL}},
  note         = {Machine review of arXiv:2507.15241}
}
read the original abstract

Despite the critical threat posed by software security vulnerabilities, reports are often incomplete, lacking the proof-of-vulnerability (PoV) tests needed to validate fixes and prevent regressions. These tests are crucial not only for ensuring patches work, but also for helping developers understand how vulnerabilities can be exploited. Generating PoV tests is a challenging problem, requiring reasoning about the flow of control and data through deeply nested levels of a program. We present FaultLine, an LLM agent workflow that uses a set of carefully designed reasoning steps, inspired by aspects of traditional static and dynamic program analysis, to automatically generate PoV test cases. Given a software project with an accompanying vulnerability report, FaultLine 1) traces the flow of an input from an externally accessible API ("source") to the "sink" corresponding to the vulnerability, 2) reasons about the conditions that an input must satisfy in order to traverse the branch conditions encountered along the flow, and 3) uses this reasoning to generate a PoV test case in a feedback-driven loop. FaultLine does not use language-specific static or dynamic analysis components, which enables it to be used across programming languages. To evaluate FaultLine, we collate a challenging multi-lingual dataset of 100 known vulnerabilities in Java, C and C++ projects. On this dataset, FaultLine is able to generate PoV tests for 16 projects, compared to just 9 for CodeAct 2.1, a popular state-of-the-art open-source agentic framework. Thus, FaultLine represents a 77% relative improvement over the state of the art. Our findings suggest that hierarchical reasoning can enhance the performance of LLM agents on PoV test generation, but the problem in general remains challenging. We make our code and dataset publicly available in the hope that it will spur further research in this area.

Figures

Figures reproduced from arXiv: 2507.15241 by the authors.

Figure 1
Figure 1. Motivating example - A Code Injection vulnerability from the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 1
Figure 1. We can see that there are 6 if conditions and 2 try..except blocks. Each of these corresponds to a constraint on the input that has to be satisfied in order for the control flow to reach the vulnerability sink. For example, the condition on lines 9-11 of CronParser.java expects the expression to have a certain number of fields (number of space-separated components). If there is a mismatch between expressionLength an… view at source ↗
Figure 2
Figure 2. Our system FaultLine, for generating Proof of Vulnerability test cases. execution feedback. Although the final reasoning generated in one stage is passed on to subsequent stages, the various stages do not share a common conversation memory. This keeps the length of each conversation in check. Full LLM prompts for all these stages are available in the appendix. 3.1 Data Flow Reasoning We start with a project containi… view at source ↗
Figures from the paper (5 more)
Figure 3
Figure 3. Figure 3: An example of a vulnerability report, for CVE-2021- [PITH_FULL_IMAGE:figures/full_fig_p004_3.png]
Figure 4
Figure 4. Figure 4: A portion of our test generation prompt for a CWE [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: A detailed analysis of the test generation performance of CodeAct 2.1 (on the left) vs [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Percentage of correct PoV tests per CWE category. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: A portion of the output of flow and branch rea [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?

    cs.CR 2026-05 unverdicted novelty 7.0 of 10

    SEC-bench Pro benchmark with 183 real vulnerabilities shows frontier LLM coding agents achieve at most 38.8% success on SpiderMonkey and 32% on V8.

  2. Agentic Vulnerability Reasoning on COTS Binaries

    cs.CR 2026-05 conditional novelty 7.0 of 10

    SLYP agentic pipeline discovers race condition vulnerabilities in Windows COM binaries and generates debugger-verified PoCs, scoring 0.973 F1 on a 40-case benchmark and finding 28 new confirmed vulnerabilities in prod...

  3. AXE: Grey-Box Exploitability Confirmation for Localized Vulnerability Reports

    cs.CR 2026-02 conditional novelty 7.0 of 10

    Grey-box metadata (CWE + code location) plus a multi-agent LLM workflow raises automated web-exploit confirmation from ~10% to 30% on CVE-Bench, with actionable PoC output.

  4. Neuro-Symbolic Proof-of-Vulnerability Generation with Open-Weight Models

    cs.SE 2026-08 conditional novelty 6.0 of 10

    POVGEN generates proof-of-vulnerability inputs for roughly three quarters of real-world C/C++ memory-safety CVEs using fine-tuned open-weight LLMs guided by symbolic path analysis.

Reference graph

Works this paper leans on

44 extracted references · 21 canonical work pages · cited by 4 Pith papers

  1. [1]

    [n. d.]. https://codeql.github.com/

    work page

  2. [2]

    Talor Abramovich, Meet Udeshi, Minghao Shao, Kilian Lieret, Haoran Xi, Kim- berly Milner, Sofija Jancheska, John Yang, Carlos E Jimenez, Farshad Khorrami, et al. [n. d.]. EnIGMA: Interactive Tools Substantially Assist LM Agents in Find- ing Security Vulnerabilities. In Forty-second International Conference on Machine Learning

    work page

  3. [3]

    Toufique Ahmed, Jatin Ganhotra, Rangeet Pan, Avraham Shinnar, Saurabh Sinha, and Martin Hirzel. 2025. Otter: Generating Tests from Issues to Validate SWE Patches. arXiv preprint arXiv:2502.05368 (2025)

    arXiv 2025

  4. [4]

    Toufique Ahmed, Martin Hirzel, Rangeet Pan, Avraham Shinnar, and Saurabh Sinha. 2024. TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved? arXiv preprint arXiv:2412.02883 (2024)

    arXiv 2024

  5. [5]

    Abeer Alhuzali, Birhanu Eshete, Rigel Gjomemo, and VN Venkatakrishnan. 2016. Chainsaw: Chained automated workflow-based exploit generation. InProceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security . 641–652

    work page 2016

  6. [6]

    Thanassis Avgerinos, Sang Kil Cha, Alexandre Rebert, Edward J Schwartz, Mav- erick Woo, and David Brumley. 2014. Automatic exploit generation. Commun. ACM 57, 2 (2014), 74–84

    work page 2014

  7. [7]

    Masudul Hasan Masud Bhuiyan, Adithya Srinivas Parthasarathy, Nikos Vasilakis, Michael Pradel, and Cristian-Alexandru Staicu. 2023. SecBench. js: An executable security benchmark suite for server-side JavaScript. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 1059–1070

    2023

  8. [8]

    David Brumley, Pongsin Poosankam, Dawn Song, and Jiang Zheng. 2008. Auto- matic patch-based exploit generation is possible: Techniques and implications. In 2008 IEEE Symposium on Security and Privacy (sp 2008) . IEEE, 143–157

    work page 2008

Show all 44 references
  1. [9]

    Quang-Cuong Bui, Riccardo Scandariato, and Nicolás E Díaz Ferreyra. 2022. Vul4J: a dataset of reproducible Java vulnerabilities geared towards the study of program repair techniques. In Proceedings of the 19th International Conference on Mining Software Repositories. 464–468

    2022

  2. [10]

    Runxiang Cheng, Michele Tufano, Jürgen Cito, José Cambronero, Pat Rondon, Renyao Wei, Aaron Sun, and Satish Chandra. 2025. Agentic Bug Reproduction for Effective Automated Program Repair at Google. arXiv preprint arXiv:2502.01821 (2025)

    2025 arXiv

  3. [11]

    Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. 2024. Vulnerability detection with code language models: How far are we? arXiv preprint arXiv:2403.18624 (2024)

    2024 arXiv

  4. [12]

    Jiahao Fan, Yi Li, Shaohua Wang, and Tien N Nguyen. 2020. AC/C++ code vulnerability dataset with code changes and CVE summaries. In Proceedings of the 17th international conference on mining software repositories . 508–512

    2020

  5. [13]

    Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. 2024. Llm agents can autonomously exploit one-day vulnerabilities. arXiv preprint arXiv:2404.08144 13 (2024), 14

    2024 arXiv

  6. [14]

    Nima Shiri Harzevili, Alvine Boaye Belle, Junjie Wang, Song Wang, Zhen Ming, Nachiappan Nagappan, et al . 2023. A survey on automated software vulner- ability detection using machine learning and deep learning. arXiv preprint arXiv:2306.11673 (2023)

    2023 arXiv

  7. [15]

    Jingxuan He and Martin Vechev. 2023. Large language models for code: Secu- rity hardening and adversarial testing. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security . 1865–1879

    2023

  8. [16]

    Hong Hu, Zheng Leong Chua, Sendroiu Adrian, Prateek Saxena, and Zhenkai Liang. 2015. Automatic Generation of{Data-Oriented} Exploits. In 24th USENIX Security Symposium (USENIX Security 15) . 177–192

    2015

  9. [17]

    Shih-Kun Huang, Han-Lin Lu, Wai-Meng Leong, and Huan Liu. 2013. Craxweb: Automatic web application testing and attack generation. In 2013 IEEE 7th Inter- national Conference on Software Security and Reliability . IEEE, 208–217

    2013

  10. [18]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 (2023)

    2023 arXiv

  11. [19]

    Sungmin Kang, Juyeon Yoon, Nargiz Askarbekkyzy, and Shin Yoo. 2024. Evaluat- ing diverse large language models for automatic and general bug reproduction. IEEE Transactions on Software Engineering (2024)

    2024

  12. [20]

    Ziyang Li, Saikat Dutta, and Mayur Naik. 2025. IRIS: LLM-assisted static analysis for detecting security vulnerabilities. In The Thirteenth International Conference on Learning Representations

    2025

  13. [21]

    Xiang Mei, Pulkit Singh Singaria, Jordi Del Castillo, Haoran Xi, Tiffany Bao, Ruoyu Wang, Yan Shoshitaishvili, Adam Doupé, Hammond Pearce, Brendan Dolan-Gavitt, et al. 2024. ARVO: Atlas of Reproducible Vulnerabilities for Open Source Software. arXiv preprint arXiv:2408.02153 (2024)

    2024 arXiv

  14. [22]

    MITRE Corporation. 2025. Common Weakness Enumeration. https://cwe.mitre. org Accessed: July 18, 2025

    2025

  15. [23]

    Dongliang Mu, Alejandro Cuevas, Limin Yang, Hang Hu, Xinyu Xing, Bing Mao, and Gang Wang. 2018. Understanding the reproducibility of crowd-reported security vulnerabilities. In 27th USENIX Security Symposium (USENIX Security 18). 919–936

    2018

  16. [24]

    National Institute of Standards and Technology. 2025. National Vulnerability Database. https://nvd.nist.gov/vuln Accessed: July 15, 2025

    2025

  17. [25]

    Graham Neubig and Xingyao Wang. 2024. OpenHands CodeAct 2.1: An Open, State-of-the-Art Software Development Agent. All Hands AI Blog (1 November 2024). https://www.all-hands.dev/blog/openhands-codeact-21-an-open-state- of-the-art-software-development-agent

    2024

  18. [26]

    Georgios Nikitopoulos, Konstantina Dritsa, Panos Louridas, and Dimitris Mitropoulos. 2021. CrossVul: a cross-language vulnerability dataset with commit data. In Proceedings of the 29th ACM Joint Meeting on European Software Engi- neering Conference and Symposium on the Foundat...

    2021

  19. [27]

    2017.{OSS-Fuzz}-Google’s continuous fuzzing service for open source software

    Kostya Serebryany. 2017.{OSS-Fuzz}-Google’s continuous fuzzing service for open source software. (2017)

    2017

  20. [28]

    Deniz Simsek, Aryaz Eghbali, and Michael Pradel. 2025. PoCGen: Generating Proof-of-Concept Exploits for Vulnerabilities in Npm Packages. arXiv preprint arXiv:2506.04962 (2025)

    2025 arXiv

  21. [29]

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024. Executable code actions elicit better llm agents. In Forty-first International Conference on Machine Learning

    2024

  22. [30]

    Xingyao Wang, Bowen Li, and Graham Neubig. 2024. Introducing OpenDevin CodeAct 1.0, a new State-of-the-art in Coding Agents. Blog (7 May 2024). https: //xwang.dev/blog/2024/opendevin-codeact-1.0-swebench/

    2024

  23. [31]

    Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brenna...

    2025

  24. [32]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems 37 (2024), 50528–50652

    2024

  25. [33]

    Wei You, Peiyuan Zong, Kai Chen, XiaoFeng Wang, Xiaojing Liao, Pan Bian, and Bin Liang. 2017. Semfuzz: Semantics-based automatic generation of proof-of- concept exploits. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security. 2139–2154

    2017

  26. [34]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Au- tocoderover: Autonomous program improvement. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis . 1592–1604

    2024

  27. [35]

    Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang, Conner Jensen, Eric Ihli, Jason Benn, et al. 2025. CVE-Bench: A Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Conference acronym ’XX, June 03–05, 2018, Wood...

    2025 arXiv

  28. [36]

    Yuxuan Zhu, Antony Kellermann, Akul Gupta, Philip Li, Richard Fang, Rohan Bindu, and Daniel Kang. 2024. Teams of llm agents can exploit zero-day vulnera- bilities. arXiv preprint arXiv:2406.01637 (2024). FaultLine: Automated Proof-of-Vulnerability Generation using LLM Agents C...

    2024 arXiv

  29. [37]

    EXPLORATION: Thoroughly explore relevant files and understand the context before proposing solutions

  30. [38]

    ANALYSIS: Consider multiple approaches and select the most promising one

  31. [39]

    IMPLEMENTATION: Make focused, minimal changes to address the problem </PROBLEM_SOLVING_WORKFLOW> <TROUBLESHOOTING> * If you 've made repeated attempts to solve a problem but tests still fail or the user reports it 's still broken:

  32. [40]

    Step back and reflect on 5-7 different possible sources of the problem

  33. [41]

    Assess the likelihood of each possible cause

  34. [42]

    Methodically address the most likely causes, starting with the highest probability

  35. [43]

    flow" through the program. The flow should take the form of a sequence of program points, each in the following format: {

    Document your reasoning process </TROUBLESHOOTING> Listing 2: Flow Reasoning The project I am working with has a vulnerability, reported as a CWE. The issue description says: {description} You do not have access to the internet or GitHub to look up more details. There are no v...

    2018

  36. [44]

    simulate\

    Condition 2 ... </CONDITIONS> Listing 5: Test Generation The project I am working with has a vulnerability, reported as a CWE. The issue description says: {description} You do not have access to the internet or GitHub to look up more details. There are no vulnerability reports...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.