Pith. sign in

REVIEW 3 major objections 4 minor 63 references

PoCEvolve: Generating Proof-of-Concept Exploits from Security Patches with Vulnerability-Aware Prompt Evolution

T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a working proof-of-concept exploit can be generated from a vulnerability-fixing commit and repository context alone, and that vulnerability-aware prompt evolution is what makes this possible.

desk verdict Patch-only PoC generation is a real, well-scoped contribution; prompt-evolution gains over PoCGen are plausible but not cleanly isolated from the extra generation budget, and the verifier oracle needs more scrutiny. read the letter →

arxiv 2607.22076 v1 pith:EPQTJNYG submitted 2026-07-24 cs.CR cs.SE

Duc Manh Tran , Ratnadira Widyasari , Ivana Clairine Irsan , Huihui Huang , Ting Zhang , Shar Lwin Khin , Ouh Eng Lieh , Hong Jin Kang
show 1 more author
David Lo
This is my paper · ORCID
classification cs.CRcs.SE
keywords exploitgenerationproof-of-conceptsecuritypatchespromptevolutionvulnerabilitycontextlargelanguagemodelsJavaScript/npmpatch-to-disclosurewindow
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that the security community has been waiting on the wrong input: a working proof-of-concept exploit can be synthesized from the vulnerability-fixing commit and the surrounding repository alone, before any detailed vulnerability report exists. It backs this with a measurement showing most fixes precede public disclosure (median 18 days) and that most patches contain no tests, then introduces a framework that reconstructs a vulnerability description from the commit, generates candidate exploits, and evolves its generation prompt using scores along eight vulnerability-context dimensions. On a curated set of 190 real-world JavaScript vulnerabilities, the framework lifts success from 48.4% (prior state of the art) to 58.4% with one LLM, and to 85.3% with a newer model using direct prompting. If right, defenders can get executable evidence during the window where they currently have none.

What carries the argument

The named central object is the vulnerability-context-guided Prompt Evolver: a loop that converts failed exploit-generation attempts into structured feedback by scoring each failed prompt on eight dimensions of vulnerability-relevant context, then uses Pareto-front selection and LLM-based synthesis to craft the next generation prompt. It is what turns a patch-only input into a working exploit.

What would settle it

Pick the successful exploits from the evaluation, run each in a clean container against both the vulnerable and patched package versions, and check manually whether the exploit actually triggers the vulnerable behavior only in the vulnerable version; if a material fraction fail that check, the oracle-based success rates are optimistic.

Watch

Extended reading notes

Core claim

The central claim is that the missing link between a patch and a trigger can be rebuilt automatically. The framework first analyzes the fixing commit to infer vulnerability type, vulnerable API, and a description; then feeds that reconstruction to an exploit generator; when generated candidates fail verification, an evolution phase scores the failed prompts along eight vulnerability-related dimensions (API, description, usage, skeleton, similar exploits, taint paths, debugger output, code coverage), selects Pareto-optimal candidates, and synthesizes a revised prompt. The paper shows this loop improves both a direct LLM generator and the prior program-analysis-assisted generator, with the lar

Load-bearing premise

The reported success rates are whatever the automated verifier says they are: the verifier is inherited from the prior generator, corrected only for one vulnerability type, and no precision/recall against manually confirmed ground truth is reported.

Editorial extensions

If this is right

  • Defenders could generate an executable trigger for a vulnerability within minutes of the fix commit, before any advisory is published.
  • The patch-only setting is not a niche: 97.1% of analyzed CVEs were patched before disclosure, and 70.2% of patches contain no tests.
  • Prompt evolution helps even when a detailed report is available, so the mechanism is additive to existing exploit generators.
  • The cost profile ($0.03–$0.25 per vulnerability) makes running the pipeline across a whole patch stream feasible within an 18-day median window.
  • The largest gains are on command injection, the type where the prior method produced no successes at all with one model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The verifier is the silent partner: if its accept/reject does not match ground truth, every headline percentage moves. A defender should re-validate any generated PoC against patched versus unpatched behavior before acting.
  • The eight scoring dimensions are generic enough that they could seed a similar evolution loop for other executable-artifact tasks, such as generating regression tests or crash reproducers from bug-fixing commits.
  • The runtime numbers suggest a continuous-monitoring deployment: as soon as a security fix lands in a watched repository, the pipeline can emit candidate evidence before the next CI run finishes.
  • The JavaScript-only evaluation leaves open whether the commit-analyzer reconstruction transfers to compiled languages, where build and runtime setup are more complex; the authors' language-agnostic claim is an untested extrapolation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. PoCEvolve addresses a timely problem: generating executable proof-of-concept (PoC) exploits from only a vulnerability-fixing commit and repository context, before a detailed vulnerability report is available. The paper first reports an empirical study of 26,803 CVEs showing that 97.1% were patched before public disclosure, with a median gap of 18 days, motivating a patch-only setting. It then presents PoCEvolve, which reconstructs vulnerability information from the commit, generates candidate exploits via an underlying Exploit Generator (either PoCGen or direct LLM prompting), and, after failures, uses a Prompt Evolver that scores failed prompts along eight vulnerability-context dimensions, applies Pareto-based selection, and iteratively evolves new prompts for up to five evolution iterations. Evaluation on SecBench.VFC.js (190 vulnerabilities) reports success rates of 58.4% for full PoCEvolve with GPT-4o-mini versus 48.4% for PoCGen, and 79.5% versus 64.2% with Qwen3.7-Plus; in a report-guided setting, PoCEvolve improves over PoCGen from 64.6% to 71.7%. Cost and runtime are reported per configuration.

Significance. If the reported results hold, the paper makes a useful practical contribution: defenders could obtain executable vulnerability evidence in the patch-to-disclosure window without waiting for a human-written advisory. The patch-to-disclosure statistics are a valuable empirical finding in their own right, and the construction of SecBench.VFC.js from SecBench.js is a reasonable benchmark adaptation. The evaluation uses an executable verifier, and the authors are transparent about the verifier correction for Command Injection and report detailed cost/runtime numbers. The main weaknesses are interpretive: the paper's central attribution claim—that vulnerability-aware prompt evolution is the driver of improvement—is not yet supported by controlled comparisons, and the verifier used as ground truth is not itself validated beyond a single correction.

major comments (3)
  1. [§V-C, §VI-B3, Table IV] The central comparison is not budget-matched. PoCEvolve's pipeline draws an initial population of five failed prompt candidates from prior PoCGen runs and then performs T=5 evolution iterations, each invoking the Exploit Generator G(p')—which itself uses PoCGen's up-to-30-round refinement loop (§VI-B5). The baselines (direct LLM prompting and PoCGen) are run once. Table IV shows that the full GPT-4o-mini configuration costs 2.47x its generator-only cost ($0.0895 vs $0.0363). Consequently, the 20.7% and 23.8% relative gains could be due to the substantially larger number of generation attempts rather than to vulnerability-aware prompt evolution. The paper needs an equal-budget control, e.g., running PoCGen multiple times with different seeds, or repeated LLM sampling with the same number of LLM calls, and reporting success as a function of attempts/cost.
  2. [§VI-B4, §VIII-B] All success rates and improvements are measured against the PoCGen verifier, with only the Command Injection verifier corrected. The paper does not report precision or recall of the corrected verifier against a ground-truth set of known-triggering and known-non-triggering inputs for the other four vulnerability types. If the verifier accepts false positives or rejects valid exploits for Path Traversal, Prototype Pollution, Code Injection, or ReDoS, the reported numbers change materially. The claim in §VIII-B that the verifiers and benchmarking code were manually validated is not a substitute for a systematic audit; a per-type validation, or at least a clear statement of the verifier's known failure modes, is needed to support the headline quantitative claims.
  3. [§V-C, Table I, Algorithm 1] The attribution to 'vulnerability-aware' prompt evolution is not isolated. The only comparisons are against PoCGen and direct LLM prompting, neither of which uses the Prompt Evolver; there is no ablation that removes the eight vulnerability-context dimensions, and no generic prompt-evolution baseline (e.g., GEPA without the vulnerability-specific rubric) under the same evolution budget. The observed gains could in principle be reproduced by non-vulnerability-specific self-consistency or random prompt mutation. An ablation varying the feedback mechanism, or an equal-budget comparison to a generic prompt-evolution method, is required to substantiate the claim that the vulnerability-context scoring, rather than the extra attempts or the evolutionary loop in general, is the active ingredient.
minor comments (4)
  1. [Algorithm 2] The Pareto selection pseudo-code uses the index d in lines 8-9, but the criteria are denoted K; this is confusing. Align the notation with Algorithm 1.
  2. [Listing 1 and Section III] The regular expressions use a caret-like typographic symbol [ˆ"\\]+ and [ˆ"] that may be confusing; use standard ASCII regex notation. Also, 'REDOS' should be 'ReDoS' for consistency.
  3. [§VI-B1] The text says the benchmark is 'the same subset of vulnerabilities from SecBench.js evaluated by Simsek et al.,' then introduces SecBench.VFC.js as a filtered, deduplicated 190-vulnerability set. The relationship and the reason for the large reduction (559 to 190) beyond commit availability could be stated more explicitly.
  4. [Table III] With Qwen3.7-Plus, the full PoCEvolve (79.5%) is lower than PoCEvolve(LLM) (85.3%) and both are close to the direct LLM baseline (77.9%). This deserves discussion: why does the more elaborate PoCGen-based generator perform worse than direct LLM prompting with this model, and what configuration is actually recommended in practice?

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PoCEvolve's success rates are measured by an external executable verifier, and the prompt-evolution mechanism is not equivalent to its inputs by construction.

full rationale

The paper's central claims are evaluated against an external executable oracle: a PoC is counted as successful only if the PoCGen verifier (with a manually corrected Command Injection check) executes it and confirms the trigger. Success rates are therefore not derived by substituting the Prompt Evolver's internal usefulness scores for the outcome; the eight vulnerability-context dimensions and the 5-iteration evolution budget are fixed design choices, not parameters fitted to reproduce the benchmark's success counts. No equation in the paper defines the final success metric in terms of the evolved prompts or the LLM-generated rubric scores, so no prediction reduces to an input by construction. The self-citations that appear (e.g., VulCurator [25] for commit monitoring realism, Test mimicry [29] for API/useful-context support) are background or secondary citations, are corroborated by external references, and do not carry the load-bearing derivation of the reported improvements. The paper itself discloses limitations such as context-length failures and propagation of generator errors, which are validity concerns rather than circular steps. The most substantive threat to the attribution claim is the budget mismatch: full PoCEvolve consumes more generator and evolver calls than the baselines (cost is 2.47x the generator-only cost), so the 20.7% relative improvement could partly reflect extra attempts rather than vulnerability-aware prompt evolution. This is an experimental confound, not a definitional equivalence, and under the stated rubric it does not constitute circularity. Since no specific reduction of the claimed result to its own inputs can be exhibited, the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or formal entities are introduced; the only novel artifacts are prompts, scoring rubrics, and a filtered benchmark, none of which are invented entities in the sense of new mechanisms. The ledger instead consists of empirical assumptions about the verifier oracle, LLM behavior, commit informativeness, benchmark representativeness, and the hand-designed scoring rubric.

free parameters (4)
  • Evolution budget T = 5
    Prompt Evolver terminates after 5 evolution iterations (Section V-C); no sensitivity analysis shows whether this value was tuned on the benchmark.
  • Initial failed-candidate pool size = 5
    Five semantically diverse failed prompt candidates are retained to seed the Prompt Evolver (Section V-C); hand-chosen, no ablation.
  • POCEVOLVE(LLM) refinement-loop limit = 5
    For the direct-LLM configuration, the generator's refinement loop is limited to five iterations to balance success and cost (Section VI-B5); not justified by sensitivity analysis.
  • Temperature = 1
    Inherited from PoCGen and used for all LLM calls; affects stochastic variation and success rates, but only 'three repeated runs showed little variation' is reported.
assumptions (5)
  • domain assumption The PoCGen-derived verifier, with the authors' Command Injection correction, correctly determines whether a generated PoC triggers the target vulnerability.
    Load-bearing for all success rates; Section VI-B4 describes reuse and one correction but no independent precision/recall validation.
  • domain assumption LLM-generated exploits reflect synthesis rather than memorization or training-data contamination of the benchmark vulnerabilities.
    Raised as a threat in Section VIII-B; the proposed mitigation (comparing against baselines that would also benefit from memorization) does not rule out recall inflation.
  • domain assumption Vulnerability-fixing commits contain sufficient signal for an LLM to reconstruct a usable vulnerability description.
    Core premise of the problem statement in Section IV; the motivating example shows it can work, but the general sufficiency is unproven.
  • domain assumption SecBench.VFC.js is representative of patch-only JavaScript vulnerabilities.
    The benchmark is filtered from SecBench.js by requiring source-changing fix commits and deduplication (Section VI-B1); success rates may not transfer to other samples.
  • ad hoc to paper The eight vulnerability-context dimensions adequately capture the information needed to diagnose exploit-generation failures.
    Table I defines eight hand-designed dimensions; no ablation isolates each dimension or the rubric from the effect of simply spending more LLM calls.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PoCEvolve: Generating Proof-of-Concept Exploits from Security Patches with Vulnerability-Aware Prompt Evolution." pith.science (2026). https://pith.science/paper/EPQTJNYG

@misc{pith2026260722076,
  author       = {Pith},
  title        = {Pith review of: PoCEvolve: Generating Proof-of-Concept Exploits from Security Patches with Vulnerability-Aware Prompt Evolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EPQTJNYG}},
  note         = {Machine review of arXiv:2607.22076}
}
read the original abstract

Ideally, the detailed information about a vulnerability should be made available together with the fixing commit. In practice, however, such details often become available only long after the commit, even when a CVE has already been published. During this window, the patch is already public, so attackers can reverse-engineer it, yet defenders lack the details needed to assess exposure, prioritize, and validate the fix. Executable evidence, such as a proof-of-concept (PoC) exploit, could fill this gap. Prior work has automated PoC generation, but the state-of-the-art approach, PoCGen, assumes that a detailed vulnerability report is already available, which is precisely what is missing during this window. In this paper, we first present an empirical study quantifying the long delay between the fixing commit and the availability of a detailed vulnerability report. We then introduce PoCEvolve, a vulnerability-aware prompt-evolution framework that generates PoCs directly from vulnerability-fixing commits. Given a vulnerability-fixing commit, PoCEvolve synthesizes a corresponding PoC exploit. To learn from unsuccessful generation attempts, PoCEvolve assesses the usefulness of different dimensions of vulnerability-related context, including the inferred vulnerable API and code-coverage information. These assessments guide prompt evolution towards more effective exploit-generation prompts. We evaluate PoCEvolve on SecBench.VFC.js, where PoCEvolve achieves a PoC generation success rate of 58.4%, corresponding to relative improvements of 20.7% over PoCGen and 200.0% over the LLM baseline with GPT-4o-mini. With a recent model, Qwen3.7-Plus, PoCEvolve achieves a higher success rate of 85.3%. When detailed vulnerability reports are available, PoCEvolve achieves a success rate of 71.7%, improving over PoCGen by 11.1%.

Figures

Figures reproduced from arXiv: 2607.22076 by the authors.

Figure 1
Figure 1. presents an overview of POCEVOLVE, which operates in three phases. In Phase 1, given a vulnerability￾fixing commit and accessible repository context, the Commit Analyzer reconstructs a vulnerability report containing the vulnerability type, vulnerable API, and a natural-language description of the vulnerability (Section V-A). No ground￾truth vulnerability report is provided in this phase. In Phase 2, the Exploit Gen… view at source ↗
Figure 2
Figure 2. Overview of the Prompt Evolver. Starting from failed [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. PoC exploit generation success rates by vulnerability type using GPT-4o-mini (left panel) and Qwen3.7-Plus (right panel) [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Experimental results by vulnerability type in the [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 1 canonical work pages

  1. [1]

    What are cves? understanding common vul- nerabilities,

    Cymulate, “What are cves? understanding common vul- nerabilities,” https://cymulate.com/cybersecurity-glossary/ common-vulnerabilities-and-exposures-cve/, accessed: 2026-07-02

    2026

  2. [2]

    A large-scale empirical study of security patches,

    F. Li and V . Paxson, “A large-scale empirical study of security patches,” inProceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, 2017, pp. 2201–2215

    2017

  3. [3]

    Detecting

    X. Wang, K. Sun, A. Batcheller, and S. Jajodia, “Detecting” 0-day” vulnerability: An empirical study of secret security patch in oss,” in 2019 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN). IEEE, 2019, pp. 485–492

    2019

  4. [4]

    Finding a needle in a haystack: Automated mining of silent vulnerability fixes,

    J. Zhou, M. Pacheco, Z. Wan, X. Xia, D. Lo, Y . Wang, and A. E. Hassan, “Finding a needle in a haystack: Automated mining of silent vulnerability fixes,” in2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2021, pp. 705–716

    2021

  5. [5]

    What lies beneath: an empirical study of silent vulnerability fixes in open-source software,

    J. Dong, X. Chen, W. Susilo, N. Sun, A. Shaghaghi, and S. Ma, “What lies beneath: an empirical study of silent vulnerability fixes in open-source software,” in2025 55th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN). IEEE, 2025, pp. 345–357

    2025

  6. [6]

    Automatic patch- based exploit generation is possible: Techniques and implications,

    D. Brumley, P. Poosankam, D. Song, and J. Zheng, “Automatic patch- based exploit generation is possible: Techniques and implications,” in Proceedings of the 2008 IEEE Symposium on Security and Privacy, ser. SP ’08. USA: IEEE Computer Society, 2008, p. 143–157. [Online]. Available: https://doi.org/10.1109/SP.2008.17

    doi:10.1109/sp.2008.17 2008

  7. [7]

    Scaling javascript abstract interpretation to detect and exploit node. js taint-style vulnerability,

    M. Kang, Y . Xu, S. Li, R. Gjomemo, J. Hou, V . Venkatakrishnan, and Y . Cao, “Scaling javascript abstract interpretation to detect and exploit node. js taint-style vulnerability,” in2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2023, pp. 1059–1076

    2023

  8. [8]

    Automated exploit generation for node. js packages,

    F. Marques, M. Ferreira, A. Nascimento, M. E. Coimbra, N. Santos, L. Jia, and J. Fragoso Santos, “Automated exploit generation for node. js packages,”Proceedings of the ACM on Programming Languages, vol. 9, no. PLDI, pp. 1341–1366, 2025

    2025

Show all 63 references
  1. [9]

    Nodemedic- fine: Automatic detection and exploit synthesis for node. js vulnerabili- ties

    D. Cassel, N. Sabino, M.-C. Hsu, R. Martins, and L. Jia, “Nodemedic- fine: Automatic detection and exploit synthesis for node. js vulnerabili- ties.” inNDSS, 2025

    2025

  2. [10]

    Pocgen: Generating proof- of-concept exploits for vulnerabilities in npm packages,

    D. Simsek, A. Eghbali, and M. Pradel, “Pocgen: Generating proof- of-concept exploits for vulnerabilities in npm packages,”Proc. ACM Softw. Eng., vol. 3, no. FSE, Jun. 2026. [Online]. Available: https://doi.org/10.1145/3808178

    2026 doi

  3. [11]

    Code change intention, development artifact, and history vulnerability: Putting them together for vulnerability fix detection by llm,

    X. Yang, W. Zhu, M. Pacheco, J. Zhou, S. Wang, X. Hu, and K. Liu, “Code change intention, development artifact, and history vulnerability: Putting them together for vulnerability fix detection by llm,”Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 489–510, 2025

    2025

  4. [12]

    Not every patch is an island: Llm-enhanced identification of multiple vulnerability patches,

    Y . Song, D. Xie, L. Xu, H. Zhang, C. Zhou, and X. Xie, “Not every patch is an island: Llm-enhanced identification of multiple vulnerability patches,” in2025 40th IEEE/ACM International Conference on Auto- mated Software Engineering (ASE). IEEE, 2025, pp. 996–1007

    2025

  5. [13]

    CVE List V5,

    CVE Program, “CVE List V5,” https://github.com/CVEProject/ cvelistV5, official repository of CVE records in CVE JSON 5 format. Accessed: 2026-07-02

    2026

  6. [14]

    Large language models are human-level prompt engineers,

    Y . Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba, “Large language models are human-level prompt engineers,” inThe Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=92gvk82DE-

    2023

  7. [15]

    Promptbreeder: Self-referential self-improvement via prompt evolution,

    C. Fernando, D. S. Banarse, H. Michalewski, S. Osindero, and T. Rockt ¨aschel, “Promptbreeder: Self-referential self-improvement via prompt evolution,” inProceedings of the 41st International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, R. Sal...

    2024

  8. [16]

    Self-refine: Iter- ative refinement with self-feedback,

    A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y . Yanget al., “Self-refine: Iter- ative refinement with self-feedback,”Advances in neural information processing systems, vol. 36, pp. 46 534–46 594, 2023

    2023

  9. [17]

    Re- flexion: Language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Re- flexion: Language agents with verbal reinforcement learning,”Advances in neural information processing systems, vol. 36, pp. 8634–8652, 2023

    2023

  10. [18]

    Automatic prompt optimization with “gradient descent

    R. Pryzant, D. Iter, J. Li, Y . Lee, C. Zhu, and M. Zeng, “Automatic prompt optimization with “gradient descent” and beam search,” in Proceedings of the 2023 conference on empirical methods in natural language processing, 2023, pp. 7957–7968

    2023

  11. [19]

    Critic: Large language models can self-correct with tool-interactive critiquing,

    Z. Gou, Z. Shao, Y . Gong, Y . Yang, N. Duan, W. Chenet al., “Critic: Large language models can self-correct with tool-interactive critiquing,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 57 734–57 811

    2024

  12. [20]

    Large language models as optimizers,

    C. Yang, X. Wang, Y . Lu, H. Liu, Q. V . Le, D. Zhou, and X. Chen, “Large language models as optimizers,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 12 028–12 068

    2024

  13. [21]

    GEPA: Reflective prompt evolution can outperform reinforcement learning,

    L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl- Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, C. Potts, K. Sen, A. Dimakis, I. Stoica, D. Klein, M. Zaharia, and O. Khattab, “GEPA: Reflective prompt evolution can outperform reinforcement learning,” inThe ...

    2026

  14. [22]

    dirty-json,

    npm, Inc., “dirty-json,” https://www.npmjs.com/package/dirty-json, 2026, npm package page. Accessed: 2026-05-24

    2026

  15. [23]

    dirty-json Vulnerability: Prototype Pollution,

    Snyk, “dirty-json Vulnerability: Prototype Pollution,” https: //security.snyk.io/vuln/npm:dirty-json:20180213, 2018, snyk Vulnerability Database. Accessed: 2026-05-24

    2018

  16. [24]

    SSPCatcher: Learning to catch security patches,

    A. D. Sawadogo, T. F. Bissyand ´e, N. Moha, K. Allix, J. Klein, L. Li, and Y . L. Traon, “SSPCatcher: Learning to catch security patches,”Empirical Software Engineering, vol. 27, no. 6, p. 151, 2022

    2022

  17. [25]

    VulCurator: A vulnerability-fixing commit detector,

    T. G. Nguyen, T. Le-Cong, H. J. Kang, X.-B. D. Le, and D. Lo, “VulCurator: A vulnerability-fixing commit detector,” inProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 1726–1730. 14

    2022

  18. [26]

    SPI: Automated identification of security patches via commits,

    Y . Zhou, J. K. Siow, C. Wang, S. Liu, and Y . Liu, “SPI: Automated identification of security patches via commits,”ACM Transactions on Software Engineering and Methodology, vol. 31, no. 1, pp. 13:1–13:27, 2022

    2022

  19. [27]

    Semfuzz: Semantics-based automatic generation of proof-of-concept exploits,

    W. You, P. Zong, K. Chen, X. Wang, X. Liao, P. Bian, and B. Liang, “Semfuzz: Semantics-based automatic generation of proof-of-concept exploits,” inProceedings of the 2017 ACM SIGSAC conference on computer and communications security, 2017, pp. 2139–2154

    2017

  20. [28]

    Exploiting library vulnerability via migration based automating test generation,

    Z. Chen, X. Hu, X. Xia, Y . Gao, T. Xu, D. Lo, and X. Yang, “Exploiting library vulnerability via migration based automating test generation,” inProceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1–12

    2024

  21. [29]

    Test mimicry to assess the exploitability of library vulnerabilities,

    H. J. Kang, T. G. Nguyen, B. Le, C. S. P ˘as˘areanu, and D. Lo, “Test mimicry to assess the exploitability of library vulnerabilities,” in Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, 2022, pp. 276–288

    2022

  22. [30]

    {APPATCH}: Automated adaptive prompting large language models for{Real-World} software vulnerability patching,

    Y . Nong, H. Yang, L. Cheng, H. Hu, and H. Cai, “{APPATCH}: Automated adaptive prompting large language models for{Real-World} software vulnerability patching,” in34th USENIX Security Symposium (USENIX Security 25), 2025, pp. 4481–4500

    2025

  23. [31]

    Docter: Documentation-guided fuzzing for testing deep learning api functions,

    D. Xie, Y . Li, M. Kim, H. V . Pham, L. Tan, X. Zhang, and M. W. God- frey, “Docter: Documentation-guided fuzzing for testing deep learning api functions,” inProceedings of the 31st ACM SIGSOFT international symposium on software testing and analysis, 2022, pp. 176–188

    2022

  24. [32]

    {CarpetFuzz}: Automatic program option constraint extraction from documentation for fuzzing,

    D. Wang, Y . Li, Z. Zhang, and K. Chen, “{CarpetFuzz}: Automatic program option constraint extraction from documentation for fuzzing,” in32nd USENIX Security Symposium (USENIX Security 23), 2023, pp. 1919–1936

    2023

  25. [33]

    Learning from the past: Real-world exploit migration for smart contract poc generation,

    K. Sun, Z. Xu, K. Li, L. Zhang, Y . Feng, D. Wu, and Y . Liu, “Learning from the past: Real-world exploit migration for smart contract poc generation,” in2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2025, pp. 52–64

    2025

  26. [34]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B...

    2020

  27. [35]

    {ChainFuzz}: Exploiting upstream vulnerabilities in{Open-Source}supply chains,

    P. Deng, L. Zhang, Y . Meng, Z. Yang, and Y . Zhang, “{ChainFuzz}: Exploiting upstream vulnerabilities in{Open-Source}supply chains,” in34th USENIX Security Symposium (USENIX Security 25), 2025, pp. 6199–6218

    2025

  28. [36]

    {NA VEX}: Precise and scalable exploit generation for dynamic web applications,

    A. Alhuzali, R. Gjomemo, B. Eshete, and V . Venkatakrishnan, “{NA VEX}: Precise and scalable exploit generation for dynamic web applications,” in27th USENIX Security Symposium (USENIX Security 18), 2018, pp. 377–392

    2018

  29. [37]

    Revery: From proof-of-concept to exploitable,

    Y . Wang, C. Zhang, X. Xiang, Z. Zhao, W. Li, X. Gong, B. Liu, K. Chen, and W. Zou, “Revery: From proof-of-concept to exploitable,” inProceedings of the 2018 ACM SIGSAC conference on computer and communications security, 2018, pp. 1914–1927

    2018

  30. [38]

    1dfuzz: Reproduce 1-day vulnerabilities with directed differ- ential fuzzing,

    S. Yang, Y . He, K. Chen, Z. Ma, X. Luo, Y . Xie, J. Chen, and C. Zhang, “1dfuzz: Reproduce 1-day vulnerabilities with directed differ- ential fuzzing,” inProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2023, pp. 867–879

    2023

  31. [39]

    Bullseye: Detecting prototype pollution in npm packages with proof of concept exploits,

    T. Houis, S. Jiang, M. Mannan, and A. Youssef, “Bullseye: Detecting prototype pollution in npm packages with proof of concept exploits,” in Network and Distributed System Security Symposium (NDSS’26), San Diego, CA, USA, 2026

    2026

  32. [40]

    Secbench. js: An executable security benchmark suite for server-side javascript,

    M. H. M. Bhuiyan, A. S. Parthasarathy, N. Vasilakis, M. Pradel, and C.-A. Staicu, “Secbench. js: An executable security benchmark suite for server-side javascript,” in2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 1059– 1070

    2023

  33. [41]

    Prompt engineering in llms for automated unit test generation: A large-scale study,

    W. C. Ou ´edraogo, A. K. Kabor ´e, Y . Li, H. Tian, A. Koyuncu, J. Klein, D. Lo, and T. F. Bissyand ´e, “Prompt engineering in llms for automated unit test generation: A large-scale study,”Empirical Software Engineer- ing, vol. 31, no. 4, p. 103, 2026

    2026

  34. [42]

    Large language model for vulnerability detection: Emerging results and future directions,

    X. Zhou, T. Zhang, and D. Lo, “Large language model for vulnerability detection: Emerging results and future directions,” inProceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineer- ing: New Ideas and Emerging Results, 2024, pp. 47–51

    2024

  35. [43]

    Examining zero-shot vulnerability repair with large language models,

    H. Pearce, B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt, “Examining zero-shot vulnerability repair with large language models,” in2023 IEEE symposium on security and privacy (SP). IEEE, 2023, pp. 2339–2356

    2023

  36. [44]

    Regular expression denial of service (ReDoS) in codemir- ror,

    Snyk, “Regular expression denial of service (ReDoS) in codemir- ror,” https://security.snyk.io/vuln/SNYK-JS-CODEMIRROR-1016937, Oct. 2020, snyk ID: SNYK-JS-CODEMIRROR-1016937; CVE-2020- 7760; accessed 14 July 2026

    2020

  37. [45]

    Prototype Pollution in immer,

    ——, “Prototype Pollution in immer,” https://security.snyk.io/vuln/ SNYK-JS-IMMER-1540542, snyk ID: SNYK-JS-IMMER-1540542; CVE-2021-23436; published 1 Sept. 2021; disclosed 12 Aug. 2021; accessed 6 Jul. 2026

    2021

  38. [46]

    Just-in-time defect prediction on javascript projects: A replication study,

    C. Ni, X. Xia, D. Lo, X. Yang, and A. E. Hassan, “Just-in-time defect prediction on javascript projects: A replication study,”ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 31, no. 4, pp. 1–38, 2022

    2022

  39. [47]

    Jit-smart: A multi-task learning framework for just-in-time defect prediction and localization,

    X. Chen, F. Xu, Y . Huang, N. Zhang, and Z. Zheng, “Jit-smart: A multi-task learning framework for just-in-time defect prediction and localization,”Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, pp. 1–23, 2024

    2024

  40. [48]

    Bridging expert knowledge with deep learning techniques for just-in-time defect prediction,

    X. Zhou, D. Han, and D. Lo, “Bridging expert knowledge with deep learning techniques for just-in-time defect prediction,”Empirical Soft- ware Engineering, vol. 30, no. 1, p. 37, 2025

    2025

  41. [49]

    A large-scale empirical study of just-in-time quality assurance,

    Y . Kamei, E. Shihab, B. Adams, A. E. Hassan, A. Mockus, A. Sinha, and N. Ubayashi, “A large-scale empirical study of just-in-time quality assurance,”IEEE Transactions on Software Engineering, vol. 39, no. 6, pp. 757–773, 2012

    2012

  42. [50]

    An empirical study of just-in-time defect prediction using cross-project models,

    T. Fukushima, Y . Kamei, S. McIntosh, K. Yamashita, and N. Ubayashi, “An empirical study of just-in-time defect prediction using cross-project models,” inProceedings of the 11th working conference on mining software repositories, 2014, pp. 172–181

    2014

  43. [51]

    Deep just-in-time defect prediction: how far are we?

    Z. Zeng, Y . Zhang, H. Zhang, and L. Zhang, “Deep just-in-time defect prediction: how far are we?” inProceedings of the 30th ACM SIGSOFT international symposium on software testing and analysis, 2021, pp. 427–438

    2021

  44. [52]

    Cc2vec: Distributed representations of code changes,

    T. Hoang, H. J. Kang, D. Lo, and J. Lawall, “Cc2vec: Distributed representations of code changes,” inProceedings of the ACM/IEEE 42nd international conference on software engineering, 2020, pp. 518–529

    2020

  45. [53]

    Deepjit: an end-to-end deep learning framework for just-in-time defect prediction,

    T. Hoang, H. K. Dam, Y . Kamei, D. Lo, and N. Ubayashi, “Deepjit: an end-to-end deep learning framework for just-in-time defect prediction,” in2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR). IEEE, 2019, pp. 34–45

    2019

  46. [54]

    Toward realistic evaluations of just-in-time vulnerability prediction,

    D. Nguyen, T. Le-Cong, T. H. M. Le, M. A. Babar, and Q.-T. Huynh, “Toward realistic evaluations of just-in-time vulnerability prediction,” in2025 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2025, pp. 1–10

    2025

  47. [55]

    Vccfinder: Finding potential vulnerabilities in open-source projects to assist code audits,

    H. Perl, S. Dechand, M. Smith, D. Arp, F. Yamaguchi, K. Rieck, S. Fahl, and Y . Acar, “Vccfinder: Finding potential vulnerabilities in open-source projects to assist code audits,” inProceedings of the 22nd ACM SIGSAC conference on computer and communications security, 2015, pp...

    2015

  48. [56]

    Fine-grained commit-level vulnerability type prediction by cwe tree structure,

    S. Pan, L. Bao, X. Xia, D. Lo, and S. Li, “Fine-grained commit-level vulnerability type prediction by cwe tree structure,” in2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 957–969

    2023

  49. [57]

    Commit- level, neural vulnerability detection and assessment,

    Y . Li, A. Yadavally, J. Zhang, S. Wang, and T. N. Nguyen, “Commit- level, neural vulnerability detection and assessment,” inProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023, pp. 1024–1036

    2023

  50. [58]

    Deepcva: automated commit-level vulnerability assessment with deep multi-task learning,

    T. H. Minh Le, D. Hin, R. Croft, and M. Ali Babar, “Deepcva: automated commit-level vulnerability assessment with deep multi-task learning,” in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2021, p. 69

    2021

  51. [59]

    Testora: Using natural language intent to detect behavioral regressions,

    M. Pradel, “Testora: Using natural language intent to detect behavioral regressions,”arXiv preprint arXiv:2503.18597, 2025

    2025

  52. [60]

    Pareto prompt optimization,

    G. Zhao, B.-J. Yoon, G. Park, S. Jha, S. Yoo, and X. Qian, “Pareto prompt optimization,” inInternational Conference on Learning Repre- sentations, vol. 2025, 2025, pp. 6770–6785

    2025

  53. [61]

    Metagpt: Meta programming for a multi-agent collaborative framework,

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, J. Wang, C. Zhang, S. Yau, Z. Lin, L. Zhouet al., “Metagpt: Meta programming for a multi-agent collaborative framework,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 23 247–23 275

    2024

  54. [62]

    Dspy: Compiling declarative language model calls into self-improving pipelines,

    O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. Vardhamanan, S. Haq, A. Sharma, T. T. Joshi, H. Moazamet al., “Dspy: Compiling declarative language model calls into self-improving pipelines,”arXiv preprint arXiv:2310.03714, 2023

    2023 arXiv

  55. [63]

    Meta-prompting: Enhancing language mod- els with task-agnostic scaffolding,

    M. Suzgun and A. T. Kalai, “Meta-prompting: Enhancing language mod- els with task-agnostic scaffolding,”arXiv preprint arXiv:2401.12954, 2024

    2024 arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.