Pith. sign in

REVIEW 4 major objections 6 minor 46 references

Vul4Py gives every one of 100 real Python vulnerabilities a paired exploit-and-functional oracle, then shows that software agents repair 41 of them while directly prompted LLMs repair 4.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 00:30 UTC pith:LYJAGMAG

load-bearing objection Vul4Py is a genuinely useful paired-oracle Python repair benchmark, but the abstract oversells the 41/100 headline by omitting that all approaches got exact human-fix file/line localization hints. the 4 major comments →

arxiv 2608.00692 v1 pith:LYJAGMAG submitted 2026-08-01 cs.SE

Vul4Py: Benchmarking Automated Vulnerability Repair in Python with Paired Exploit and Functional Oracles

Tan Bui , Ting Zhang , Ferdian Thung , Yunpeng Xiong , Penghao Jiang , Xin Zhou , David Lo This is my paper
classification cs.SE
keywords automated vulnerability repairbenchmarklarge language modelsLLM agentsPythonexploit oraclefunctional oracleCVE
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Vul4Py is proposed as the first Python benchmark for automated vulnerability repair in which every entry must pass two executable checks: a security exploit test that fails on the vulnerable code and passes on the fixed code, and the project's own functional test suite that must keep passing. The paper argues that existing Python benchmarks accept patches on the strength of an exploit proof-of-concept alone, or apply functional tests only when a project happens to ship one, so they silently credit patches that break unrelated behavior. On 100 real vulnerabilities from 60 projects, the paper finds that agents that can run tests and iterate repair most of the vulnerabilities, whereas one-shot approaches on the same backbone model repair almost none: OpenHands produces 41 manually confirmed correct repairs, versus 4 for the best directly prompted LLM and 2 for a specialized repair tool. The paired oracle is what makes those counts trustworthy: it rejects 15 of the 119 patches an exploit-only oracle would accept, and 94% of the patches it admits are manually judged semantically equivalent to the developer's fix.

Core claim

The paper defines a correct patch operationally rather than by resemblance to the human fix: the exploit oracle (a security test shipped with the fix commit) must fail on the vulnerable revision and pass on the fixed one, while the project's native pytest functional oracle must pass on both. Entries are admitted only when this before-and-after behavior reproduces in a pinned environment, and candidate patches face the same two gates. With a matched budget and the same backbone model, three software agents repaired 22 to 41 of the 100 vulnerabilities, against 4 for the best directly prompted LLM and 2 for the specialized tool. The functional oracle is not redundant: it rejects 15 of 119 explo

What carries the argument

The paired oracle is the load-bearing object: an exploit oracle — a security test taken from the test shipped by the fix commit — that must fail on the vulnerable revision and pass on the fixed one, plus a functional oracle, the project's own non-security pytest suite, that must pass on both. The construction pipeline (advisory from the OSV open-source vulnerability database, linked to exactly one fix commit, keeping only fixes that add or modify tests, manual validation that the test targets the reported flaw, and a pinned reproducible environment) is what lets every entry carry both oracles, which then serve as the benchmark's admission gate and as the acceptance criterion that turns a can

Load-bearing premise

The benchmark's validity rests on the assumption that the test shipped by the fix commit, together with the project's existing test suite, is a sufficient oracle of correct repair, and the reported repair rates apply only to the strongly localized setting in which every approach is handed the human fix's exact files and line ranges.

What would settle it

Re-run the six approaches on Vul4Py with the line-range localization hints removed; if OpenHands' 41 correct repairs collapses toward the 4 of the best directly prompted LLM, the measured agent advantage is an artifact of being handed the fix's location rather than of iterating against test feedback.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Exploit-only evaluation is misleading: within Vul4Py, 15 of 119 patches (12.6%) that defeat the exploit break the project's own functional suite, so benchmarks that check only the exploit credit one in eight incorrect repairs.
  • Agentic iteration, not model quality or specialized prompting, is the deciding factor: on a shared backbone model, agents repaired 22 to 41 of 100 vulnerabilities while direct prompting and a specialized tool managed at most 5.
  • Passing the paired oracle is a high-precision proxy for real repair: 98 of 104 admitted patches (94%) were manually confirmed semantically equivalent to the developer fix, making the benchmark's counts comparable across approaches.
  • Repair coverage comes at a price: agents cost 26 to 150 times as much per instance as direct prompting, so the choice between approaches is a cost-versus-coverage trade-off.
  • Vul4Py enables the first verifiable head-to-head comparison of automated vulnerability repair approaches in Python, spanning a specialized tool, prompted LLMs, and agents.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The requirement that every fix commit ship a test selects for vulnerabilities whose developers wrote regression tests; un-reproducible or test-less fixes are excluded, so the absolute repair rates likely overstate what the same systems would achieve on a random sample of Python CVEs.
  • Every approach received the human fix's exact files and line ranges as localization hints, so the 41/4/2 numbers measure repair with localization already solved; an unlocalized variant would be a harder and arguably more realistic task.
  • The six overfitted admitted patches suggest a cheap extension: add a third held-out oracle, such as tests added by later upstream commits or an independent proof-of-concept, and re-measure precision; the released pipeline makes that directly testable.
  • The same paired-oracle recipe could be applied to fix commits that ship tests in other languages, enabling cross-language comparisons of agentic repair; whether the agent advantage survives on open-weight backbones also remains an open test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Vul4Py, a Python automated vulnerability repair (AVR) benchmark containing 100 real vulnerabilities from 60 open-source projects and 60 CWEs, each packaged with a paired oracle: an exploit test that fails on the vulnerable revision and passes on the fixed one, plus a project-native pytest functional suite that must pass on both revisions. The construction pipeline starts from OSV advisories, links to fix commits, keeps only fixes that add or modify tests, manually validates the oracle's connection to the reported flaw, and requires reproducible before/after behavior. The paper evaluates six approaches—APPATCH, GPT-4o, Claude Sonnet 4, OpenHands, Trae Agent, and SWE-agent—under a shared interaction budget and reports that software engineering agents dominate, with OpenHands producing 41 manually confirmed correct patches versus 4 for the strongest directly prompted LLM and 2 for the specialized tool. The paired oracle is shown to filter out 15 of 119 exploit-passing patches, and 98 of 104 plausible patches are manually judged semantically equivalent to the developer fix.

Significance. If the benchmark and results hold, Vul4Py is a useful contribution: it is the first Python AVR benchmark to require both an exploit oracle and a functional oracle on every instance, it ships reproducible per-instance environments, and its admission gates are transparent and machine-checkable. The empirical comparison is internally well-controlled—all agent and direct-prompting conditions share the Claude Sonnet 4 backbone—and the paired-oracle statistics provide concrete evidence that exploit-only validation is insufficient. However, the headline absolute repair rates are conditioned on giving every approach the exact file/line locations of the human fix, and the benchmark only includes vulnerabilities whose fix commits ship tests; both facts materially affect how the absolute numbers should be interpreted. The manual-correctness labels also rest on moderate inter-rater agreement. These issues are fixable with additional experiments or careful reframing, but they are load-bearing for the paper's central claims.

major comments (4)
  1. [Abstract; §4.2, Listing 1.1; Conclusion] The headline 'OpenHands repairs 41 of 100 vulnerabilities' is reported without disclosing that every approach—including the agents—receives a VULNERABILITY LOCALIZATION HINTS block listing the exact non-test files and line ranges changed by the human fix. This turns the task into a strongly localized patch-within-known-lines problem, not end-to-end AVR. The internal comparison between approaches remains fair, but the absolute rate is an upper bound for a much easier task and may not generalize to settings without localization. Please either run a no-hint condition to quantify the effect, or reframe all absolute claims as 'given perfect localization' and state this prominently in the abstract and conclusion.
  2. [§3.3; §3.4; §6.3] The admission criterion keeps only vulnerabilities whose fix commit adds or modifies a test. This selects a subpopulation that is testable and likely more amenable to test-driven agentic repair than the general population of Python vulnerabilities. The paper does not report the funnel from OSV advisories to the final 100 instances (how many have PoC indicators, link to a unique fix, add/modify tests, pass manual validation, and reproduce). Please provide these counts and discuss the resulting selection bias; without this, the claim of '100 real vulnerabilities' is overbroad and the absolute repair rates may not generalize.
  3. [§4.2; Table 2] The empirical counts appear to be based on a single run per approach per instance. The paper explicitly says APPATCH is run once, and the agent description implies one final patch per instance, but no variance or confidence intervals are reported. Agents are stochastic even at temperature 0—tool execution order, shell output, and environment state can vary—so the exact counts (41, 29, 22, 4, 2, 0) are uncertain. At minimum, report repeated runs on a subset of instances, or explicitly state that all results are single-run and discuss the expected variability. This is load-bearing for the quantitative comparison.
  4. [§4.3; Table 2] Manual correctness labels are produced by two authors with a pre-discussion κ of 0.67, and disagreements are resolved by discussion. The paper reports only the resolved labels (98 of 104 plausible patches correct). Given moderate inter-rater agreement, the correctness column is substantially less reliable than the oracle-based columns. Please report per-reviewer counts and the number/type of disagreements, and state how discussion resolved them. If the disagreement rate is concentrated in specific instances, consider a third reviewer or an adjudication protocol, or report a confidence range for the correctness counts.
minor comments (6)
  1. [Listing 1.1] The placeholders 'modify_or_remove lines' and 'add lines' are not defined. Clarify the exact format of line ranges and whether they are 1-indexed and inclusive.
  2. [§4.2] The number of runs per instance is only explicit for APPATCH ('once'). Please state the run count for each LLM and agent condition, and whether any runs were discarded due to infrastructure failures.
  3. [§5.1] The text says 'all six approaches build on the same class of backbone model,' but GPT-4o is a different model family and APPATCH is a port of a C/C++ tool. This is imprecise; rephrase to 'the same backbone model where applicable' or clarify the comparison.
  4. [Table 2] Mean cost and runtime are reported; given the Trae Agent outlier (mean spend $7.52), also report medians to show skew.
  5. [§5.2] The phrase 'the functional half of the paired oracle removes 15 patches' is accurate, but calling them 'rejected' in the abstract could imply they are incorrect. Some of the 15 may be correct patches that fail brittle tests, as the paper acknowledges in §5.2. Use 'filtered out' consistently and quantify how many of the 15 were manually inspected.
  6. [§3.6] The patch-size discussion notes one fix changes no .py file, but the metric is defined as 'non-test .py lines changed.' Explain how that instance is handled in the distribution.

Circularity Check

0 steps flagged

No significant circularity: benchmark construction and empirical results are anchored to external artifacts (reproducible oracle behavior, manual semantic review) rather than to fitted values or a self-citation chain.

full rationale

Vul4Py's central claims—that the paired oracle filters functional regressions and that agents repair more vulnerabilities—are empirical measurements, not derivations from the benchmark's own inputs. The oracle is defined by the upstream fix commit's shipped test plus the project's native functional suite; candidate patches are evaluated by executing them against these external tests. The 15-rejection and 94%-precision findings are checked by manual inspection against developer patches, an independent (if author-judged) criterion. The paper's self-citations (e.g., [15], [35], [37], [38], [42], [44], [45], [46]) appear only in background/related work and do not carry the benchmark's validity. The explicit localization hints and the requirement that fixes ship tests are threats to external validity/construct validity, but they are not circular: the evaluation does not reduce to a fitted parameter or to a self-citation. The admitted limitations (kappa=0.67, 6 overfitted patches, possible training-data leakage) are validity threats, not logical circularity. Thus no step in the derivation chain is equivalent to its input by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

No numeric parameters are fitted; the empirical counts are measured outcomes, and design choices such as step limits, cost caps, and the hint protocol are policy variables. The central claim rests on the five domain assumptions listed above, especially the sufficiency of the paired oracle and the manual validation of oracle-vulnerability correspondence. No new conceptual entities such as particles, forces, or conserved quantities are introduced.

axioms (5)
  • domain assumption OSV advisories carrying a PoC indicator and linked to exactly one fix commit are a valid source population for reproducible Python vulnerabilities.
    Used in Sections 3.1 and 3.2 to filter the initial pool; it restricts the benchmark to vulnerabilities with public reproducers and unambiguous fixes.
  • domain assumption A fix commit that adds or modifies a test whose assertions match the advisory provides a developer-authored, vulnerability-revealing oracle.
    Sections 3.3 and 3.4. This is the heart of the construction; manual review is used to verify the correspondence.
  • domain assumption Passing the project's existing functional test suite on both vulnerable and fixed revisions is an adequate proxy for absence of functional regression.
    Sections 3.5 and 5.2. The paper itself notes 6 admitted patches omit conditions that the human fix handles but tests do not cover, so the proxy is imperfect.
  • domain assumption Manual validation by two authors captures whether a test reveals the reported vulnerability and whether a patch is semantically equivalent to the developer fix.
    Sections 3.4 and 4.3. Inter-rater agreement before discussion was kappa 0.67, a moderate level.
  • ad hoc to paper Providing all LLM approaches with the exact non-test files and line ranges of the human fix is a fair way to compare process types.
    Listing 1.1 and Section 4.2. The hints equalize localization effort but make the headline repair counts conditional on strong localization, a design choice specific to this study.

pith-pipeline@v1.3.0-alltime-deepseek · 12984 in / 11514 out tokens · 122750 ms · 2026-08-05T00:30:19.767769+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Vul4Py: Benchmarking Automated Vulnerability Repair in Python with Paired Exploit and Functional Oracles." pith.science (2026). https://pith.science/paper/LYJAGMAG

@misc{pith2026260800692,
  author       = {Pith},
  title        = {Pith review of: Vul4Py: Benchmarking Automated Vulnerability Repair in Python with Paired Exploit and Functional Oracles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LYJAGMAG}},
  note         = {Machine review of arXiv:2608.00692}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Automated Vulnerability Repair (AVR) has advanced rapidly across program analysis, machine learning, and Large Language Models (LLMs), but a verifiable, head-to-head comparison of AVR approaches on Python is still missing. Python underpins critical web, data, and machine-learning infrastructure, yet existing Python benchmarks accept a patch on the strength of a proof-of-concept exploit alone, or apply a functional test only on the subset of entries whose upstream project happens to ship one. Both therefore miss functional regressions, in which a patch defeats the exploit but breaks unrelated behavior. We present Vul4Py, a Python AVR benchmark in which every entry carries a paired oracle: an exploit oracle that must fail on the vulnerable revision and pass on the fixed one, together with a project-native pytest functional oracle that must pass on both. Vul4Py comprises 100 real vulnerabilities from 60 open-source projects, spanning 60 distinct CWEs and the years 2017 to 2025, each packaged with a pinned, reproducible per-instance environment. Using Vul4Py, we compare six approaches in three categories: a specialized vulnerability repair tool, directly prompted LLMs, and software engineering agents. The agents dominate: OpenHands repairs 41 of 100 vulnerabilities, against 4 for the strongest directly prompted LLM and 2 for the specialized tool, despite all three sharing the same backbone model. The paired oracle is what makes these counts trustworthy: it rejects 15 of the 119 patches that an exploit-only oracle would accept, and 98 of the 104 patches it admits are manually confirmed to be semantically equivalent to the developer's patches

Figures

Figures reproduced from arXiv: 2608.00692 by David Lo, Ferdian Thung, Penghao Jiang, Tan Bui, Ting Zhang, Xin Zhou, Yunpeng Xiong.

Figure 1
Figure 1. Figure 1: The Vul4Py construction pipeline. Open-source advisories from OSV are linked to fix commits, filtered to fixes that add or modify tests, manually vali￾dated through Manual Oracle Validation, and packaged into reproducible per-instance workspaces, each carrying a paired functional and exploit oracle. 3 Constructing Vul4Py We construct Vul4Py from real Python vulnerability reports with executable pytest-base… view at source ↗
Figure 2
Figure 2. Figure 2: Overall characteristics of the 100 Vul4Py instances: (a) distribution by CVE year, (b) the Python version provisioned for each instance, and (c) patch size measured as .py lines changed by the upstream fix. than the 100 instances. The top categories by count are CWE-770, Allocation of Resources Without Limits, CWE-400, Uncontrolled Resource Consumption, and CWE-20, Improper Input Validation, each with 8 in… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 29 canonical work pages · 1 internal anchor

  1. [1]

    Empirical Softw

    Alfadel, M., Costa, D.E., Shihab, E.: Empirical analysis of security vulnerabilities in python packages. Empirical Softw. Engg.28(3) (Mar 2023).https://doi.org/ 10.1007/s10664-022-10278-4

    doi:10.1007/s10664-022-10278-4 2023

  2. [2]

    anthropic.com/news/model-context-protocol(2024), accessed: 2026-06-02

    Anthropic: Introducing the model context protocol (mcp).https://www. anthropic.com/news/model-context-protocol(2024), accessed: 2026-06-02

    work page 2024

  3. [3]

    In: Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering

    Bhandari, G., Naseer, A., Moonen, L.: Cvefixes: automated collection of vulner- abilities and their fixes from open-source software. In: Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering. pp. 30–39 (2021)

    work page 2021

  4. [4]

    IEEE Transactions on Software Engineering49(1), 147–165 (2022)

    Chen, Z., Kommrusch, S., Monperrus, M.: Neural transfer learning for repairing se- curity vulnerabilities in c code. IEEE Transactions on Software Engineering49(1), 147–165 (2022)

    work page 2022

  5. [5]

    In: Proceedings of the 17th international confer- ence on mining software repositories

    Fan, J., Li, Y., Wang, S., Nguyen, T.N.: A c/c++ code vulnerability dataset with code changes and cve summaries. In: Proceedings of the 17th international confer- ence on mining software repositories. pp. 508–512 (2020)

    work page 2020

  6. [6]

    arXiv preprint arXiv:2002.08155 (2020)

    Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al.: Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020)

    Pith/arXiv arXiv 2002

  7. [7]

    In: Proceedings of the 30th ACM joint european software engineering conference and symposium on the foundations of software engineering

    Fu, M., Tantithamthavorn, C., Le, T., Nguyen, V., Phung, D.: Vulrepair: a t5- based automated software vulnerability repair. In: Proceedings of the 30th ACM joint european software engineering conference and symposium on the foundations of software engineering. pp. 935–947 (2022)

    work page 2022

  8. [8]

    In: Pro- ceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis

    Gao, X., Mechtaev, S., Roychoudhury, A.: Crash-avoiding program repair. In: Pro- ceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. pp. 8–18 (2019)

    work page 2019

  9. [9]

    ACM Transactions on Software Engineering and Methodology (TOSEM)30(2), 1–27 (2021)

    Gao, X., Wang, B., Duck, G.J., Ji, R., Xiong, Y., Roychoudhury, A.: Beyond tests: Program vulnerability repair via crash constraint extraction. ACM Transactions on Software Engineering and Methodology (TOSEM)30(2), 1–27 (2021)

    work page 2021

  10. [10]

    GitHub: Octoverse 2024: AI Leads Python to the Top Language as the Number of Global Developers Surges.https://github.blog/news-insights/octoverse/ octoverse-2024/(2024), accessed: 2026-06-02 Vul4Py 17

    work page 2024

  11. [11]

    Google: OSV: Open Source Vulnerabilities.https://osv.dev(2024), accessed: 2026-06-02

    work page 2024

  12. [12]

    Hu, Y., Liu, Z., Shu, K., Guan, S., Zou, D., Xu, S., Yuan, B., Jin, H.:{SoK}: Automatedvulnerabilityrepair:Methods,tools,andassessments.In:34thUSENIX Security Symposium (USENIX Security 25). pp. 4421–4440 (2025)

    work page 2025

  13. [13]

    Computer Standards & Interfaces92, 103917 (2025)

    Husein, R.A., Aburajouh, H., Catal, C.: Large language models for code comple- tion: A systematic literature review. Computer Standards & Interfaces92, 103917 (2025)

    work page 2025

  14. [14]

    Ieee transactions on software engineering38(1), 54–72 (2011)

    Le Goues, C., Nguyen, T., Forrest, S., Weimer, W.: Genprog: A generic method for automatic software repair. Ieee transactions on software engineering38(1), 54–72 (2011)

    work page 2011

  15. [15]

    Li, Y., Bui, N.T., Zhang, T., Yang, C., Zhou, X., Weyssow, M., Jiang, J., Chen, J., Huang, H., Nguyen, H.H., et al.: Out of distribution, out of luck: How well can llms trained on vulnerability datasets detect top 25 cwe weaknesses? arXiv preprint arXiv:2507.21817 (2025)

    arXiv 2025

  16. [16]

    In: 34th USENIX Security Symposium (USENIX Security 25)

    Li, Y., Hossain Shezan, F., Wei, B., Wang, G., Tian, Y.: SoK: Towards effective automated vulnerability repair. In: 34th USENIX Security Symposium (USENIX Security 25). pp. 4441–4462 (2025)

    work page 2025

  17. [17]

    Measuring the Exploitation of Weaknesses in the Wild

    Mell, P., Bojanova, I., Galhardo, C.: Measuring the exploitation of weaknesses in the wild. arXiv preprint arXiv:2405.01289 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  18. [18]

    In: 34th USENIX Security Symposium (USENIX Security 25)

    Nong, Y., Yang, H., Cheng, L., Hu, H., Cai, H.: APPATCH: Automated adap- tive prompting large language models for Real-World software vulnerability patch- ing. In: 34th USENIX Security Symposium (USENIX Security 25). pp. 4481–4500 (2025)

    work page 2025

  19. [19]

    arXiv preprint arXiv:2506.10204 (2025)

    Paleyes, A., Sendyka, R., Robinson, D., Cabrera, C., Lawrence, N.D.: Prompt variability effects on llm code generation. arXiv preprint arXiv:2506.10204 (2025)

    arXiv 2025

  20. [20]

    In: 2023 IEEE Symposium on Security and Privacy (SP)

    Pearce, H., Tan, B., Ahmad, B., Karri, R., Dolan-Gavitt, B.: Examining zero-shot vulnerability repair with large language models. In: 2023 IEEE Symposium on Security and Privacy (SP). pp. 2339–2356. IEEE (2023)

    work page 2023

  21. [21]

    Journal of machine learning research21(140), 1–67 (2020)

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research21(140), 1–67 (2020)

    2020

  22. [22]

    In: 33rd USENIX Security Symposium (USENIX Security

    Risse, N., Böhme, M.: Uncovering the limits of machine learning for automatic vulnerability detection. In: 33rd USENIX Security Symposium (USENIX Security

    work page

  23. [23]

    Advances in Neural Information Processing Systems36, 68539–68551 (2023)

    Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Hambro, E., Zettle- moyer, L., Cancedda, N., Scialom, T.: Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems36, 68539–68551 (2023)

    work page 2023

  24. [24]

    ACM Transactions on Software Engineering and Methodology34(4), 1–27 (2025)

    Shariffdeen, R., Timperley, C.S., Noller, Y., Le Goues, C., Roychoudhury, A.: Vul- nerability repair via concolic execution and code mutations. ACM Transactions on Software Engineering and Methodology34(4), 1–27 (2025)

    work page 2025

  25. [25]

    Advances in Neural Information Processing Systems36, 8634–8652 (2023)

    Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., Yao, S.: Reflexion: Lan- guage agents with verbal reinforcement learning. Advances in Neural Information Processing Systems36, 8634–8652 (2023)

    2023

  26. [26]

    stackoverflow.co/2025/(2025), accessed: 2026-06-02

    Stack Overflow: Stack Overflow Developer Survey 2025.https://survey. stackoverflow.co/2025/(2025), accessed: 2026-06-02

    work page 2025

  27. [27]

    arXiv preprint arXiv:2407.07959 (2024) 18 T

    Sun, W., Miao, Y., Li, Y., Zhang, H., Fang, C., Liu, Y., Deng, G., Liu, Y., Chen, Z.: Source code summarization in the era of large language models. arXiv preprint arXiv:2407.07959 (2024) 18 T. Bui et al

    Pith/arXiv arXiv 2024

  28. [28]

    Team, T.R., Gao, P., Tian, Z., Meng, X., Wang, X., Hu, R., Xiao, Y., Liu, Y., Zhang, Z., Chen, J., et al.: Trae agent: An llm-based agent for software engineering with test-time scaling.https://arxiv.org/abs/2507.23370(2025)

    Pith/arXiv arXiv 2025

  29. [29]

    TIOBE Software BV: TIOBE Index.https://www.tiobe.com/tiobe-index/ (2025), accessed: 2026-06-02

    work page 2025

  30. [30]

    In: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)

    Wang, P., Liu, X., Xiao, C.: Cve-bench: Benchmarking llm-based software engi- neering agent’s ability to repair real-world cve vulnerabilities. In: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). pp. 4207–4224 (2025)

    work page 2025

  31. [31]

    arXiv preprint arXiv:2509.03331 (2025)

    Wang, W., Ma, W., Hu, Q., Zhang, Y., Sun, J., Wu, B., Liu, Y., Xu, G., Jiang, L.: VulnRepairEval: An exploit-based evaluation framework for assessing large language model vulnerability repair capabilities. arXiv preprint arXiv:2509.03331 (2025)

    Pith/arXiv arXiv 2025

  32. [32]

    In: DSN (2021)

    Wang, X., Wang, S., Feng, P., Sun, K., Jajodia, S.: PatchDB: A large-scale security patch dataset. In: DSN (2021)

    work page 2021

  33. [33]

    arXiv preprint arXiv:2407.16741 (2024)

    Wang, X., Li, B., Song, Y., Xu, F.F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., et al.: Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741 (2024)

    Pith/arXiv arXiv 2024

  34. [34]

    arXiv preprint arXiv:2511.11019 (2025)

    Wei, Z., Zeng, J., Wen, M., Yu, Z., Cheng, K., Zhu, Y., Guo, J., Zhou, S., Yin, L., Su, X., Ma, Z.: PATCHEVAL: A new benchmark for evaluating LLMs on patching real-world vulnerabilities. arXiv preprint arXiv:2511.11019 (2025)

    arXiv 2025

  35. [35]

    arXiv preprint arXiv:2504.04699 (2025)

    Weyssow, M., Yang, C., Chen, J., Widyasari, R., Zhang, T., Huang, H., Nguyen, H.H., Tun, Y.N., Bui, T., Li, Y., et al.: R2vul: Learning to reason about software vulnerabilities with reinforcement learning and structured reasoning distillation. arXiv preprint arXiv:2504.04699 (2025)

    Pith/arXiv arXiv 2025

  36. [36]

    In: Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering

    Widyasari, R., Sim, S.Q., Lok, C., Qi, H., Phan, J., Tay, Q., Tan, C., Wee, F., Tan, J.E., Yieh, Y., et al.: Bugsinpy: a database of existing bugs in python programs to enable controlled testing and debugging studies. In: Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engi...

    work page 2020

  37. [37]

    arXiv preprint arXiv:2601.22952 (2026)

    Xiong, Y., Zhang, T.: Sifting the noise: A comparative study of llm agents in vulnerability false positive filtering. arXiv preprint arXiv:2601.22952 (2026)

    Pith/arXiv arXiv 2026

  38. [38]

    In: Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Yang, C., Zhang, T., Jiang, J., Zhou, X., Tian, H., Du, M., Shi, J., Chen, J., Li, Y., Ouh, E.L., et al.: Securepair: Semantics-aligned, curriculum-driven, and reasoning-enhanced vulnerability repair framework. In: Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 32108–32123 (2026)

    work page 2026

  39. [39]

    Advances in Neural Information Processing Systems37, 50528–50652 (2024)

    Yang, J., Jimenez, C.E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., Press, O.: Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems37, 50528–50652 (2024)

    2024

  40. [40]

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K.R., Cao, Y.: React: Synergizingreasoningandactinginlanguagemodels.In:Theeleventhinternational conference on learning representations (2022)

    2022

  41. [41]

    arXiv preprint arXiv:2405.01466 (2024)

    Zhang, Q., Fang, C., Xie, Y., Ma, Y., Sun, W., Yang, Y., Chen, Z.: A systematic literature review on large language models for automated program repair. arXiv preprint arXiv:2405.01466 (2024)

    arXiv 2024

  42. [42]

    ACM Transactions on Software Engineering and Methodology34(3), 1–30 (2025) Vul4Py 19

    Zhang, T., Irsan, I.C., Thung, F., Lo, D.: Revisiting sentiment analysis for software engineering in the era of large language models. ACM Transactions on Software Engineering and Methodology34(3), 1–30 (2025) Vul4Py 19

    work page 2025

  43. [43]

    In: Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis

    Zhang, Y., Gao, X., Duck, G.J., Roychoudhury, A.: Program vulnerability repair via inductive inference. In: Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. pp. 691–702 (2022)

    work page 2022

  44. [44]

    ACM Transactions on Software Engineering and Methodology34(5), 1–31 (2025)

    Zhou, X., Cao, S., Sun, X., Lo, D.: Large language model for vulnerability detection and repair: Literature review and the road ahead. ACM Transactions on Software Engineering and Methodology34(5), 1–31 (2025)

    2025

  45. [45]

    In: Proceedings of the IEEE/ACM 46th international conference on software engineering

    Zhou, X., Kim, K., Xu, B., Han, D., Lo, D.: Out of sight, out of mind: Better auto- matic vulnerability repair by broadening input ranges and sources. In: Proceedings of the IEEE/ACM 46th international conference on software engineering. pp. 1–13 (2024)

    work page 2024

  46. [46]

    In: Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results

    Zhou, X., Zhang, T., Lo, D.: Large language model for vulnerability detection: Emerging results and future directions. In: Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results. pp. 47–51 (2024)

    work page 2024