Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

VulnRepairEval: An Exploit-Based Evaluation Framework for Assessing Large Language Model Vulnerability Repair Capabilities

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A benchmark that requires a patch to block a working exploit finds the best LLM repairs only 21.7% of 23 real CVEs, with most failures caused by missed localization and malformed patches.

desk verdict Useful benchmark with a robust core finding, but the 21.7% headline overstates true repair because the payload-variant reruns that would quantify the gap are never reported. read the letter →

arxiv 2509.03331 v1 pith:7Q6ET3R6 submitted 2025-09-03 cs.SE cs.CR

Weizhe Wang , Wei Ma , Qiang Hu , Yao Zhang , Jianfei Sun , Bin Wu , Yang Liu , Guangquan Xu
show 1 more author
Lingxiao Jiang
This is my paper
classification cs.SEcs.CR
keywords evaluationvulnerabilityframeworkvulnrepairevalapplicationsassessmentautomatedcomprehensive
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VulnRepairEval is a new way to test whether large language models can actually fix software security holes. Older benchmarks often use unit tests, which only check that a program still runs; a patch can pass those tests while the vulnerability remains exploitable. This paper instead collects 23 real Python vulnerabilities from known CVEs and pairs each with a proof-of-concept exploit, a small program that provably triggers the flaw. The evaluation then runs the exploit against the original vulnerable code, and again against the code with the model's patch applied. A repair is counted as successful only when the exploit fails on the patched version.

The authors tested 12 well-known models, including Gemini, GPT, DeepSeek, and Qwen. The best result was 5 out of 23, or about 21.7%, from Gemini 2.5 Pro. Several models fixed nothing at all. Looking at the failures, most came from the model not locating the bug in the first place, rather than from writing a bad patch. Adding extra guidance to the prompts, such as revealing the vulnerability type or asking for step-by-step reasoning, barely changed the results. A more interactive agent framework reduced the number of missed bugs but still ended with low repair rates.

The framework itself is a contribution: it gives security researchers a reproducible, containerized way to judge 'did the attack actually stop?'. However, the paper also admits that blocking one specific exploit does not guarantee the vulnerability is truly gone, and the PoCs and code are not yet public, so the exact numbers should be read cautiously.

Extended reading notes

Core claim

The paper's headline result: 'even the top-performing model successfully addresses merely 5/23 instances (about 21.7%)' under the criterion that a repair is successful only if '(i) the model emits a patch that applies to the target project... and (ii) the PoC that succeeds on the baseline fails after patching.' If correct, this means current LLMs rarely neutralize real-world exploits even when the vulnerable file is given directly.

Load-bearing premise

The differential proof relies on the premise that a PoC failing against the patched build implies the vulnerability is repaired. The paper itself concedes in Section 7 that 'defining success as original PoC failure risks blocking the trigger rather than eliminating the underlying bug (e.g., brittle input guards)', and acknowledges PoC failure is 'necessary, not sufficient'. If a patch merely hardens the specific exploit path, the benchmark counts it as a fix even though the attack surface remains.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. VulnRepairEval constructs a benchmark of 23 real-world Python CVE instances with working proof-of-concept (PoC) exploits, and introduces a containerized differential evaluation pipeline: a repair is counted as successful only if the model emits an applicable patch and the original PoC, which succeeds on the vulnerable baseline, fails on the patched build. The paper evaluates 12 LLMs under a unified prompt and reports that the best model, Gemini 2.5 Pro, achieves P_succ = 5/23 ≈ 21.7%, with an average success rate around 7%. It also provides a failure taxonomy (localization vs. patch generation), a difficulty-tier analysis, and an evaluation of prompt enhancements and an agentic workflow (SWE-Agent). The central claim is that current LLMs rarely neutralize real-world exploits even when given the vulnerable source file directly.

Significance. If the validation protocol is accepted, the paper makes a useful contribution: it moves vulnerability-repair evaluation from unit-test or PoV-based checks toward exploit-level differential validation, provides a reproducible containerized harness, and reports a broad model comparison with a clear negative result. The paper is also commendably cautious in separating the primary metric P_succ from the leaderboard-only composite S_p, and it explicitly acknowledges in Section 7 that P_succ is 'necessary, not sufficient' for true repair. The stress-test concern about circularity does not land: P_succ is an operational definition based on observed PoC outcomes, and S_p is not used for scientific conclusions. However, the acknowledged necessary-not-sufficient limitation, combined with the absence of the promised payload-variant rerun data, means the reported success rates are upper bounds on true vulnerability elimination. The benchmark's representativeness is also weakened by an underspecified selection of 23 CVEs from 448. These issues do not invalidate the qualitative finding that LLMs perform poorly, but they do affect the benchmark's validity as a measure of 'repair' rather than 'tri

major comments (4)
  1. [Section 7 (Discussion, Threats & biases)] The paper explicitly concedes that 'defining success as original PoC failure risks blocking the trigger rather than eliminating the underlying bug (e.g., brittle input guards)' and that 'PoC failure is necessary, not sufficient.' It then states this is 'mitigated... via containerized differential validation and reruns with payload variants,' but no protocol, data, or results from any payload-variant reruns are reported anywhere in the evaluation. Since Table 3 and the headline 5/23 figure rest entirely on P_succ, the reported rates are upper bounds on actual vulnerability elimination, and the magnitude of overstatement is unknown. This is load-bearing for the benchmark's central claim; the authors should either report the variant-rerun results or reframe P_succ and all derived conclusions as 'trigger-blocking rate' rather than 'repair success.'
  2. [Section 3 (Data Construction)] The selection of the final 23 CVEs from the 448-CVE corpus is not specified. The paper describes an LLM-based URL classification pipeline that flagged 109 CVEs with executable PoCs and a validation step with one false positive, but it does not state the criteria that reduced this set to 23 CVEs (e.g., containerizability, dependency resolution, diversity across categories/years, or removal of PoCs that failed in the harness). The benchmark's difficulty distribution and all absolute success rates depend directly on this curation. The authors should provide a reproducible selection protocol or at least a detailed per-CVE inclusion/exclusion table so readers can assess representativeness.
  3. [Sections 3 and 4.2 (PoC validation)] The verification of PoC functionality is described only as 'confirming that it produced the expected outcome,' with no per-CVE expected-outcome definitions, an audit trail, or a discussion of PoCs that failed during validation. Furthermore, PoCs are intentionally not released publicly, and access is 'limited research access,' while the evaluation scripts are not yet available at the time of writing. This limits independent verification of the differential basis for the entire benchmark. At minimum, the authors should provide an artifact appendix with per-CVE baseline execution logs, exit codes, and output hashes, so reviewers and subsequent researchers can confirm the baseline-succeeds condition.
  4. [Section 6.1 (RQ1 evaluation setup)] The paper states that 'we conducted repeated experiments for verification, and the results showed pretty small variation,' but no repetition count, temperature/sampling details, variance measures, or per-model error bars are reported. Given the small instance count (n=23) and the stochasticity of LLM outputs, this claim is not verifiable. The authors should report the number of runs, the per-CVE stability across runs, and the resulting confidence intervals for P_succ, or at least provide a reproducibility statement detailing how variation was quantified.
minor comments (5)
  1. [Table 2] The per-CVE result table is difficult to parse: many cells are empty, and the repeated 'F R' column headers without explicit '✓/✗' values for every cell make it hard to verify per-model outcomes. Consider a matrix with explicit symbols for 'not attempted', 'failed', and 'success', plus a legend.
  2. [Throughout] There are inconsistent model name spellings: 'GPT-o4 mini' in Section 5.1 vs. 'GPT o4 mini' in Tables 3/5/6, and 'GPT 40 mini' in Section 6.2. Please standardize.
  3. [Table 1 and References] VUL4C is cited to reference [12], which is the 'SoK: Automated Vulnerability Repair' paper rather than the original VUL4C dataset paper. Please cite the correct source for VUL4C.
  4. [Section 4.1] The prompt in Appendix A instructs the model to output either a unified diff or the literal '[VulnRepairEval No Patch]', but the main text presents the task as a 'two-part objective' of localization and patching. The prompt does not ask for an explicit localization output. This discrepancy should be clarified.
  5. [Section 6.3] The claim that 'the agent framework cannot override the inherent characteristics and limitations of the base models' is illustrated with GPT-series conservatism, but no quantitative breakdown of agent behavior by model is given beyond Table 7. A brief discussion of why DeepSeek V3 improved so markedly under the agent while others did not would strengthen the analysis.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: P_succ is an observed endpoint, not a fitted or self-referential quantity; the only self-citation is non-load-bearing and the key limitation is explicitly acknowledged.

full rationale

The paper's derivation chain is self-contained. The primary metric P_succ is defined in Section 5.2 as the observed proportion of CVEs where a generated patch applies and the PoC that succeeds on the unpatched baseline fails on the patched build; it is not a quantity fitted to the data it is used to summarize. The composite score S_p in Eq. (1) uses hand-set weights (beta=2, 0.5 penalty), but the paper explicitly states it is used only for leaderboard ordering and not for hypothesis testing, and all scientific conclusions are drawn from P_succ. No parameter is fit to a subset of data and then renamed as a prediction. The benchmark construction used an LLM classifier for PoC candidate identification, but this was validated by executing the PoCs against vulnerable projects, and it is not the evaluation endpoint. The only self-citation ([27]) points to the released artifact and carries no load in the argument. The manuscript itself flags in Section 7 that defining success as original PoC failure 'risks blocking the trigger rather than eliminating the underlying bug' and that PoC failure is 'necessary, not sufficient'; the claimed mitigation via reruns with payload variants is not reported, which is a validity and evidence gap, but it does not exhibit a reduction of an output to an input. Under the stated circularity criteria, there is no step where a prediction equals its input by construction.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The benchmark's central metric rests on the correctness and representative selection of PoCs, the parent-commit baseline, and the assumption that PoC failure equals repair. The authors themselves flag the last as necessary but not sufficient.

free parameters (2)
  • beta in Sp composite = 2
    Hand-set weight for the F-beta-style composite used only for leaderboard ordering, not for hypothesis testing; does not affect P_succ.
  • abstention penalty weight in Sp = 0.5
    Arbitrary multiplicative penalty for V_dnf in the composite score; leaderboard-only, not a scientific endpoint.
assumptions (5)
  • domain assumption A working PoC for each selected CVE exists and is correctly validated
    The entire benchmark depends on the authenticity and correctness of the PoCs; the paper describes a validation process but with limited detail, and the PoCs are not released.
  • domain assumption The parent commit of the official fix is the vulnerable baseline
    Section 4.3 checks out the direct parent of the security-fixing commit to establish the vulnerable state; if the fix is spread across multiple commits, this baseline may be incomplete.
  • domain assumption PoC failure after patching indicates the vulnerability is neutralized
    Section 7 explicitly warns this is necessary but not sufficient: a patch may block the specific PoC trigger while leaving the underlying bug reachable.
  • domain assumption Providing only the vulnerable file to the model is a valid task framing
    Section 4.1 restricts input to the single vulnerable file, which is a design choice that can both help and hinder real-world performance.
  • domain assumption The 23 selected CVEs are representative of the broader population of Python vulnerabilities
    Section 3 narrows from 448 CVEs to 109 with executable PoCs to a final 23; the inclusion criteria for the final set are not fully specified, creating selection-bias risk.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VulnRepairEval: An Exploit-Based Evaluation Framework for Assessing Large Language Model Vulnerability Repair Capabilities." pith.science (2026). https://pith.science/paper/7Q6ET3R6

@misc{pith2026250903331,
  author       = {Pith},
  title        = {Pith review of: VulnRepairEval: An Exploit-Based Evaluation Framework for Assessing Large Language Model Vulnerability Repair Capabilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7Q6ET3R6}},
  note         = {Machine review of arXiv:2509.03331}
}
read the original abstract

The adoption of Large Language Models (LLMs) for automated software vulnerability patching has shown promising outcomes on carefully curated evaluation sets. Nevertheless, existing datasets predominantly rely on superficial validation methods rather than exploit-based verification, leading to overestimated performance in security-sensitive applications. This paper introduces VulnRepairEval, an evaluation framework anchored in functional Proof-of-Concept (PoC) exploits. Our framework delivers a comprehensive, containerized evaluation pipeline that enables reproducible differential assessment, where repair success requires the original exploit to fail execution against the modified code. The benchmark construction involved extensive data curation: we processed over 400 CVEs and approximately 2,500 potential sources to extract a collection of authentic vulnerability instances (23 Python CVEs) amenable to automated testing with working PoCs. Through VulnRepairEval, we conduct a comprehensive evaluation of 12 popular LLMs and observe a significant performance deficit: even the top-performing model successfully addresses merely 5/23 instances (about 21.7%), exposing critical weaknesses in security-focused applications. Our failure analysis reveals that most unsuccessful attempts stem from imprecise vulnerability identification and patches containing syntactic or semantic errors. Enhanced prompting strategies and multi-agent approaches yield minimal improvements, with overall effectiveness remaining largely unaffected. This work contributes a stringent, practical evaluation framework for LLM-driven vulnerability remediation and underscores the necessity for assessment protocols that authentically reflect real-world exploitation scenarios.

Figures

Figures reproduced from arXiv: 2509.03331 by the authors.

Figure 1
Figure 1. Distribution of vulnerability types of VulnRe [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The overview of the VulnRepairEval framework. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Impact of Static Prompt Enhancements on Vulnerability Repair Performance. [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Prompt for Patch Generation </FILENAME> <CODE> File code to be analyzed. </CODE> ----- Let’s think step by step. Proceed with the analysis and generate the output now. A.3 Prompt for including an example # ROLE You are a top-tier software engineer and cybersecurity exp…
Figure 5
Figure 5. Figure 5: Different LLMs’ generation of patches for CVE-2022-21797 [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Vul4Py: Benchmarking Automated Vulnerability Repair in Python with Paired Exploit and Functional Oracles

    cs.SE 2026-08 conditional novelty 6.0 of 10

    Vul4Py gives 100 real Python vulnerabilities a paired exploit-and-functional oracle, and on this benchmark agentic repair systems outperform direct prompting and a specialized tool by a wide margin.

Reference graph

Works this paper leans on

38 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [1]

    git-apply Documentation

    [n.d.]. git-apply Documentation. https://git-scm.com/docs/git-apply Accessed: 2025-08-26

    work page 2025

  2. [2]

    patch Command – IBM AIX Documentation

    [n.d.]. patch Command – IBM AIX Documentation . https://www.ibm.com/docs/ en/aix/7.2.0?topic=p-patch-command Accessed: 2025-08-26

    work page 2025

  3. [3]

    Md Basim Uddin Ahmed, Nima Shiri Harzevili, Jiho Shin, Hung Viet Pham, and Song Wang. 2025. SecVulEval: Benchmarking LLMs for Real-World C/C++ Vulnerability Detection. arXiv:2505.19828 [cs.SE] https://arxiv.org/abs/2505. 19828

    arXiv 2025

  4. [4]

    Jafar Akhoundali, Sajad Rahim Nouri, Kristian Rietveld, and Olga Gadyatskaya

    work page

  5. [5]

    Guru Bhandari, Amara Naseer, and Leon Moonen. 2021. CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software. In Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering (PROMISE 2021) . ACM, New York, NY, USA, 30–39. https://doi.org/10.1145/3475960.3475985

    work page arXiv 2021

  6. [6]

    Smith, Donald Stufft

    Brett Cannon, Nathaniel J. Smith, Donald Stufft. 2016. PEP 518 – Specifying Minimum Build System Requirements for Python Projects. https://peps.python. org/pep-0518/. Accessed: 2025-07-29

    work page 2016

  7. [7]

    Díaz Ferreyra

    Quang-Cuong Bui, Riccardo Scandariato, and Nicolás E. Díaz Ferreyra. 2022. Vul4J: A Dataset of Reproducible Java Vulnerabilities Geared Towards the Study of Program Repair Techniques. In Proceedings of the 19th International Conference on Mining Software Repositories (MSR 2022), Data and Tool Showcase . ACM, 464–

    work page 2022

  8. [8]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025)

    arXiv 2025

Show all 38 references
  1. [9]

    CVE Program. 2020. CVE-2020-14343. https://cve.org/CVERecord?id=CVE-2020- 14343 PyYAML incomplete fix enabling code execution

    2020

  2. [10]

    2024.{PentestGPT}: Evaluating and harnessing large language models for automated penetration testing

    Gelei Deng, Yi Liu, Víctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. 2024.{PentestGPT}: Evaluating and harnessing large language models for automated penetration testing. In 33rd USENIX Security Symposium (USENIX ...

    2024

  3. [11]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

    2025 arXiv

  4. [12]

    Yu Hu, Zhendong Li, Kai Shu, et al. 2025. SoK: Automated Vulnerability Repair: Methods, Tools, and Assessments. arXiv:2506.11697 [cs.SE] https://arxiv.org/ abs/2506.11697

    2025 arXiv

  5. [13]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024)

    2024 arXiv

  6. [14]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. InProceedings of the 12th International Conference on Learning Representations (ICLR 2024) . ht...

    2024 arXiv

  7. [15]

    René Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA 2014). ACM, New York, NY, USA, 43...

    2014 doi

  8. [16]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)

    2024 arXiv

  9. [17]

    Fei Long, Fei Gao, Zhiyong Zha, Jialin Chen, Mingyang Yu, Wei Zhan, Haohua Meng, Chen Chen, Dai Hou, and Junguo Hu. 2023. PoC Self-Generation Tech- nology Based on Vulnerability Verification Program. In 2023 IEEE 4th Annual Flagship India Council International Subsections Conf...

    2023

  10. [18]

    Aman Madaan, Katherine Hermann, and Amir Yazdanbakhsh. 2023. What Makes Chain-of-Thought Prompting Effective? A Counterfactual Study. In Findings of the Association for Computational Linguistics: EMNLP 2023 , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Co...

    2023 doi

  11. [19]

    Chao Ni, Liyu Shen, Xiaohu Yang, Yan Zhu, and Shaohua Wang. 2024. MegaVul: A C/C++ Vulnerability Dataset with Comprehensive Code Representations. In Proceedings of the 21st International Conference on Mining Software Repositories (MSR 2024), Data and Tool Showcase . ACM. https...

    2024 doi

  12. [20]

    NIST. 2025. National Vulnerability Database. https://nvd.nist.gov/. Accessed: 2025-07-29

    2025

  13. [21]

    NVD - NIST. 2021. CVE-2021-25289. https://nvd.nist.gov/vuln/detail/CVE-2021- 25289 Pillow TIFF decoding overflow; attributed to incomplete fix

    2021

  14. [22]

    NVD - NIST. 2024. CVE-2024-27351. https://nvd.nist.gov/vuln/detail/CVE-2024- 27351 Django ReDoS linked to incomplete fix of earlier CVEs

    2024

  15. [23]

    OpenAI. 2023. Function calling and other API updates. https://openai.com/index/ function-calling-and-other-api-updates/. Accessed: 2025-07-29

    2023

  16. [24]

    OpenAI. 2025. Introducing OpenAI o3 and o4-mini. https://openai.com/index/ introducing-o3-and-o4-mini/. Accessed: 2025-07-29

    2025

  17. [25]

    Pedro Orvalho and Marta Kwiatkowska. 2025. Are Large Language Models Robust in Understanding Code Against Semantics-Preserving Mutations? arXiv preprint arXiv:2505.10443 (2025)

    2025 arXiv

  18. [26]

    Deniz Simsek, Aryaz Eghbali, and Michael Pradel. 2025. PoCGen: Gen- erating Proof-of-Concept Exploits for Vulnerabilities in Npm Packages. arXiv:2506.04962 [cs.CR] https://arxiv.org/abs/2506.04962

    2025 arXiv

  19. [27]

    VulnRepairEval. 2025. VulnRepairEval. (Pleasecontacttheauthors.). Accessed: 2025-07-29

    2025

  20. [28]

    Peiran Wang, Xiaogeng Liu, and Chaowei Xiao. 2025. CVE-Bench: Benchmark- ing LLM-based Software Engineering Agent’s Ability to Repair Real-World CVE Vulnerabilities. In Proceedings of the 2025 Conference of the Nations of the Ameri- cas Chapter of the Association for Computati...

    2025

  21. [29]

    Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, and Bing Xiang. 2023. Re- Code: Robustness Evaluation of Code Generation Models. In P...

    2023

  22. [30]

    Xing Wang, Ruiqi Hu, Chao Gao, et al . 2024. ReposVul: A Repository-Level High-Quality Vulnerability Dataset. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings (ICSE Companion 2024). IEEE, 472–483. https://doi.org/...

    2024

  23. [31]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. InProceedings of the 36th International Conference on Neural Information Processin...

    2022

  24. [32]

    Yizheng Wei, Olivier Duchenne, Jade Copet, et al . 2025. SWE-RL: Advanc- ing LLM Reasoning via Reinforcement Learning on Open Software Evolution. arXiv:2502.18449 [cs.LG] https://arxiv.org/abs/2502.18449

    2025 arXiv

  25. [33]

    Ratnadira Widyasari, Sheng Qin Sim, Camellia Lok, Haodi Qi, Jack Phan, Qijin Tay, Constance Tan, Fiona Wee, Jodie Ethelda Tan, Yuheng Yieh, Brian Goh, Ferdian Thung, Hong Jin Kang, Thong Hoang, David Lo, and Eng Lieh Ouh. 2020. BugsInPy: A Database of Existing Bugs in Python P...

    2020

  26. [34]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025)

    2025 arXiv

  27. [35]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Inter- faces Enable Automated Software Engineering. In The Thirty-eighth Annual Con- ference on Neural Information Processing Systems...

    2024 arXiv

  28. [36]

    Tianyi Zhuo, Minh Chien Vu, Jingkai Chim, et al. 2024. BigCodeBench: Bench- marking Code Generation with Diverse Function Calls and Complex Instructions. arXiv:2406.15877 [cs.LG] https://arxiv.org/abs/2406.15877

    2024 arXiv

  29. [468]

    https://doi.org/10.1145/3524842.3528482

  30. [2024]

    In Proceedings of the 20th International Confer- ence on Predictive Models and Data Analytics in Software Engineering (PROMISE 2024)

    MoreFixes: A Large-Scale Dataset of CVE Fix Commits Mined through Enhanced Repository Discovery. In Proceedings of the 20th International Confer- ence on Predictive Models and Data Analytics in Software Engineering (PROMISE 2024). Association for Computing Machinery, New York,...

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.