Pith. sign in

REVIEW 4 major objections 5 minor 80 references

Code Change Intention, Development Artifact and History Vulnerability: Putting Them Together for Vulnerability Fix Detection by LLM

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that feeding an LLM a distilled change intention, linked issue and pull-request context, and a retrieved similar historical fix detects vulnerability-fixing commits with F1 up to 0.54, beating prior tools by 68–145% and…

desk verdict A useful dataset and a plausible LLM-based framework for vulnerability fix detection, but the headline gains rest on a leak-prone CCI design and an unverified date-based contamination control; both are fixable in revision. read the letter →

arxiv 2501.14983 v1 pith:PAEFZHOG submitted 2025-01-24 cs.SE

Xu Yang , Wenhan Zhu , Michael Pacheco , Jiayuan Zhou , Shaowei Wang , Xing Hu , Kui Liu This is my paper · ORCID
classification cs.SE
keywords vulnerabilityfixdetectionlargelanguagemodelschain-of-thoughtpromptingin-contextlearningretrieval-augmentedgenerationdevelopmentartifactstangledcommitsBigVulFixes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that large language models, when fed three kinds of context—a distilled intention of the code change, linked issue reports and pull requests, and a retrieved similar historical vulnerability fix—can detect vulnerability fix commits much more effectively than prior specialized machine learning tools, and can say why in a form security experts find useful. The motivating problem is that vulnerability fixes are often disclosed only after a delay, so downstream users need to spot the fixing commit as soon as it lands; existing code-only methods miss tangled commits and silent fixes. If true, the result matters because it would give open-source maintainers a practical screening tool that both flags candidate fixes and explains its reasoning, and it would show that context, not just patch text, is the key signal for this task.

What carries the argument

The load-bearing device is a structured LLM-generated "three-aspect summary"—code change summary, purpose, and implications—produced by Chain-of-Thought prompts, applied both to the commit and to its linked issue/PR artifacts, and embedded to retrieve a similar historical vulnerability fix whose CVE description and summary are fed into a final Comparison-and-Analysis prompt. That machinery converts raw diffs into intention-level representations, which is what lets the model see through tangled commits, vague commit messages, and subtle one-line checks.

What would settle it

Run a memorization probe: for each 2024 CVE fix commit in BigVulFixes, strip the fix-specific lines or ask the LLM to name the CVE from the raw diff; if the models can complete or identify the fixed vulnerability at high rates, the date-cutoff assumption fails and the F1 gains are suspect. Alternatively, rebuild the same evaluation with pre-2023 fixes and no historical-retrieval component; if the advantage over vanilla LLMs collapses, the post-2023 design is carrying the result.

Watch

Extended reading notes

Core claim

The central claim is that vulnerability fix detection can be recast as a context-enriched LLM reading task: instead of classifying a raw diff, the framework first has the LLM write a three-aspect summary of the commit (what changed, why, and what it implies), summarizes any linked issue report or pull request the same way, retrieves the nearest historical vulnerability fix by embedding those summaries, and then asks the LLM to compare and decide. On a new post-2023 dataset of 1,689 vulnerability fixes and 26,468 non-fixes across seven languages, the best configuration reaches F1 0.54 and MCC 0.52, outperforming the best prior tool by 68.1–145.4% in F1 and the same LLM without the added context by 12.7–105.6%. The paper also reports that the generated analysis helped security experts understand commit intent in 95% of the 40 cases tested and improved their efficiency in identifying fixes in 80%.

Load-bearing premise

The approach assumes that choosing vulnerabilities whose CVE entries were published after 2023 keeps the test cases out of what the LLMs memorized during training, so the measured gains reflect reasoning rather than recall; the paper does not verify this with contamination checks.

Editorial extensions

If this is right

  • On the BigVulFixes test set, every LLM tested, from 7B to 236B parameters, beats the strongest prior approach on F1, MCC, and recall, so the gain is not tied to one model family.
  • Smaller models improve the most—on average 64% F1 gain versus 14.4% for larger ones—so the approach can make modest-size models competitive for this task.
  • Ablation shows all three components add value, with the intention summary contributing most to precision; removing it drops precision by 13–15%.
  • The explanations are usable: 10 security experts said the analysis helped them understand the intent in 95% of cases and improved screening efficiency in 80% of cases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the paper relies on calendar dates rather than contamination probing, the cleanest test of the result is a memorization check; until then, part of the gain could come from the LLMs having seen these CVEs.
  • Editorial inference: the three-aspect summary is a reusable abstraction; it could be applied to other commit classification problems such as refactoring detection or bug versus feature classification.
  • Editorial inference: the failure analysis shows the historical retrieval can actively mislead the model, so a stronger retriever with reranking should improve both precision and recall; the framework makes this a modular swap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LLM4VFD, a framework for detecting vulnerability-fixing (VF) commits that combines three sources of LLM-distilled context: a Code Change Intention (CCI) summary, a Development Artifact (DA) summary derived from issue reports and pull requests, and a Historical Vulnerability (HV) retrieval component. These are merged in a Comprehensive Analysis and Vulnerability Fix Detection (CAVFD) prompt that asks an LLM to produce a yes/no vulnerability_fix judgment together with a free-text analysis. The authors introduce a new dataset, BigVulFixes, containing 1,689 VF commits and 26,468 non-VF commits sampled at 1:16, and they evaluate LLM4VFD on six LLMs from the Llama, Qwen, and DeepSeek families. They report F1 gains of 68.1%–145.4% over the best PLM baseline and 12.7%–105.6% over vanilla LLMs, plus an ablation study and a user study with ten security experts. The central claim is that combining CCI, DA, and HV context improves VF detection and provides useful explanations.

Significance. If the reported results withstand scrutiny, the paper would be a useful contribution to vulnerability fix detection: it is the first systematic LLM-based treatment of this task in the manuscript, it evaluates a multi-source prompting framework rather than a single prompt, and it releases a new post-2023 dataset and a replication package, which are concrete assets for the community. The multi-family, multi-size LLM evaluation and the ablation of each component are appropriate experimental structures. The user study, although flawed as a controlled experiment, is a reasonable first step toward assessing explainability utility. The main value is in demonstrating that artifacts beyond the raw diff—issue reports, pull requests, and historical fixes—can improve LLM-based VF detection, and that smaller LLMs benefit more from the framework. However, the reliability of the quantitative claims depends on resolving the contamination-control gap, the potential label leakage in the CCI prompt, and the absence of any uncertainty quantification.

major comments (4)
  1. [Section 5.2.1 and Section 5.4] The post-knowledge-cutoff control is applied to CVE publication dates, not to commit dates, so the claim that BigVulFixes is outside LLM pretraining data is not established. NVD references for a CVE published in 2024 can point to commits made in 2022 or 2023, exactly the silent-fix cases this paper targets; the paper never reports commit dates or checks whether the diffs and commit messages are absent from the training corpora of Llama3.1, Qwen2, and DeepSeek-Coder-V2. Because Table 1 is the sole support for RQ1, and vanilla LLM baselines can benefit from memorized fixes while PLM baselines cannot, the headline F1 gains over PLMs and vanilla LLMs could be inflated by memorization. The authors should filter by commit timestamps relative to each model's knowledge cutoff, run contamination probes, or report an analysis restricted to commits demonstrably after each model's training data collection window.
  2. [Section 4.1, Figure 5 and Section 4.4, Figure 7] The CCI prompt asks the LLM to categorize the 'Purpose of the Change' into categories that include 'fixing a vulnerability,' and that same CCI output is then inserted into the CAVFD prompt that asks for the final 'vulnerability_fix' label. The final decision can therefore be a restatement of a label that the same model already produced, rather than an independent judgment on the patch; this is a form of label leakage that differentially benefits LLM4VFD over the vanilla baseline. It also biases the HV retrieval, whose query is the CCI summary. To support the claim that CCI contributes reasoning rather than leaking the answer, the CCI prompt should avoid target-label vocabulary, or the evaluation should measure how often the CCI output contains the phrase 'fixing a vulnerability' before the CAVFD prediction and condition on its absence.
  3. [Section 5.6, Table 1, Table 2] All results are single point estimates with no variance, confidence intervals, or significance tests, despite stochastic LLM decoding and random non-VF sampling at a 1:16 ratio. The phrase 'significantly outperforms' is therefore not supported by the reported evidence. The authors should report repeated runs with different seeds or sample draws, and use paired tests such as bootstrap or McNemar for the RQ1 and RQ2 comparisons.
  4. [Section 5.7.3 and Section 6.3] The user study cannot support the claim that LLM4VFD's analysis improves efficiency, because there is no control condition in which participants see the raw commit without the generated analysis, no measurement of decision time or accuracy, and all 40 cases are positive VF examples. The yes/no questions are retrospective self-reports. The authors should compare conditions with and without the generated analysis on a mixed VF/NVF sample, ideally measuring decision time and verification accuracy.
minor comments (5)
  1. [Section 5.2.1 versus Section 6.3] The dataset is described as containing vulnerabilities 'after 2023,' but Section 6.3 discusses CVE-2023-48014, CVE-2023-37061, and CVE-2023-48657; please clarify whether BigVulFixes includes 2023 CVEs or whether the user study used a different sample.
  2. [Abstract and Section 4] The abstract says LLM4VFD comprises three components, while Section 4 describes four components including CAVFD; please align the naming convention.
  3. [Table 1] Please clarify the grouping of the three PLM rows under 'CodeBERT' and the meaning of the asterisk on CoLeFunDa; the current layout makes it easy to misread the foundation model column.
  4. [Throughout] There are typographical inconsistencies, including 'CoLeFunDa' versus 'ColeFunda'/'CoLeFunda', 'VulCurator' versus 'Vulcurator', and 'an 15.4% reduction'; a careful proofreading pass is needed.
  5. [References [44]-[46]] The reference URLs for CVE-2023-37061, CVE-2023-48014, and CVE-2023-48657 point to CVE-2024-37061, CVE-2024-48014, and CVE-2024-48657, respectively; the identifiers should be corrected.

Circularity Check

1 steps flagged · score 6.0 of 10

The final vulnerability-fix decision is a potential restatement of the CCI component's label-containing 'Purpose of the Change' category, making part of the claimed pipeline circular.

  1. self definitional [Section 4.1 (Code Change Intention) and Section 4.4 / Figure 7 (CAVFD)]
    "'This step categorizes the commit into broader categories such as refactoring, feature enhancement, or fixing a vulnerability.' ... Figure 7: 'Three Aspect Analysis of the Patch: {CCI component output}' ... 'Determine whether the current patch is intended to fix a vulnerability. You must provide evidence if you think its a vulnerability fix.'"

    The CCI 'Purpose of the Change' step explicitly names 'fixing a vulnerability' as one of its output categories, which is the same binary target the CAVFD step is supposed to predict. The CAVFD prompt then feeds the CCI output verbatim into the prompt and asks the LLM to determine whether the patch fixes a vulnerability. Consequently, the final 'vulnerability_fix' answer is, by construction, contained within the intermediate 3-aspect summary: the LLM can simply restate the CCI's purpose category instead of performing an independent synthesis. The reported gain of CCI in the ablation is therefore confounded, because removing CCI removes a feature that already encodes the target answer. This is a partial self-definitional reduction rather than a prediction from independent evidence.

full rationale

The central circular element is the CCI-to-CAVFD flow: the 'Purpose of the Change' aspect is explicitly defined to include the category 'fixing a vulnerability', and that output is inserted into the final prompt that asks for the vulnerability-fix decision. The final prediction can therefore be read off from the intermediate summary rather than derived from raw commit and artifact context. This is a genuine structural reduction, not merely a modeling choice, because the target label is one of the predefined output categories of the intermediate component. I did not find load-bearing self-citation circularity: the authors' own prior PLM systems are used as baselines, but no uniqueness claim or theoretical result is imported from self-citations to force the framework. The date-based data-leakage concern (filtering by CVE publication date rather than commit date) is a serious validity threat to the evaluation, but it is a contamination/memorization concern, not a circular-derivation concern, and I therefore exclude it from the score. The paper is otherwise self-contained, with external baselines and an ablation study; however, the CCI label-containment issue reduces part of the claimed causal contribution of the framework to a restatement of its own intermediate output, supporting a partial-circularity score of 6 rather than a higher score.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The framework has no fitted model parameters; the listed free parameters are dataset construction and retrieval settings that shape the reported numbers. The axioms are the unverified data-handling and pipeline assumptions that the evaluation depends on.

free parameters (3)
  • VF:NVF sampling ratio = 1:16
    Chosen to mirror the Big-Vul dataset; affects class balance and therefore precision/recall scores.
  • Retrieval neighbor count k = 1
    HV component returns a single nearest historical fix; the paper does not report sensitivity to k.
  • Patch token length cutoff = ~30,000 tokens (99th percentile)
    Patches above the cutoff are excluded to fit LLM context limits; may remove very large or tangled commits.
assumptions (5)
  • domain assumption Post-2023 CVE fix commits are outside LLM pretraining knowledge
    Section 5.2.1: the entire leakage mitigation rests on date ranges, but no verification is provided that the specific LLMs did not see these commits.
  • domain assumption NVD CVE references with GitHub commit URLs correctly identify vulnerability fix commits
    Section 5.2.2: ground truth positive labels come from these links; errors would bias evaluation.
  • domain assumption Randomly sampled commits without CVE links are non-vulnerability-fix commits
    Section 5.2.2: negative labels are assumed clean, yet the failure analysis itself reports 'potential unreported vulnerability fix' false positives.
  • domain assumption The 3-aspect LLM summary is a faithful, label-free description of the commit
    Sections 4.1 to 4.4: CCI prompt asks the LLM to categorize purpose including 'fixing a vulnerability', and this summary is fed into the final prediction, risking leakage of the answer.
  • domain assumption Same-language nearest-neighbor retrieval returns relevant historical fixes
    Section 5.5.2: retrieval quality is assumed from embedding similarity; the failure analysis shows irrelevant retrievals mislead the model in some cases.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Code Change Intention, Development Artifact and History Vulnerability: Putting Them Together for Vulnerability Fix Detection by LLM." pith.science (2026). https://pith.science/paper/PAEFZHOG

@misc{pith2026250114983,
  author       = {Pith},
  title        = {Pith review of: Code Change Intention, Development Artifact and History Vulnerability: Putting Them Together for Vulnerability Fix Detection by LLM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PAEFZHOG}},
  note         = {Machine review of arXiv:2501.14983}
}
read the original abstract

Detecting vulnerability fix commits in open-source software is crucial for maintaining software security. To help OSS identify vulnerability fix commits, several automated approaches are developed. However, existing approaches like VulFixMiner and CoLeFunDa, focus solely on code changes, neglecting essential context from development artifacts. Tools like Vulcurator, which integrates issue reports, fail to leverage semantic associations between different development artifacts (e.g., pull requests and history vulnerability fixes). Moreover, they miss vulnerability fixes in tangled commits and lack explanations, limiting practical use. Hence to address those limitations, we propose LLM4VFD, a novel framework that leverages Large Language Models (LLMs) enhanced with Chain-of-Thought reasoning and In-Context Learning to improve the accuracy of vulnerability fix detection. LLM4VFD comprises three components: (1) Code Change Intention, which analyzes commit summaries, purposes, and implications using Chain-of-Thought reasoning; (2) Development Artifact, which incorporates context from related issue reports and pull requests; (3) Historical Vulnerability, which retrieves similar past vulnerability fixes to enrich context. More importantly, on top of the prediction, LLM4VFD also provides a detailed analysis and explanation to help security experts understand the rationale behind the decision. We evaluated LLM4VFD against state-of-the-art techniques, including Pre-trained Language Model-based approaches and vanilla LLMs, using a newly collected dataset, BigVulFixes. Experimental results demonstrate that LLM4VFD significantly outperforms the best-performed existing approach by 68.1%--145.4%. Furthermore, We conducted a user study with security experts, showing that the analysis generated by LLM4VFD improves the efficiency of vulnerability fix identification.

Figures

Figures reproduced from arXiv: 2501.14983 by the authors.

Figure 1
Figure 1. An example of tangled commit with 164 lines changed, while only two lines (in red box) are related to [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. An example of a commit [21] with only 2 lines changed (in red box), while related issues reports [22] (in blue box) provided critical information [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. A commit only with small change by adding condition check (left [ [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The framework of LLM4VFD. includes similar changes to the same function in a prior commit, where the addition of a check for the __proto__ property was introduced to mitigate a known prototype pollution vulnerability. This example shows that historical vulnerability fi…
Figure 5
Figure 5. Figure 5: The prompt template of Code Change Intention. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The prompt template of Development Artifact. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: The prompt template of Comprehensive Analysis and Vulnerability Fix Detection. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

80 extracted references · 53 canonical work pages

  1. [1]

    LLM4VFD replication pacakge

    2024. LLM4VFD replication pacakge. Zenodo. https://doi.org/10.5281/zenodo.13776994

    work page doi:10.5281/zenodo.13776994 2024

  2. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  3. [3]

    Patrick Bareiß, Beatriz Souza, Marcelo d’Amorim, and Michael Pradel. 2022. Code generation tools (almost) for free? a study of few-shot, pre-trained language models on code. arXiv preprint arXiv:2206.01335 (2022)

    arXiv 2022

  4. [4]

    Mike Barnett, Christian Bird, João Brunet, and Shuvendu K Lahiri. 2015. Helping developers help themselves: Automatic decomposition of code review changesets. In2015 IEEE/ACM 37th IEEE International Conference on Software Engineering

    work page 2015

  5. [5]

    Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. 2021. Deep learning based vulnerability detection: Are we there yet? IEEE Transactions on Software Engineering (2021)

    work page 2021

  6. [6]

    Aaron Chan, Anant Kharkar, Roshanak Zilouchian Moghaddam, Yevhen Mohylevskyy, Alec Helyar, Eslam Kamal, Mohamed Elkamhawy, and Neel Sundaresan. 2023. Transformer-based vulnerability detection in code at EditTime: Zero-shot, few-shot, or fine-tuning? arXiv preprint arXiv:2306.01754 (2023)

    arXiv 2023

  7. [7]

    Banghao Chen, Zhaofeng Zhang, Nicolas Langrené, and Shengxin Zhu. 2023. Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review. arXiv preprint arXiv:2310.14735 (2023)

    arXiv 2023

  8. [8]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    arXiv 2021

Show all 80 references
  1. [9]

    Yang Chen, Andrew E Santosa, Ang Ming Yi, Abhishek Sharma, Asankhaya Sharma, and David Lo. 2020. A machine learning approach for vulnerability curation. In Proceedings of the 17th International Conference on Mining Software Repositories

    2020

  2. [10]

    Yiran Cheng, Lwin Khin Shar, Ting Zhang, Shouguo Yang, Chaopeng Dong, David Lo, Shichao Lv, Zhiqiang Shi, and Limin Sun. 2024. LLM-Enhanced Static Analysis for Precise Identification of Vulnerable OSS Versions. arXiv preprint arXiv:2408.07321 (2024)

    2024 arXiv

  3. [11]

    Chroma. 2024. Chroma is the open-source AI application database. Batteries included. https://www.trychroma.com/ accessed 2024-09-12

    2024

  4. [12]

    GitHub Docs. 2022. GitHub REST API documentation. https://docs.github.com/en/rest?apiVersion=2022-11-28 accessed 2024-08-19

    2022

  5. [13]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234 (2022)

    2022 arXiv

  6. [14]

    Xueying Du, Geng Zheng, Kaixin Wang, Jiayi Feng, Wentai Deng, Mingwei Liu, Bihuan Chen, Xin Peng, Tao Ma, and Yiling Lou. 2024. Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG. arXiv preprint arXiv:2406.11147 (2024)

    2024 arXiv

  7. [15]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  8. [16]

    Jiahao Fan, Yi Li, Shaohua Wang, and Tien N Nguyen. 2020. A C/C++ code vulnerability dataset with code changes and CVE summaries. In Proceedings of the 17th International Conference on Mining Software Repositories

    2020

  9. [17]

    Lishui Fan, Jiakun Liu, Zhongxin Liu, David Lo, Xin Xia, and Shanping Li. 2024. Exploring the Capabilities of LLMs for Code Change Related Tasks. arXiv preprint arXiv:2407.02824 (2024)

    2024 arXiv

  10. [18]

    Richard Fang, Rohan Bindu, Akul Gupta, Qiusi Zhan, and Daniel Kang. 2024. Teams of LLM Agents can Exploit Zero-Day Vulnerabilities. arXiv preprint arXiv:2406.01637 (2024)

    2024 arXiv

  11. [19]

    Henry Gilbert, Michael Sandborn, Douglas C Schmidt, Jesse Spencer-Smith, and Jules White. 2023. Semantic compres- sion with large language models. In 2023 Tenth International Conference on Social Networks Analysis, Management and Security (SNAMS)

    2023

  12. [20]

    GitHub. 2024. GitHub Advisory Database. https://github.com/advisories accessed 2024-08-19

    2024

  13. [21]

    gpac. 2023. fixed #2475 · gpac/gpac@c88df2e · GitHub. https://github.com/gpac/gpac/commit/ c88df2e202efad214c25b4e586f243b2038779ba accessed 2024-08-19. , Vol. 1, No. 1, Article . Publication date: January 2025. 20 Xu Yang, Wenhan Zhu, Michael Pacheco, Jiayuan Zhou, Shaowei Wa...

    2023

  14. [22]

    gpac. 2023. OOB Read segfault · Issue #2475 · gpac/gpac · GitHub. https://github.com/gpac/gpac/issues/2475 accessed 2024-08-19

    2023

  15. [23]

    Haibo He and Edwardo A Garcia. 2009. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering (2009)

    2009

  16. [24]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 (2020)

    2020 arXiv

  17. [25]

    Kim Herzig and Andreas Zeller. 2013. The impact of tangled code changes. In 2013 10th Working Conference on Mining Software Repositories (MSR)

    2013

  18. [26]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang

  19. [27]

    Nafis Tanveer Islam, Joseph Khoury, Andrew Seong, Gonzalo De La Torre Parra, Elias Bou-Harb, and Peyman Najafirad

  20. [28]

    Shuyang Jiang, Yuhao Wang, and Yu Wang. 2023. Selfevolve: A code evolution framework via large language models. arXiv preprint arXiv:2306.02907 (2023)

    2023 arXiv

  21. [29]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems (2022)

    2022

  22. [30]

    Ummay Kulsum, Haotian Zhu, Bowen Xu, and Marcelo d’Amorim. 2024. A Case Study of LLM for Automated Vulnerability Repair: Assessing Impact of Reasoning and Patch Validation Feedback. In Proceedings of the 1st ACM International Conference on AI-Powered Software

    2024

  23. [31]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  24. [32]

    Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023. Comparing code explanations created by students and large language models. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Sci...

    2023

  25. [33]

    Yi Li, Shaohua Wang, and Tien N Nguyen. 2021. Vulnerability detection with fine-grained interpretations. InProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

    2021

  26. [34]

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281 (2023)

    2023 arXiv

  27. [35]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems (2024)

    2024

  28. [36]

    Shigang Liu, Guanjun Lin, Lizhen Qu, Jun Zhang, Olivier De Vel, Paul Montague, and Yang Xiang. 2020. CD-VulD: Cross-domain vulnerability discovery based on deep domain adaptation. IEEE Transactions on Dependable and Secure Computing (2020)

    2020

  29. [37]

    Brian W Matthews. 1975. Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure (1975)

    1975

  30. [38]

    Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2022. MTEB: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316 (2022)

    2022 arXiv

  31. [39]

    Son Nguyen, Thanh Trong Vu, and Hieu Dinh Vo. 2023. VFFINDER: A Graph-based Approach for Automated Silent Vulnerability-Fix Identification. In 2023 15th International Conference on Knowledge and Systems Engineering (KSE)

    2023

  32. [40]

    Truong Giang Nguyen, Thanh Le-Cong, Hong Jin Kang, Xuan-Bach D Le, and David Lo. 2022. Vulcurator: a vulnerability- fixing commit detector. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

    2022

  33. [41]

    Truong Giang Nguyen, Thanh Le-Cong, Hong Jin Kang, Ratnadira Widyasari, Chengran Yang, Zhipeng Zhao, Bowen Xu, Jiayuan Zhou, Xin Xia, Ahmed E Hassan, et al . 2023. Multi-granularity detector for vulnerability fixes. IEEE Transactions on Software Engineering (2023)

    2023

  34. [42]

    Giang Nguyen-Truong, Hong Jin Kang, David Lo, Abhishek Sharma, Andrew E Santosa, Asankhaya Sharma, and Ming Yi Ang. 2022. Hermes: Using commit-issue linking to detect vulnerability-fixing commits. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengi...

    2022

  35. [43]

    NIST. 2024. NVD - Home. https://nvd.nist.gov/ accessed 2024-09-10

    2024

  36. [44]

    NVD. 2023. NVD - CVE-2023-37061. https://nvd.nist.gov/vuln/detail/CVE-2024-37061 accessed 2024-08-25

    2023

  37. [45]

    NVD. 2023. NVD - CVE-2023-48014. https://nvd.nist.gov/vuln/detail/CVE-2024-48014 accessed 2024-08-25

    2023

  38. [46]

    NVD. 2023. NVD - CVE-2023-48657. https://nvd.nist.gov/vuln/detail/CVE-2024-48657 accessed 2024-08-25. , Vol. 1, No. 1, Article . Publication date: January 2025. Code Change Intention, Development Artifact and History Vulnerability: Putting Them Together for Vulnerability Fix D...

    2023

  39. [47]

    NVD. 2024. NVD - CVE-2024-28103. https://nvd.nist.gov/vuln/detail/CVE-2024-28103 accessed 2024-08-25

    2024

  40. [48]

    NVD. 2024. NVD - CVE-2024-29199. https://nvd.nist.gov/vuln/detail/CVE-2024-29199 accessed 2024-08-25

    2024

  41. [49]

    OpenSSF. 2024. Guide to coordinated vulnerability disclosure for open source software projects. https://github.com/ ossf/oss-vulnerability-guide accessed 2024-07-26

    2024

  42. [50]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35...

    2022

  43. [51]

    Shengyi Pan, Lingfeng Bao, Xin Xia, David Lo, and Shanping Li. 2023. Fine-grained commit-level vulnerability type prediction by CWE tree structure. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)

    2023

  44. [52]

    Shengyi Pan, Jiayuan Zhou, Filipe Roseiro Cogo, Xin Xia, Lingfeng Bao, Xing Hu, Shanping Li, and Ahmed E Hassan

  45. [53]

    Max Peeperkorn, Tom Kouwenhoven, Dan Brown, and Anna Jordanous. 2024. Is temperature the creativity parameter of large language models? arXiv preprint arXiv:2405.00492 (2024)

    2024 arXiv

  46. [54]

    protobufjs. 2022. fix: do not let setProperty change the prototype (#1731) · protobufjs/protobuf.js@3357ef7 · GitHub. https://github.com/protobufjs/protobuf.js/commit/3357ef753871b394b825d15429ceb27b26e24d63 accessed 2024-08-19

    2022

  47. [55]

    protobufjs. 2023. fix: do not let setProperty change the prototype (#1899) · protobufjs/protobuf.js@e66379f · GitHub. https://github.com/protobufjs/protobuf.js/commit/e66379f451b0393c27d87b37fa7d271619e16b0d accessed 2024-08-19

    2023

  48. [56]

    Md Mahbubur Rahman, Ira Ceka, Chengzhi Mao, Saikat Chakraborty, Baishakhi Ray, and Wei Le. 2024. Towards causal deep learning for vulnerability detection. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering

    2024

  49. [57]

    Matthew Renze and Erhan Guven. 2024. The effect of sampling temperature on problem solving in large language models. arXiv preprint arXiv:2402.05201 (2024)

    2024 arXiv

  50. [58]

    Antonino Sabetta and Michele Bezzi. 2018. A practical approach to the automatic classification of security-relevant commits. In 2018 IEEE International conference on software maintenance and evolution (ICSME)

    2018

  51. [59]

    Da Shen, Xinyun Chen, Chenguang Wang, Koushik Sen, and Dawn Song. 2022. Benchmarking Language Models for Code Syntax Understanding. In Findings of the Association for Computational Linguistics: EMNLP 2022

    2022

  52. [60]

    spring projects. 2023. Polishing and minor refactoring in HandlerMappingIntrospector · spring- projects/spring-framework@202fa5c · GitHub. https://github.com/spring-projects/spring-framework/commit/ 202fa5cdb3a3d0cfe6967e85fa167d978244f28a accessed 2024-08-19

    2023

  53. [61]

    Chandra Thapa, Seung Ick Jang, Muhammad Ejaz Ahmed, Seyit Camtepe, Josef Pieprzyk, and Surya Nepal. 2022. Transformer-based language models for software vulnerability detection. In Proceedings of the 38th Annual Computer Security Applications Conference

    2022

  54. [62]

    The New York Times. 2019. Equifax to Pay at Least $650 Million in Largest-Ever Data Breach Settlement. https: //www.nytimes.com/2019/07/22/business/equifax-settlement.html accessed 2024-08-19

    2019

  55. [63]

    Yue Wang, Hung Le, Akhilesh Gotmare, Nghi Bui, Junnan Li, and Steven Hoi. 2023. CodeT5+: Open Code Large Language Models for Code Understanding and Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  56. [64]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al . 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems (2022)

    2022

  57. [65]

    Yi Wu, Nan Jiang, Hung Viet Pham, Thibaud Lutellier, Jordan Davis, Lin Tan, Petr Babkin, and Sameena Shah. 2023. How effective are neural networks for fixing security vulnerabilities. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

    2023

  58. [66]

    Zhengzi Xu, Bihuan Chen, Mahinthan Chandramohan, Yang Liu, and Fu Song. 2017. Spain: security patch analysis for binaries towards understanding the pain and pills. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE)

    2017

  59. [67]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671 (2024)

    2024 arXiv

  60. [68]

    Xu Yang, Shaowei Wang, Yi Li, and Shaohua Wang. 2023. Does data sampling improve deep learning-based vulnerability detection? Yeas! and Nays!. In Proceedings of the 45th IEEE/ACM International Conference on Software Engineering (ICSE)

    2023

  61. [69]

    JD Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–21

    2023

  62. [70]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911 (2023). , Vol. 1, No. 1, Article . Publication date: January 2025. 22 Xu...

    2023 arXiv

  63. [71]

    Jiayuan Zhou, Michael Pacheco, Jinfu Chen, Xing Hu, Xin Xia, David Lo, and Ahmed E Hassan. 2023. Colefunda: Ex- plainable silent vulnerability fix identification. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)

    2023

  64. [72]

    Jiayuan Zhou, Michael Pacheco, Zhiyuan Wan, Xin Xia, David Lo, Yuan Wang, and Ahmed E Hassan. 2021. Finding a needle in a haystack: Automated mining of silent vulnerability fixes. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)

    2021

  65. [73]

    Xin Zhou, Sicong Cao, Xiaobing Sun, and David Lo. 2024. Large Language Model for Vulnerability Detection and Repair: Literature Review and the Road Ahead. arXiv preprint arXiv:2404.02525 (2024)

    2024 arXiv

  66. [74]

    Xin Zhou, Kisub Kim, Bowen Xu, DongGyun Han, and David Lo. 2024. Out of Sight, Out of Mind: Better Automatic Vulnerability Repair by Broadening Input Ranges and Sources. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering

    2024

  67. [75]

    Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. 2019. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. Advances in neural information processing systems (2019)

    2019

  68. [76]

    Yaqin Zhou and Asankhaya Sharma. 2017. Automated identification of security issues from commit messages and bug reports. In Proceedings of the 2017 11th joint meeting on foundations of software engineering

    2017

  69. [77]

    Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. 2024. DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence. arXiv preprint arXiv:2406.11931 (2024). , Vol. 1, No. 1, Article . ...

    2024 arXiv

  70. [2022]

    In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

    Automated unearthing of dangerous issue reports. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

  71. [2023]

    arXiv preprint arXiv:2308.10620 (2023)

    Large language models for software engineering: A systematic literature review. arXiv preprint arXiv:2308.10620 (2023)

    2023 arXiv

  72. [2024]

    arXiv preprint arXiv:2401.03374 (2024)

    LLM-Powered Code Vulnerability Repair with Reinforcement Learning and Semantic Reward. arXiv preprint arXiv:2401.03374 (2024)

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.