Pith. sign in

REVIEW 4 major objections 4 minor 65 references

Many software vulnerabilities are not fixed by a single patch, and existing tools can't tell whether the first patch actually completes the fix.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 05:51 UTC pith:K332DUSD

load-bearing objection First systematic map of multi-patch fixes with a released dataset, but the NVD patch-label ground truth is noisier than the paper admits. the 4 major comments →

arxiv 2607.13206 v1 pith:K332DUSD submitted 2026-07-14 cs.CR cs.SE

Why Not Fix It Once and for All? An Empirical Study of Multiple Patches for Vulnerability Fixes in Open-Source Software

Weiliang Qi , Youpeng Li , Xinda Wang This is my paper
classification cs.CR cs.SE
keywords multi-patch fixessecurity patchesvulnerability remediationopen-source softwarevulnerability detectionpatch completenesstaxonomydefective fixes
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper studies open-source vulnerabilities that were fixed through more than one patch, rather than a single commit. By manually examining 1,646 such records from the public vulnerability database, the authors find that multi-patch fixes are common and can be grouped into three root causes: fixes that span multiple code locations, fixes accompanied by related changes like workarounds or documentation, and defective fixes where the first patch was incomplete or introduced new bugs. They then show that mainstream vulnerability-detection and vulnerable-code-clone tools perform poorly on multi-patch scenarios, with all tested detectors scoring below 50% accuracy and F1 on incomplete-fix detection. The paper argues that treating a single patch as the endpoint of remediation produces noisy datasets and risky patch-management decisions.

Core claim

The central claim is that multi-patch vulnerability fixes are a structurally distinct and under-recognized phenomenon in open-source software. Analyzing 1,646 multi-patch records from the NVD, the paper introduces a taxonomy with three primary categories: multi-location fixes (same vulnerability in different branches/projects or same-branch locations), fix-and-related changes (workaround-and-formal-fix or fix-plus-documentation), and defective fixes (incomplete fixes or bug-introducing fixes). It further shows that these categories have distinct characteristics — for example, cross-branch fixes often have large code divergence, and 31.7% of multi-patch fixes take more than a day between patc

What carries the argument

The load-bearing mechanism is the taxonomy itself, built from a manual, consensus-based analysis of 1,646 multi-patch fix records. The taxonomy divides multi-patch fixes into three primary categories — multi-location fixes, fix-and-related changes, and defective fixes — and each into two subcategories (A1 cross-branch/project, A2 same-branch multi-location; B1 workaround-and-formal-fix, B2 fix-and-documentation; C1 incomplete fix, C2 bug-introducing fix). This categorization not only organizes the observed phenomena but also defines the evaluation datasets used to test detection methods. The paper then measures the performance of existing detectors on these categorized cases, using standard

Load-bearing premise

The entire analysis rests on the assumption that the vulnerability database's patch labels and the parsed commit IDs accurately and completely identify the set of fix-related commits for each CVE; if those labels are missing or include unrelated commits, the taxonomy and all downstream evaluations inherit that bias.

What would settle it

A definitive test would be an independent audit of a random sample of CVEs using repository commit history and developer interviews; if a substantial fraction of records labeled as multi-patch turn out to contain only one security-relevant fix (with the others being documentation or test commits), or if many true multi-patch fixes are never recorded in the database, the paper's prevalence estimates and category proportions would need to be revised.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Vulnerability datasets built by reverting patches are likely contaminated: reversing only the final patch of a multi-patch fix yields an intermediate state that is neither fully vulnerable nor fully fixed.
  • Patch users should apply multi-patch fixes sequentially in commit order, and must not assume that the first patch is sufficient; 31.7% of multi-patch fixes have gaps exceeding one day, during which systems remain partially exposed.
  • Existing deep-learning vulnerability detectors, even those with high accuracy on single-patch benchmarks, cannot be trusted to determine whether an initial patch completely remediates a vulnerability; all six tested models scored below 50% accuracy and F1 on incomplete-fix detection.
  • Clone-based detectors are also insufficient for multi-location fixes: they miss many recurring vulnerabilities at other code sites, meaning that first patches can lull maintainers into a false sense of security.
  • The taxonomy provides a foundation for future tooling: developers could use the category labels to guide patch-application strategies and to warn when a fix looks incomplete or defective.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the NVD-based ground truth is noisy (the paper itself notes 5.8% unrelated patches and possible silent fixes), the true prevalence of multi-patch fixes may be even higher than the 1,646 records suggest, and the category distribution could shift once missing patches are recovered.
  • A promising testable extension is to build a sequence-aware model that takes the full patch history into account, rather than classifying each commit in isolation; the paper's taxonomy could serve as the label space for such a model.
  • The finding that even senior committers introduce defective fixes suggests that code-review processes should incorporate automated checks for return-value handling and type mismatches, which were the root causes in the paper's examples.
  • The taxonomy may generalize beyond security patches: similar multi-commit patterns (workaround-then-fix, incomplete corrections) likely occur for ordinary bug fixes in large projects, so the methodology could be reapplied to non-security maintenance commits.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents an empirical study of multi-patch vulnerability fixes in open-source software, constructed from 1,646 NVD records that list multiple patch URLs for a single CVE. The authors manually classify these fixes into three primary categories and six subcategories (multi-location fixes, fix-and-related changes, defective fixes), analyze their characteristics relative to single-patch fixes, and evaluate seven vulnerability detection models and two vulnerable-code-clone tools on tasks derived from the multi-patch dataset. The central claims are that multi-patch fixes are common, have identifiable root causes, and that existing detection methods perform poorly when the initial patch is incomplete or when the vulnerability recurs at multiple locations.

Significance. If the taxonomy and evaluation are valid, this is a useful contribution: it provides a manually labeled dataset, a reasonably granular categorization of an important but under-studied phenomenon, and concrete evidence that current vulnerability detection and clone detection tools degrade substantially on incomplete-fix and multi-location tasks. The manual analysis by three experienced researchers with a maintained codebook, the public dataset release, and the use of externally anchored NVD records are strengths. The paper opens a practical direction: developing methods to predict whether an initial patch will require follow-up patches. However, the study's evidentiary value depends heavily on the completeness and correctness of NVD patch lists, and several numerical inconsistencies in the descriptive tables need to be resolved before the results can be taken at face value.

major comments (4)
  1. [§3, §5, §7.1] The entire dataset and downstream evaluation rest on the assumption that the NVD patch URLs, after commit-ID de-duplication, are a complete and accurate set of fix-related commits per CVE. The paper itself provides evidence that this assumption fails: §5 reports that 5.8% of multi-patch records contain unrelated patches, CVE-2022-2522's record includes a parent of the real patch, and CVE-2023-40173 has an unrecorded prerequisite commit. §9 concedes that silent fixes are missing. Section 7.1 labels code after the 'final fix' as non-vulnerable under the assumption that the last NVD-listed patch is the true remediation. If the true fix sequence extends beyond the NVD list, these labels are false and directly contaminate Table 6's TPR/TNR/F1. The manual pass mitigates but cannot fully repair this. I request validation against independent sources (OSV, GitHub Security Advisories, vendor relea
  2. [§4, Table 1] The manual taxonomy is the core contribution, but no inter-rater reliability statistic is reported. The text states that three researchers independently coded samples and resolved disagreements by consensus, yet without a kappa/agreement metric the reader cannot assess the stability of the categories. The multi-label issue also needs explicit handling: Table 1 sums to 1,764 for 1,646 CVEs, and Table 4 sums to 1,515, neither of which is reconciled. Please report per-annotator agreement on a held-out sample, the multi-label assignment rule, and consistent denominators for all category-level statistics.
  3. [Table 2, Table 3] Several descriptive statistics are internally inconsistent. In Table 2, the multi-patch language percentages sum to 104.63%, and the project percentages are incompatible with the stated totals: linux has 1,415 single-patch fixes (10.10%) and 84 multi-patch fixes (2.08%), which imply denominators of about 14,010 and 4,038, respectively, not 23,467 and 1,646. In Table 3, the top-10 CWE counts sum to 1,644, nearly equal to the entire multi-patch corpus of 1,646, which is implausible unless the dataset is nearly single-CWE. Please correct these tables and report exact denominators and overlap handling.
  4. [§7.1, Table 6] The headline claim that 'all models achieve accuracy and F1 below 50%, which is worse than random guessing' is not well supported. F1 below 50 is not a meaningful random-guessing threshold, and accuracy should be compared to a majority-class baseline (on the test set, always predicting non-vulnerable gives ~43% accuracy, so some models exceed it). The text also states TPR drops of 7.17–28.38 percentage points, but VulBERTa's drop in the table is 3.10. The experimental setup also lacks training hyperparameters and an explicit description of how the single-patch comparison was constructed. Please re-express the results with appropriate baselines and report precision/recall and the comparison protocol.
minor comments (4)
  1. [Fig. 6] The right-hand patch for CVE-2012-0038 contains 'foreach ($fileList as $n $file)' and a 'file_put_contents' call, which appears to be a copy-paste error from the PHP example in Fig. 4. The figure should be regenerated.
  2. [§5] The claim of over-reliance on vendor committers ('almost all... founder, core developer, or early collaborator') is not backed by any quantitative data. If this is a manual observation, report counts; if it is a hypothesis, label it as such. The causal link to post-2020 CNA rule changes is speculative.
  3. [Fig. 10] The two y-axes (distance and complexity) are not labeled clearly, and the relationship between 'Patches distance' and 'Modified lines distance' is not defined. Please add a legend and define the exact metric.
  4. [§7.2] The clone detection experiment uses only 252 signatures and 266 test samples; it would help to report the overlap between A1 and A2, the criteria for choosing one patch as the fingerprint source, and whether the negative samples are temporally aligned with the actual fix sequence.

Circularity Check

0 steps flagged

No significant circularity: the paper is an empirical taxonomy and benchmark study built on external NVD patch records, with self-citations only for context.

full rationale

The paper's central contribution is a manually derived taxonomy (Section 4) and an empirical evaluation of existing detection tools (Section 7), anchored to externally maintained NVD/CVE patch records. The multi-patch dataset is defined by counting distinct commit IDs parsed from NVD 'patch' URLs (Section 3); this is an input definition, not a derived prediction, and the taxonomy is an interpretive classification of those records rather than something fitted to force a result. The evaluation in RQ3 is a benchmark construction: models are trained on standard single-patch data and tested on code states around the final recorded fix. The reported performance drops are empirical measurements of tool behavior on that externally grounded test set, not quantities defined in terms of a fitted parameter. References to the authors' own prior work (GraphSPD, PatchRNN) appear only in related-work summaries and are not load-bearing for any new claim. The paper itself flags limitations about NVD label fidelity (Section 5: 5.8% unrelated patches, CVE-2022-2522 parent-of-fix, CVE-2023-40173 missing prerequisite; Section 9: silent fixes may be missing). These are data-quality and construct-validity threats, but they do not make the derivation circular: the study's conclusions could be wrong or biased without the reasoning reducing to its own inputs. No prediction is defined in terms of the values it claims to predict, no fitted quantity is renamed as a prediction, and no uniqueness result is imported from the authors' prior work. Therefore the appropriate circularity score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No new entities are postulated. The paper's central assumptions are about the reliability of NVD patch metadata and the chosen manual codebook. There are no free numeric parameters fitted to data; the only parameters are curation choices such as the 'more than one day' interval threshold in Table 4, which are not adjusted to produce a specific outcome.

axioms (4)
  • domain assumption NVD 'patch' labels and parsed Git commit IDs accurately identify all fix-related commits for each CVE.
    Section 3: data collection relies on NVD JSON feeds and commit URL parsing; if labels are wrong or incomplete, the dataset is biased. The authors themselves note 5.8% unrelated patches and silent fixes.
  • ad hoc to paper The manual taxonomy (3 primary categories, 6 subcategories) is exhaustive and correctly applied to all 1,646 samples.
    Section 4: the codebook was developed from 100 samples by three researchers; no inter-rater reliability metric is reported, and categories are not mutually exclusive.
  • domain assumption For C1 (incomplete fixes), code before the final patch is vulnerable and code after it is non-vulnerable.
    Section 7.1: this labeling is used to train and evaluate VD models; if the final patch is also defective or the intermediate state is not truly vulnerable, the benchmark is flawed.
  • domain assumption The selected VD and VCC tools are representative of 'existing vulnerability detection methods.'
    Section 7: six VD models and two clone detectors are chosen; the paper excludes LLMs due to data leakage concerns, but other classes of methods are not covered.

pith-pipeline@v1.3.0-alltime-deepseek · 16993 in / 7984 out tokens · 95099 ms · 2026-08-02T05:51:51.867174+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Why Not Fix It Once and for All? An Empirical Study of Multiple Patches for Vulnerability Fixes in Open-Source Software." pith.science (2026). https://pith.science/paper/K332DUSD

@misc{pith2026260713206,
  author       = {Pith},
  title        = {Pith review of: Why Not Fix It Once and for All? An Empirical Study of Multiple Patches for Vulnerability Fixes in Open-Source Software},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K332DUSD}},
  note         = {Machine review of arXiv:2607.13206}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Security patches for open-source software constitute a foundational resource for vulnerability remediation research and practice. However, analyzing and applying multiple patches remains challenging, especially when trying to determine at what point in a patch sequence a vulnerability is fully remediated. This paper presents a systematic analysis of multi-patch vulnerability fixes, focusing on their root causes, characteristics, and methods for verifying remediation status throughout the fixing process. Through a manual examination of 1,646 multi-patch fix records, we develop a taxonomy with three primary categories and six subcategories based on their causes. We then compare the distinctive characteristics of multi-patch fixes with those of single-patch fixes and analyze feature variations across categories. In addition, we assess representative vulnerability detection methods for validating complete remediation during multi-patch fixing. Our findings provide new insights into multi-patch fixes and lay a foundation for future research in this field.

Figures

Figures reproduced from arXiv: 2607.13206 by Weiliang Qi, Xinda Wang, Youpeng Li.

Figure 1
Figure 1. Figure 1: The vendor’s advisory on applying security patches for CVE-2021-32648 3 Data Collection To understand why multiple patches are needed to address a single vulnerability and how this affects practical vulnerability management, we first construct a large-scale dataset of multi-patch vulnerabilities based on CVE records. We use the NVD JSON feeds API to collect all published CVEs as of May 2025. We then focus … view at source ↗
Figure 2
Figure 2. Figure 2: Patches for CVE-2022-25858: multi-location fixes across different branches [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: illustrates three commits for patching CVE-2012-6537 caused by unini￾tialized structures that lead to an information leak. Each patch adds explicit ini￾tialization with memset(0) to a different function containing similar vulnerable code, although all three functions are located in same file. While three commits use same initialization pattern, their parameters and code contexts differ. 1 @@ -689,6 +689,7 … view at source ↗
Figure 4
Figure 4. Figure 4: shows a workaround followed by a formal fix for CVE-2023-4226, an ar￾bitrary file upload vulnerability in Chamilo LMS that can enable command ex￾ecution, shell access, and stored XSS. The workaround patch (left) mitigates immediate exploitation by restricting browser parsing of risky files and blocking direct access to upload directories. The formal fix (right), released 16 days later, uses disable_dangero… view at source ↗
Figure 5
Figure 5. Figure 5: Patches for CVE-2020-13397: fix and documents. Insights for Applying Fix and Related Change Patches. Distinguishing related patches from actual vulnerability-fix patches is important for both prac￾tice and research. For software users, workarounds can reduce risk but may also disrupt normal functionality, such as blocking access to a vulnerable API. Once a formal fix is available, continued reliance on a w… view at source ↗
Figure 6
Figure 6. Figure 6: presents CVE-2012-0038, which involves an integer overflow in count in the Linux XFS file system. The first patch (left), fa8b18ed, added a maximum￾value check and exception handling. However, it overlooked that be32_to_cpu() returns a 32-bit unsigned integer, while count was declared as signed. As a result, count could be interpreted as negative in some cases and bypass the check. The vendor later release… view at source ↗
Figure 7
Figure 7. Figure 7: are patches for CVE-2018-7191 in the Linux kernel, insufficient valida￾tion of the device name before calling register_netd allows a device with ille￾gal characters to trigger a NULL pointer dereference and kernel panic, causing a denial of service. The initial patch (left) attempted to fix this by adding a device-name check. However, the developer misinterpreted the return value of dev_get_valid_name(), a… view at source ↗
Figure 8
Figure 8. Figure 8: A discussion between OSS vendor and CNA nerability type, and publication year. We then analyze the time intervals and similarities across different categories of multi-patch fixes. Language Distribution [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The number of single and multi-patch fixes over time Time Interval Between Patches [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Patch similarity and complexity across multi-patch fixes. 7 Management of Multi-Patch Fixes Given that some vulnerabilities are not fully remediated by a single patch (§4), an important practical question is whether an initial patch has completely fixed the vulnerability. As summarized in [PITH_FULL_IMAGE:figures/full_fig_p013_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 5 linked inside Pith

  1. [1]

    In: Proceedings of the 44th international conference on software engineering

    Bao, L., Xia, X., Hassan, A.E., Yang, X.: V-szz: automatic identification of version ranges affected by cve vulnerabilities. In: Proceedings of the 44th international conference on software engineering. pp. 2352–2364 (2022)

    2022

  2. [2]

    In: Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering

    Bhandari, G., Naseer, A., Moonen, L.: Cvefixes: automated collection of vulner- abilities and their fixes from open-source software. In: Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering. pp. 30–39 (2021)

    2021

  3. [3]

    In: Proceedings of the 2017 ACM SIGSAC conference on computer and communications security

    Böhme, M., Pham, V.T., Nguyen, M.D., Roychoudhury, A.: Directed greybox fuzzing. In: Proceedings of the 2017 ACM SIGSAC conference on computer and communications security. pp. 2329–2344 (2017)

    2017

  4. [4]

    Chakraborty, S., Krishna, R., Ding, Y., Ray, B.: Deep learning based vulnerability detection: Are we there yet? IEEE Transactions on Software Engineering48(9), 3280–3296 (2021)

    2021

  5. [5]

    In: Proceedings of the 26th International Symposium on Research in Attacks, Intru- sions and Defenses

    Chen, Y., Ding, Z., Alowain, L., Chen, X., Wagner, D.: Diversevul: A new vul- nerable source code dataset for deep learning based vulnerability detection. In: Proceedings of the 26th International Symposium on Research in Attacks, Intru- sions and Defenses. pp. 654–668 (2023)

    2023

  6. [6]

    Pearson Education (2007)

    Chess, B., West, J.: Secure programming with static analysis. Pearson Education (2007)

    2007

  7. [7]

    In: 2023 IEEE/ACM 45th International Conference on Software Engi- neering (ICSE)

    Croft, R., Babar, M.A., Kholoosi, M.M.: Data quality for software vulnerability datasets. In: 2023 IEEE/ACM 45th International Conference on Software Engi- neering (ICSE). pp. 121–133. IEEE (2023)

    2023

  8. [8]

    https://github.com/CVEProject/cve- documents/blob/master/archive/CNA_Rules_v2.0.pdf

    CVEProject: CNA_Rules_v2.0. https://github.com/CVEProject/cve- documents/blob/master/archive/CNA_Rules_v2.0.pdf

  9. [9]

    https://github.com/CVEProject/cve- documents/blob/master/archive/CNA_Rules_v3.0.pdf

    CVEProject: CNA_Rules_v3.0. https://github.com/CVEProject/cve- documents/blob/master/archive/CNA_Rules_v3.0.pdf

  10. [10]

    https://xdg.me/a-survey-of-git-best- practices/

    David Golden: A survey of git best practices. https://xdg.me/a-survey-of-git-best- practices/

  11. [11]

    Ding, Y., Fu, Y., Ibrahim, O., Sitawarin, C., Chen, X., Alomair, B., Wagner, D., Ray, B., Chen, Y.: Vulnerability detection with code language models: How far are we? arXiv preprint arXiv:2403.18624 (2024)

    Pith/arXiv arXiv 2024

  12. [12]

    In: Proceedings of the 19th ACM Asia Conference on Computer and Communications Security

    Dunlap, T., Lin, E., Enck, W., Reaves, B.: Vfcfinder: Pairing security advisories and patches. In: Proceedings of the 19th ACM Asia Conference on Computer and Communications Security. pp. 1128–1142 (2024) 18 W. Qi et al

    2024

  13. [13]

    In: Proceedings of the 17th international confer- ence on mining software repositories

    Fan, J., Li, Y., Wang, S., Nguyen, T.N.: Ac/c++ code vulnerability dataset with code changes and cve summaries. In: Proceedings of the 17th international confer- ence on mining software repositories. pp. 508–512 (2020)

    2020

  14. [14]

    In: 33rd USENIX Security Symposium (USENIX Security 24)

    Feng, S., Wu, Y., Xue, W., Pan, S., Zou, D., Liu, Y., Jin, H.:{FIRE}: combining {Multi-Stage}filteringwithtaintanalysisforscalablerecurringvulnerabilitydetec- tion. In: 33rd USENIX Security Symposium (USENIX Security 24). pp. 1867–1884 (2024)

    2024

  15. [15]

    arXiv preprint arXiv:2002.08155 (2020)

    Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al.: Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020)

    Pith/arXiv arXiv 2002

  16. [16]

    In: 14th USENIX workshop on offensive technologies (WOOT 20) (2020)

    Fioraldi, A., Maier, D., Eißfeldt, H., Heuse, M.:{AFL++}: Combining incremental steps of fuzzing research. In: 14th USENIX workshop on offensive technologies (WOOT 20) (2020)

    2020

  17. [17]

    In: Proceedings of the 19th International Conference on Mining Software Repositories

    Fu, M., Tantithamthavorn, C.: Linevul: A transformer-based line-level vulnera- bility prediction. In: Proceedings of the 19th International Conference on Mining Software Repositories. pp. 608–620 (2022)

    2022

  18. [18]

    Gu, Z., Barr, E.T., Hamilton, D.J., Su, Z.: Has the bug really been fixed? In: Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1. pp. 55–64 (2010)

    2010

  19. [19]

    arXiv preprint arXiv:2203.03850 (2022)

    Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., Yin, J.: Unixcoder: Unified cross- modal pre-training for code representation. arXiv preprint arXiv:2203.03850 (2022)

    Pith/arXiv arXiv 2022

  20. [20]

    In: 2022 International joint conference on neural networks (IJCNN)

    Hanif, H., Maffeis, S.: Vulberta: Simplified source code pre-training for vulnerabil- ity detection. In: 2022 International joint conference on neural networks (IJCNN). pp. 1–8. IEEE (2022)

    2022

  21. [21]

    Proceedings of the ACM on Measurement and Analysis of Computing Systems 4(3), 1–29 (2020)

    Hazimeh, A., Herrera, A., Payer, M.: Magma: A ground-truth fuzzing benchmark. Proceedings of the ACM on Measurement and Analysis of Computing Systems 4(3), 1–29 (2020)

    2020

  22. [22]

    In: 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)

    Hoang, T., Dam, H.K., Kamei, Y., Lo, D., Ubayashi, N.: Deepjit: an end-to-end deep learning framework for just-in-time defect prediction. In: 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR). pp. 34–45. IEEE (2019)

    2019

  23. [23]

    In: 2024 IEEE Symposium on Security and Privacy (SP)

    Huang, H., Yao, P., Chiu, H.C., Guo, Y., Zhang, C.: Titan: efficient multi-target directed greybox fuzzing. In: 2024 IEEE Symposium on Security and Privacy (SP). pp. 1849–1864. IEEE (2024)

    2024

  24. [24]

    IEEE Transactions on Software Engineering49(1), 44–63 (2022)

    Iannone, E., Guadagni, R., Ferrucci, F., De Lucia, A., Palomba, F.: The secret life of software vulnerabilities: A large-scale empirical study. IEEE Transactions on Software Engineering49(1), 44–63 (2022)

    2022

  25. [25]

    In: 2012 IEEE Symposium on security and privacy

    Jang, J., Agrawal, A., Brumley, D.: Redebug: finding unpatched code clones in entire os distributions. In: 2012 IEEE Symposium on security and privacy. pp. 48–62. IEEE (2012)

    2012

  26. [26]

    In: 32nd USENIX Security Symposium (USENIX Security 23)

    Kim, H., Ozmen, M.O., Celik, Z.B., Bianchi, A., Xu, D.:{PatchVerif}: Discovering faulty patches in robotic vehicles. In: 32nd USENIX Security Symposium (USENIX Security 23). pp. 3011–3028 (2023)

    2023

  27. [27]

    In: 2017 IEEE symposium on security and privacy (SP)

    Kim, S., Woo, S., Lee, H., Oh, H.: Vuddy: A scalable approach for vulnerable code clone discovery. In: 2017 IEEE symposium on security and privacy (SP). pp. 595–614. IEEE (2017)

    2017

  28. [28]

    In: Pro- ceedings of the 2018 ACM SIGSAC conference on computer and communications security

    Klees, G., Ruef, A., Cooper, B., Wei, S., Hicks, M.: Evaluating fuzz testing. In: Pro- ceedings of the 2018 ACM SIGSAC conference on computer and communications security. pp. 2123–2138 (2018)

    2018

  29. [29]

    In: Soviet physics doklady

    Levenshtein, V.I., et al.: Binary codes capable of correcting deletions, insertions, and reversals. In: Soviet physics doklady. vol. 10, pp. 707–710. Soviet Union (1966) An Empirical Study of Multiple Patches for Vulnerability Fixes 19

    1966

  30. [30]

    Li,F.,Paxson,V.:Alarge-scaleempiricalstudyofsecuritypatches.In:Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. pp. 2201–2215 (2017)

    2017

  31. [31]

    IEEE Transactions on Dependable and Secure Computing19(4), 2244–2258 (2021)

    Li, Z., Zou, D., Xu, S., Jin, H., Zhu, Y., Chen, Z.: Sysevr: A framework for using deep learning to detect software vulnerabilities. IEEE Transactions on Dependable and Secure Computing19(4), 2244–2258 (2021)

    2021

  32. [32]

    arXiv preprint arXiv:1801.01681 (2018)

    Li, Z., Zou, D., Xu, S., Ou, X., Jin, H., Wang, S., Deng, Z., Zhong, Y.: Vuldeep- ecker: A deep learning-based system for vulnerability detection. arXiv preprint arXiv:1801.01681 (2018)

    Pith/arXiv arXiv 2018

  33. [33]

    arXiv preprint arXiv:2102.04664 (2021)

    Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al.: Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664 (2021)

    Pith/arXiv arXiv 2021

  34. [34]

    Journal of Systems and Soft- ware214, 112014 (2024)

    Nguyen, S., Nguyen, T.T., Vu, T.T., Do, T.D., Ngo, K.T., Vo, H.D.: Code-centric learning-based just-in-time vulnerability detection. Journal of Systems and Soft- ware214, 112014 (2024)

    2024

  35. [35]

    In: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings

    Nguyen, V.A., Nguyen, D.Q., Nguyen, V., Le, T., Tran, Q.H., Phung, D.: Regvd: Revisiting graph neural networks for vulnerability detection. In: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings. pp. 178–182 (2022)

    2022

  36. [36]

    In: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

    Nie, X., Li, N., Wang, K., Wang, S., Luo, X., Wang, H.: Understanding and tackling label errors in deep learning-based vulnerability detection (experience paper). In: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. pp. 52–63 (2023)

    2023

  37. [37]

    https://nvd.nist.gov/

    NIST: NVD - Home. https://nvd.nist.gov/

  38. [38]

    https://nvd.nist.gov/vuln/detail/CVE-2018-8729

    NVD: CVE-2018-8729. https://nvd.nist.gov/vuln/detail/CVE-2018-8729

    2018

  39. [39]

    https://nvd.nist.gov/vuln/detail/CVE-2022-2522

    NVD: CVE-2022-2522. https://nvd.nist.gov/vuln/detail/CVE-2022-2522

    2022

  40. [40]

    https://nvd.nist.gov/vuln/detail/CVE-2023-35852

    NVD: CVE-2023-35852. https://nvd.nist.gov/vuln/detail/CVE-2023-35852

    2023

  41. [41]

    https://nvd.nist.gov/vuln/detail/CVE-2023-40173

    NVD: CVE-2023-40173. https://nvd.nist.gov/vuln/detail/CVE-2023-40173

    2023

  42. [42]

    https://nvd.nist.gov/vuln/detail/cve-2024-1394

    NVD: CVE-2024-1394. https://nvd.nist.gov/vuln/detail/cve-2024-1394

    2024

  43. [43]

    In: 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)

    Pornprasit, C., Tantithamthavorn, C.K.: Jitline: A simpler, better, faster, finer- grained just-in-time defect prediction. In: 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). pp. 369–379. IEEE (2021)

    2021

  44. [44]

    In: Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering

    Ray, B., Posnett, D., Filkov, V., Devanbu, P.: A large scale study of program- ming languages and code quality in github. In: Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering. pp. 155–165 (2014)

    2014

  45. [45]

    Śliwerski, J., Zimmermann, T., Zeller, A.: When do changes induce fixes? ACM sigsoft software engineering notes30(4), 1–5 (2005)

    2005

  46. [46]

    https://git-scm.com/

    Software Freedom Conservancy: Git. https://git-scm.com/

  47. [47]

    In: Proceedings of the ACM Web Conference 2022

    Tan, X., Zhang, Y., Cao, J., Sun, K., Zhang, M., Yang, M.: Understanding the practice of security patch management across multiple branches in oss projects. In: Proceedings of the ACM Web Conference 2022. pp. 767–777 (2022)

    2022

  48. [48]

    In: 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)

    Tang, L., Bao, L., Xia, X., Huang, Z.: Neural szz algorithm. In: 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). pp. 1024–1035. IEEE (2023)

    2023

  49. [49]

    https://www.cve.org/

    The MITRE Corporation: CVE: Common Vulnerabilities and Exposures. https://www.cve.org/

  50. [50]

    com/sogrady/2024/09/12/language-rankings-6-24/(2024) 20 W

    The RedMonk Programming Language Rankings: June 2024:https://redmonk. com/sogrady/2024/09/12/language-rankings-6-24/(2024) 20 W. Qi et al

    2024

  51. [51]

    In: 2023 IEEE Symposium on Security and Privacy (SP)

    Wang,S.,Wang,X.,Sun,K.,Jajodia,S.,Wang,H.,Li, Q.:Graphspd:Graph-based security patch detection with enriched code semantics. In: 2023 IEEE Symposium on Security and Privacy (SP). pp. 2409–2426. IEEE (2023)

    2023

  52. [52]

    In: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering

    Wang, W., Li, Y., Li, A., Zhang, J., Ma, W., Liu, Y.: An empirical study on noisy label learning for program understanding. In: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. pp. 1–12 (2024)

    2024

  53. [53]

    In: 2019 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)

    Wang, X., Sun, K., Batcheller, A., Jajodia, S.: Detecting" 0-day" vulnerability: An empirical study of secret security patch in oss. In: 2019 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN). pp. 485–

    2019

  54. [54]

    In: MIL- COM 2021-2021 IEEE Military Communications Conference (MILCOM)

    Wang, X., Wang, S., Feng, P., Sun, K., Jajodia, S., Benchaaboun, S., Geck, F.: Patchrnn: A deep learning-based system for security patch identification. In: MIL- COM 2021-2021 IEEE Military Communications Conference (MILCOM). pp. 595–

    2021

  55. [55]

    Computers & Security148, 104181 (2025)

    Woo, S., Choi, E., Lee, H.: A large-scale analysis of the effectiveness of publicly reported security patches. Computers & Security148, 104181 (2025)

    2025

  56. [56]

    Wu, Y., Lin, Z., Chen, Y., Le, D.K., Mu, D., Xing, X.: Mitigating security risks in linuxwith{KLAUS}:Amethodforevaluatingpatchcorrectness.In:32ndUSENIX Security Symposium (USENIX Security 23). pp. 4247–4264 (2023)

    2023

  57. [57]

    In: 29th USENIX Security Symposium (USENIX Security 20)

    Xiao, Y., Chen, B., Yu, C., Xu, Z., Yuan, Z., Li, F., Liu, B., Liu, Y., Huo, W., Zou, W., et al.:{MVP}: Detecting vulnerabilities using{Patch-Enhanced}vulnerabil- ity signatures. In: 29th USENIX Security Symposium (USENIX Security 20). pp. 1165–1182 (2020)

    2020

  58. [58]

    https://huntr.com/bounties/3a2d83af-9542-4d93-8784-98b115135a22

    xiaoge1001: Heap-based Buffer Overflow in func- tion ins_compl_infercase_gettext() in vim/vim. https://huntr.com/bounties/3a2d83af-9542-4d93-8784-98b115135a22

  59. [59]

    In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

    Xu, C., Chen, B., Lu, C., Huang, K., Peng, X., Liu, Y.: Tracking patches for open source software vulnerabilities. In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. pp. 860–871 (2022)

    2022

  60. [60]

    In: 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE)

    Xu, Z., Chen, B., Chandramohan, M., Liu, Y., Song, F.: Spain: security patch analysis for binaries towards understanding the pain and pills. In: 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). pp. 462–472. IEEE (2017)

    2017

  61. [61]

    IEEE Transactions on Software Engineering48(1), 82–101 (2020)

    Yan, M., Xia, X., Fan, Y., Hassan, A.E., Lo, D., Li, S.: Just-in-time defect identi- fication and localization: A two-phase framework. IEEE Transactions on Software Engineering48(1), 82–101 (2020)

    2020

  62. [62]

    arXiv preprint arXiv:2510.26086 (2025)

    Zhang, Z., Li, H., Li, X., Zhang, H., Qian, Z.: Llmbisect: Breaking barriers in bug bisection with a comparative analysis pipeline. arXiv preprint arXiv:2510.26086 (2025)

    arXiv 2025

  63. [63]

    In: 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering

    Zhong, H., Su, Z.: An empirical study on real bug fixes. In: 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering. vol. 1, pp. 913–923. IEEE (2015)

    2015

  64. [64]

    Advances in neural information processing systems32(2019)

    Zhou, Y., Liu, S., Siow, J., Du, X., Liu, Y.: Devign: Effective vulnerability identi- fication by learning comprehensive program semantics via graph neural networks. Advances in neural information processing systems32(2019)

    2019

  65. [65]

    ACM Transactions on Software Engineering and Methodology (TOSEM)31(1), 1–27 (2021)

    Zhou, Y., Siow, J.K., Wang, C., Liu, S., Liu, Y.: Spi: Automated identification of security patches via commits. ACM Transactions on Software Engineering and Methodology (TOSEM)31(1), 1–27 (2021)

    2021