Pith. sign in

REVIEW 4 major objections 6 minor 76 references

Toward Realistic Evaluations of Just-In-Time Vulnerability Prediction

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that just-in-time vulnerability prediction models lose 89–98% of their precision-recall performance when evaluated on a realistic, highly imbalanced commit stream rather than on balanced curated vulnerability data.

desk verdict Large-scale, honest empirical study showing JIT-VP models collapse on a realistic commit stream, but the headline 89-98% PR-AUC drop is inflated by the VNC-as-safe labeling assumption and the PR-AUC prevalence effect. read the letter →

arxiv 2507.10729 v2 pith:NYXPLKUV submitted 2025-07-14 cs.SE

Duong Nguyen , Thanh Le-Cong , Triet Huynh Minh Le , M. Ali Babar , Quyet-Thang Huynh This is my paper
classification cs.SE
keywords just-in-timevulnerabilitypredictionsoftwareclassimbalancecommit-levelbenchmarkevaluationprecision-recallAUCvulnerability-introducingcommitsV-SZZ
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that published evaluations of just-in-time vulnerability prediction (JIT-VP) are unrealistically favorable because they restrict the test set to vulnerability-introducing and vulnerability-fixing commits, leaving out the roughly 90–99% of commits that are neither. To test this, the authors built a dataset of 1,081,882 commits from FFmpeg and the Linux kernel and evaluated eight state-of-the-art predictors under the standard 'idealized' setting and a new 'realistic' setting that adds vulnerability-neutral commits. In the idealized setting the models average a precision-recall AUC (PR-AUC) of 0.833 on FFmpeg and 0.805 on Linux; in the realistic setting those averages fall to 0.091 and 0.016, drops of roughly 89% and 98%. The paper then shows that five common imbalance-mitigation techniques—focal loss, random oversampling, random undersampling, SMOTE, and OSS—fail to meaningfully recover the lost performance. The reason to care is that if this is right, current JIT-VP models are not ready for practical deployment, and the benchmark design itself, not the models, is driving the reported success.

What carries the argument

The central mechanism is the 'realistic setting': an evaluation protocol in which the test set contains every commit from a project's history—vulnerability-introducing, vulnerability-fixing, and vulnerability-neutral—rather than only vulnerability-related commits. Its load-bearing component is the residual labeling rule: after vulnerability-fixing commits are collected from curated datasets and CVE/NVD records, and vulnerability-introducing commits are traced with V-SZZ and Linux 'Fixes:' tags, all remaining commits are labeled 'vulnerability-neutral' and treated as safe. That single design choice changes the class ratio from roughly balanced (1:0.89 and 1:0.88 vulnerable-to-safe) to severely imbalanced (about 1:17 and 1:127–1:217), and it is what turns PR-AUC from values above 0.8 into values near zero.

What would settle it

Take a random sample of the vulnerability-neutral commits from the released dataset, apply an independent oracle such as later 'Fixes:' references or manual security review, count how many are actually vulnerability-introducing, and recompute PR-AUC with corrected labels; if the corrected positive rate differs substantially from the paper's, the 89–98% decline is not robust to labeling error.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the strong results previously reported for JIT-VP are an artifact of excluding vulnerability-neutral commits (VNCs) from training and test sets. The authors label vulnerability-introducing commits as vulnerable, label vulnerability-fixing and vulnerability-neutral commits as safe, and evaluate eight existing predictors on a dataset of 1,081,882 commits. In the idealized setting, average PR-AUC is 0.833 (FFmpeg) and 0.805 (Linux), with per-model best values of 0.959 and 0.892; in the realistic setting, the averages collapse to 0.091 and 0.016, and even the best idealized model JITFine falls to 0.111 and 0.005. The paper attributes the collapse to the shift in the vulnerable-to-safe ratio from nearly balanced (1:0.89 and 1:0.88) to severely imbalanced (about 1:17 in FFmpeg and between 1:127 and 1:217 in Linux), which biases models toward predicting the majority safe class. It further reports that imbalance mitigation through focal loss, random sampling, SMOTE, or OSS does not restore performance, and in some cases makes it worse.

Load-bearing premise

The whole realistic setting rests on labeling every commit not identified as vulnerability-introducing or vulnerability-fixing as 'vulnerability-neutral' and therefore safe; if a non-trivial share of those commits actually contain latent vulnerabilities, the ground truth, the imbalance ratios, and the measured performance drops all change.

Editorial extensions

If this is right

  • Published JIT-VP performance numbers from balanced benchmarks (PR-AUC above 0.8) should not be treated as deployable: under the realistic setting the same models average 0.091 and 0.016 PR-AUC.
  • Model rankings change under the realistic setting; JITFine, the best model in the idealized setting, is no longer best, so future evaluations should compare models on realistic, VNC-inclusive commit streams.
  • Standard imbalance fixes do not solve the problem; random undersampling gives only marginal and inconsistent gains, focal loss can drive MCC and F1 to zero, and SMOTE/OSS leave the models far below idealized performance.
  • ROC-AUC hides most of the damage, declining only 16–25% while PR-AUC falls 89–98%, so PR-AUC and MCC are the appropriate metrics for this setting.
  • The released dataset of over one million commits gives the community a public benchmark for evaluating JIT-VP under a realistic distribution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: any commit-level prediction task that curates negatives to match known positives—such as defect prediction or fix prediction—may be overstating practical performance in the same way; the structure of the negative set, not the model alone, sets the score.
  • Beyond the paper: because the drop is driven by a change in the base rate, reweighting or resampling the same features cannot recover the lost precision; the results point toward adding vulnerability-specific signal (code semantics, patch context, or external knowledge) rather than further class-balance tuning.
  • Beyond the paper: a controlled sweep of the fraction of vulnerability-neutral commits (0%, 50%, 90%, 99%) could separate 'mere imbalance' from 'distributional shift'—a gradual decline would suggest threshold-based deployment is still viable, while an abrupt collapse would confirm that the models have not learned vulnerability indicators at all.
  • Beyond the paper: the paper treats vulnerability-fixing commits as safe; re-annotating VFCs as 'vulnerability-related but not vulnerable' in a follow-up experiment would test whether part of the precision loss is a false-alarm penalty on exactly the commits reviewers should inspect.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper argues that existing just-in-time vulnerability prediction (JIT-VP) evaluations are unrealistic because they train and test only on vulnerability-introducing commits (VICs) and vulnerability-fixing commits (VFCs), omitting the large majority of ordinary commits. The authors construct a dataset of over one million commits from FFmpeg and the Linux kernel, label the residual commits as vulnerability-neutral commits (VNCs), and evaluate eight JIT-VP models in an idealized setting (VICs vs. VFCs) and a realistic setting (VICs vs. VFCs plus VNCs). They report that average PR-AUC drops by roughly 89-98% in the realistic setting, that class imbalance is the primary cause, and that five common imbalance mitigation techniques (focal loss, random oversampling, random undersampling, SMOTE, OSS) fail to restore performance. The paper concludes that current JIT-VP models are not ready for practical deployment and calls for domain-specific imbalance solutions.

Significance. The dataset and replication package are substantial contributions, and the core question—whether JIT-VP results transfer from balanced VIC/VFC benchmarks to full commit streams—is important for the field. The paper is one of the first to quantify this gap at scale, and its negative results on off-the-shelf imbalance mitigation techniques are a useful caution for practitioners. However, the headline quantitative claim is overstated because PR-AUC is prevalence-dependent: comparing PR-AUC across settings with very different base rates conflates a change in class prevalence with a change in model skill. The residual VNC labeling also injects potentially large label noise into the realistic setting. The paper's strengths—public data, reproducible baselines, broad model coverage—make the underlying question worth pursuing, but the central claim needs reframing and additional analysis before it can be accepted at face value.

major comments (4)
  1. [Section IV-B, Tables VI and VII] PR-AUC is a prevalence-dependent metric: for a random ranking, expected PR-AUC equals the positive-class prevalence. In the idealized test sets the positive prevalence is about 0.35, whereas in the realistic test sets it is approximately 0.025 for FFmpeg and 0.0058 for Linux. The reported drops from 0.833 to 0.091 and from 0.805 to 0.016 therefore partly reflect the change in base rate rather than a loss of ranking ability. Relative to the random baseline, the models are not 90% worse: the realistic FFmpeg PR-AUC of 0.091 is about 3.6 times the prevalence baseline, and the Linux value of 0.016 is about 2.8 times its baseline, while the idealized PR-AUCs are only about 2.4 times their baseline. The paper should re-express the central decline using prevalence-normalized quantities, such as lift over random, precision at fixed recall, or the ROC-AUC values already reported, and separate the ranking and threshold components of the drop.
  2. [Table I] Table I is internally inconsistent, which undermines the paper's stated imbalance ratios. For Linux, the training row sums to 802,161 rather than 800,426; the validation row sums to 35,933 rather than 35,317; and the testing row sums to 159,652 rather than 157,961. The 'All Commits' row also fails to sum: 9,715 + 8,996 + 1,069,933 = 1,088,644, not 1,081,882, and the partition totals for VFCs imply 8,794 VFCs, not 8,996. Separately, the vulnerable-to-safe ratios quoted in the text are inconsistent with the table: Section I states ratios of 1:17 and 1:217, while Section IV-B states 1:17 and 1:127; the test-set ratios computed from the printed counts are about 1:39 for FFmpeg and 1:172 for Linux. Since class imbalance is the paper's main explanatory mechanism, the authors must correct the table and reconcile all stated ratios.
  3. [Sections III-B and V-B] The residual labeling of all non-VFC/non-VIC commits as 'vulnerability-neutral' is load-bearing for the realistic-setting results. The test sets contain 157,039 (Linux) and 37,778 (FFmpeg) VNCs against only 922 and 1,020 known VICs, so even a small percentage of mislabeled latent vulnerabilities among VNCs would change both the positive count and the shape of the PR curve. The paper concedes in Section V-B that latent vulnerabilities are possible but dismisses the impact with the claim that 'a small number of mislabeled instances is unlikely to substantially impact the overall empirical findings' without providing any quantification. The authors should provide a sensitivity analysis—for example, re-estimating PR-AUC and MCC under plausible latent-positive contamination rates (1%, 5%, 10%) using the model's score distribution—or otherwise bound the effect of label noise. Without such an analysis, the magnitude of the realistic-setting performance drop is not established.
  4. [Table VIII] The RQ3 results contain an internal inconsistency that weakens the 'mitigation is ineffective' conclusion. In Table VIII, DeepJIT with random undersampling achieves PR-AUC 0.237 on FFmpeg and 0.160 on Linux, yet its MCC is 0.004 and 0.002 and its F1-score is 0.051 and 0.012. This pattern indicates that the model ranks positives reasonably well but produces useless default-threshold predictions. The paper reports these numbers together without explaining the discrepancy. Since PR-AUC and MCC/F1 measure different properties (ranking versus thresholded classification), the authors should specify the decision rule used for MCC/F1, discuss the calibration failure, and avoid presenting PR-AUC gains and threshold-based collapses as equally valid evidence for the same conclusion.
minor comments (6)
  1. [Table I and Figure 1] There are typographical errors: 'vulnerability-neural' in Table I's header and 'vulnerability-introducting'/'vulnerability-neural' in Figure 1's caption should be 'vulnerability-neutral' and 'vulnerability-introducing'.
  2. [Section V-A] The statement that applying RUS to DeepJIT increases PR-AUC 'from 0.114 to 0.237' on FFmpeg does not match Table VII, which lists DeepJIT's no-sampling FFmpeg PR-AUC as 0.082; please correct the baseline value.
  3. [Section III-C] PR-AUC and MCC are described as 'two new evaluation metrics,' but both are standard in imbalanced classification; suggest rephrasing to 'additional evaluation metrics adopted for imbalanced settings.'
  4. [Section III-B] The selection of the two anchored VICs (VIC-1 and VIC-2) is underspecified; please state how these anchors were chosen and confirm that the resulting time-wise split is strictly chronological and free of leakage.
  5. [Section IV-C] The focal loss experiments specify only gamma=2 and do not report the alpha_t class-balance coefficient; please provide the full focal loss configuration used for DeepJIT and SimCom.
  6. [Section III-B] The paper reports that FFmpeg and Linux have approximately 117,000 and 1,300,000 commits but the curated dataset contains 88,178 and 993,704 commits; please quantify the loss from the filtering steps so readers can assess representativeness.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor definitional circularity: the 'realistic' imbalance and the headline PR-AUC drop depend on the paper's own residual VNC-as-safe labeling, but the model evaluations themselves are genuinely empirical and not fitted to the result.

  1. self definitional [Section II-B (setting definitions) and Section III-B.2 (VNC labeling); acknowledged in Section V-B]
    "The remaining commits were then flagged as VNCs because there is no perfect security testing or labeling in practice, and we could only assume that those commits do not introduce or fix vulnerabilities, as per current practice... In the realistic setting, the evaluation dataset comprises VICs, VFCs, and VNCs. Within this dataset, VICs are labeled as vulnerable commits, while both VFCs and VNCs are categorized as safe commits."

    The 'realistic' test distribution is not an independently verified oracle: VNCs are defined as the residual commits after removing known VICs and VFCs, and then all VNCs are labeled safe. The imbalance ratios (1:17 and 1:127/1:217) and the reported 89-98% PR-AUC declines are computed against this self-defined label set. If a non-trivial fraction of VNCs contain latent vulnerabilities, those commits would be scored as positives by a good model but counted as false positives, directly depressing precision and PR-AUC. The paper's V-B response—'the heuristic labeling approach may fail to capture latent vulnerabilities... a small number of mislabeled instances is unlikely to substantially impact the overall empirical findings'—is an unquantified assumption, not a measurement.

full rationale

This is an empirical study with no equation-level derivation: the eight JIT-VP models are pretrained artifacts evaluated on held-out commit splits, and their PR-AUC/MCC values are genuinely computed from model outputs. There is no fitted parameter that is later renamed as a prediction, and no load-bearing self-citation chain; citations of the authors' prior work (e.g., [56], [72]) are used for metrics and for acknowledging latent-vulnerability concerns, not to justify the central result. The one defensible circularity concern is definitional: the 'realistic setting' is constructed by labeling all residual VNCs as safe, and the resulting class imbalance directly produces the large PR-AUC decline. Because the paper itself concedes that latent vulnerabilities in VNCs are possible but does not quantify their effect, the headline '89-98% decline' is partly an artifact of the paper's own labeling definition. That warrants a low score of 2 rather than 0, but the core empirical measurements of the eight models in the constructed setting are internally consistent and not fitted to the conclusions.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central empirical results rest on a self-constructed labeling of VNCs as safe, on SZZ-based VIC identification, and on CVE-derived VFC lists. The study is empirical, so no equation-level circularity exists, but the 'realistic' distribution is an artifact of these label choices.

free parameters (3)
  • Focal loss gamma = 2
    Gamma in focal loss was fixed at 2 without tuning; the conclusion that focal loss is ineffective applies only to this configuration. (Section IV-C.1)
  • Resampling ratio = 1:1
    RUS, ROS, SMOTE, and OSS all resample to a 1:1 positive-to-negative ratio; other ratios were not explored. (Section IV-C.1)
  • VIC split anchors VIC-1 and VIC-2 = not specified
    The training/validation/test split boundaries are chosen by selecting two anchored VICs to enforce a 75/5/20 split of VICs, introducing a hand-picked temporal split. (Section III-B.2)
assumptions (4)
  • domain assumption VNCs are non-vulnerable (safe)
    The realistic setting labels all commits not identified as VIC or VFC as safe; the paper acknowledges latent vulnerabilities may exist. (Section III-B.2, Section V-B)
  • domain assumption V-SZZ and 'Fixes:' tags correctly identify vulnerability-introducing commits
    VIC labels rely on V-SZZ and Linux developer 'Fixes:' annotations; SZZ is known to have false positives. (Section III-B.2, Section II-B)
  • domain assumption CVE/NVD and prior datasets contain reliable vulnerability-fixing commits
    VFCs are sourced from Liu et al., Zhou et al., and CVE/NVD mapping; imperfect matching may introduce noise. (Section III-B.2)
  • domain assumption The expert feature sets are sufficient representations for commit vulnerability
    All models use the 14 or 9 expert features in Tables II and III; feature quality is not validated, and missing signals could explain low realistic performance. (Section III-D)
invented entities (1)
  • Vulnerability-Neutral Commit (VNC)
    purpose: Defines the realistic evaluation setting by labeling all commits that are neither VIC nor VFC as a third class, which is then collapsed into the safe class.
    VNC is a residual category based on the absence of detected vulnerability labels, not on independent evidence that the commit is non-vulnerable. The imbalance ratios and performance drops depend entirely on this constructed class. (Section II-B, Section III-B.2)

how reviews work

0 comments
Cite this review

Pith. "Pith review of Toward Realistic Evaluations of Just-In-Time Vulnerability Prediction." pith.science (2026). https://pith.science/paper/NYXPLKUV

@misc{pith2026250710729,
  author       = {Pith},
  title        = {Pith review of: Toward Realistic Evaluations of Just-In-Time Vulnerability Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NYXPLKUV}},
  note         = {Machine review of arXiv:2507.10729}
}
read the original abstract

Modern software systems are increasingly complex, presenting significant challenges in quality assurance. Just-in-time vulnerability prediction (JIT-VP) is a proactive approach to identifying vulnerable commits and providing early warnings about potential security risks. However, we observe that current JIT-VP evaluations rely on an idealized setting, where the evaluation datasets are artificially balanced, consisting exclusively of vulnerability-introducing and vulnerability-fixing commits. To address this limitation, this study assesses the effectiveness of JIT-VP techniques under a more realistic setting that includes both vulnerability-related and vulnerability-neutral commits. To enable a reliable evaluation, we introduce a large-scale public dataset comprising over one million commits from FFmpeg and the Linux kernel. Our empirical analysis of eight state-of-the-art JIT-VP techniques reveals a significant decline in predictive performance when applied to real-world conditions; for example, the average PR-AUC on Linux drops 98% from 0.805 to 0.016. This discrepancy is mainly attributed to the severe class imbalance in real-world datasets, where vulnerability-introducing commits constitute only a small fraction of all commits. To mitigate this issue, we explore the effectiveness of widely adopted techniques for handling dataset imbalance, including customized loss functions, oversampling, and undersampling. Surprisingly, our experimental results indicate that these techniques are ineffective in addressing the imbalance problem in JIT-VP. These findings underscore the importance of realistic evaluations of JIT-VP and the need for domain-specific techniques to address data imbalance in such scenarios.

Figures

Figures reproduced from arXiv: 2507.10729 by the authors.

Figure 1
Figure 1. Distributions of different types of commits in the Linux [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. CVE-2022-3965 [46] references. The first url links to commit 13c131 in the git.ffmpeg.org repository, which can be traced to its mirror on GitHub [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. CVE-2023-52461 [47] upstream patch commit. Its message state to fix commit 56e4496. The introducing commit can be traced in the master branch. Next, we further extended the VFCs collections by examin￾ing CVE/NVD [45] entries and official vulnerability reports. Since 90% of the CVE entries did not refer to GitHub com￾mit links [33], we actively searched for commit-like strings instead. For example, in CVE-2022-3965 [… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of the ratio of safe to vulnerable commits [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

76 extracted references · 60 canonical work pages

  1. [1]

    Crowdstrike outage report,

    CrowdStrike, “Crowdstrike outage report,” 2024. [Online]. Available: https://www.crowdstrike.com/wp-content/uploads/2024/08/ Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf

    2024

  2. [2]

    Crowdstrike outage damage,

    TechTarget, “Crowdstrike outage damage,” 2024. [Online]. Available: https://www.techtarget.com/whatis/feature/ Explaining-the-largest-IT-outage-in-history-and-whats-next

    2024

  3. [3]

    Vccfinder: Finding potential vulnerabilities in open-source projects to assist code audits,

    H. Perl, S. Dechand, M. Smith, D. Arp, F. Yamaguchi, K. Rieck, S. Fahl, and Y . Acar, “Vccfinder: Finding potential vulnerabilities in open-source projects to assist code audits,” in Proceedings of the 22nd ACM SIGSAC conference on computer and communications security , 2015, pp. 426– 437

    work page 2015

  4. [4]

    Code-centric learning-based just-in-time vulnerability detection,

    S. Nguyen, T.-T. Nguyen, T. T. Vu, T.-D. Do, K.-T. Ngo, and H. D. V o, “Code-centric learning-based just-in-time vulnerability detection,” Journal of Systems and Software , vol. 214, p. 112014, 2024

    2024

  5. [5]

    Revisiting the vccfinder approach for the identification of vulnerability- contributing commits,

    T. Riom, A. Sawadogo, K. Allix, T. F. Bissyand ´e, N. Moha, and J. Klein, “Revisiting the vccfinder approach for the identification of vulnerability- contributing commits,” Empirical Software Engineering, vol. 26, pp. 1– 30, 2021

    work page 2021

  6. [6]

    Available: https://github.com/ FFmpeg/FFmpeg

    FFmpeg Project, “FFmpeg.” [Online]. Available: https://github.com/ FFmpeg/FFmpeg

    work page

  7. [7]

    Linux kernel

    Linux Kernel Organization, “Linux kernel.” [Online]. Available: https://github.com/torvalds/linux

    work page

  8. [8]

    The best of both worlds: integrating semantic features with expert features for defect prediction and localization,

    C. Ni, W. Wang, K. Yang, X. Xia, K. Liu, and D. Lo, “The best of both worlds: integrating semantic features with expert features for defect prediction and localization,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2022, pp. 672–683

    2022

Show all 76 references
  1. [9]

    Focal loss for dense object detection,

    T.-Y . Lin, P. Goyal, R. Girshick, K. He, and P. Doll ´ar, “Focal loss for dense object detection,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 2980–2988

    2017

  2. [10]

    Smote for learning from imbalanced data: progress and challenges, marking the 15-year anniversary,

    A. Fern ´andez, S. Garcia, F. Herrera, and N. V . Chawla, “Smote for learning from imbalanced data: progress and challenges, marking the 15-year anniversary,” Journal of artificial intelligence research , vol. 61, pp. 863–905, 2018

    2018

  3. [11]

    Addressing the curse of imbalanced training sets: one-sided selection,

    M. Kubat, S. Matwin et al., “Addressing the curse of imbalanced training sets: one-sided selection,” in ICML, vol. 97, no. 1. Citeseer, 1997, p. 179

    1997

  4. [12]

    Replication package for

    D. Nguyen, T. Le-Cong, T. Huynh Minh Le, M. A. Babar, and Q.-T. Huynh, “Replication package for ”toward realistic evaluations of just-in-time vulnerability prediction”,” 2025. [Online]. Available: https://github.com/AI4Code-HUST/VulGuard

    2025

  5. [13]

    Vuldeepecker: A deep learning-based system for vulnerability detec- tion,

    Z. Li, D. Zou, S. Xu, X. Ou, H. Jin, S. Wang, Z. Deng, and Y . Zhong, “Vuldeepecker: A deep learning-based system for vulnerability detec- tion,” arXiv preprint arXiv:1801.01681 , 2018

    2018 arXiv

  6. [14]

    Keynote address: .ql for source code analysis,

    O. d. Moor, M. Verbaere, E. Hajiyev, P. Avgustinov, T. Ekman, N. Ongk- ingco, D. Sereni, and J. Tibble, “Keynote address: .ql for source code analysis,” in Seventh IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM 2007) , 2007, pp. 3–16

    2007

  7. [15]

    Flawfinder: A modular system for predicting quality flaws in wikipedia

    O. Ferschke, I. Gurevych, and M. Rittberger, “Flawfinder: A modular system for predicting quality flaws in wikipedia.” in CLEF (Online Working Notes/Labs/Workshop), 2012, pp. 1–10. 11

    2012

  8. [16]

    Comparison of static application security testing tools and large language models for repo-level vulnerability detection,

    X. Zhou, D.-M. Tran, T. Le-Cong, T. Zhang, I. C. Irsan, J. Sumarlin, B. Le, and D. Lo, “Comparison of static application security testing tools and large language models for repo-level vulnerability detection,” arXiv preprint arXiv:2407.16235 , 2024

    2024 arXiv

  9. [17]

    Deep learning based vulnerability detection: Are we there yet?

    S. Chakraborty, R. Krishna, Y . Ding, and B. Ray, “Deep learning based vulnerability detection: Are we there yet?” IEEE Transactions on Software Engineering , vol. 48, no. 9, pp. 3280–3296, 2021

    2021

  10. [18]

    Sysevr: A framework for using deep learning to detect software vulnerabilities,

    Z. Li, D. Zou, S. Xu, H. Jin, Y . Zhu, and Z. Chen, “Sysevr: A framework for using deep learning to detect software vulnerabilities,” IEEE Transactions on Dependable and Secure Computing, vol. 19, no. 4, pp. 2244–2258, 2021

    2021

  11. [19]

    Software vulnerability prediction in low-resource languages: An empirical study of codebert and chatgpt,

    T. H. M. Le, M. A. Babar, and T. H. Thai, “Software vulnerability prediction in low-resource languages: An empirical study of codebert and chatgpt,” in Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering , 2024, pp. 679– 685

    2024

  12. [20]

    Improving data curation of software vulnerability patches through uncertainty quantification,

    H. Chen, Y . Zhao, and K. Damevski, “Improving data curation of software vulnerability patches through uncertainty quantification,” 2024. [Online]. Available: https://arxiv.org/abs/2411.11659

    2024

  13. [21]

    Vuldigger: A just-in-time and cost-aware tool for digging vulnerability-contributing changes,

    L. Yang, X. Li, and Y . Yu, “Vuldigger: A just-in-time and cost-aware tool for digging vulnerability-contributing changes,” in GLOBECOM 2017- 2017 IEEE Global Communications Conference . IEEE, 2017, pp. 1–7

    2017

  14. [22]

    Commit- level, neural vulnerability detection and assessment,

    Y . Li, A. Yadavally, J. Zhang, S. Wang, and T. N. Nguyen, “Commit- level, neural vulnerability detection and assessment,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2023, pp. 1024–1036

    2023

  15. [23]

    Just-in-time software vulnerability detection: Are we there yet?

    F. Lomio, E. Iannone, A. De Lucia, F. Palomba, and V . Lenarduzzi, “Just-in-time software vulnerability detection: Are we there yet?” Jour- nal of Systems and Software , vol. 188, p. 111283, 2022

    2022

  16. [24]

    Vulcurator: a vulnerability-fixing commit detector,

    T. G. Nguyen, T. Le-Cong, H. J. Kang, X.-B. D. Le, and D. Lo, “Vulcurator: a vulnerability-fixing commit detector,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2022, pp. 1726–1730

    2022

  17. [25]

    Multi-granularity detector for vulnerability fixes,

    T. G. Nguyen, T. Le-Cong, H. J. Kang, R. Widyasari, C. Yang, Z. Zhao, B. Xu, J. Zhou, X. Xia, A. E. Hassan, X.-B. D. Le, and D. Lo, “Multi-granularity detector for vulnerability fixes,” IEEE Transactions on Software Engineering , 2023

    2023

  18. [26]

    Mapping nvd records to their vfcs: How hard is it?

    H. H. Nguyen, D. M. Tran, Y . Cheng, T. Le-Cong, H. J. Kang, R. Widyasari, S. L. Khin, O. E. Lieh, T. Zhang, and D. Lo, “Mapping nvd records to their vfcs: How hard is it?” arXiv preprint arXiv:2506.09702, 2025

    2025 arXiv

  19. [27]

    Known vulner- abilities of open source projects: Where are the fixes?

    A. Sabetta, S. E. Ponta, R. C. Lozoya, M. Bezzi, T. Sacchetti, M. Greco, G. Balogh, P. Heged ˝us, R. Ferenc, R. Paramitha et al., “Known vulner- abilities of open source projects: Where are the fixes?” IEEE Security & Privacy, vol. 22, no. 2, pp. 49–59, 2024

    2024

  20. [28]

    Patchfinder: A two-phase approach to security patch tracing for disclosed vulnerabilities in open-source software,

    K. Li, J. Zhang, S. Chen, H. Liu, Y . Liu, and Y . Chen, “Patchfinder: A two-phase approach to security patch tracing for disclosed vulnerabilities in open-source software,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2024,...

    2024

  21. [29]

    Devign: Effective vul- nerability identification by learning comprehensive program semantics via graph neural networks,

    Y . Zhou, S. Liu, J. Siow, X. Du, and Y . Liu, “Devign: Effective vul- nerability identification by learning comprehensive program semantics via graph neural networks,” Advances in neural information processing systems, vol. 32, 2019

    2019

  22. [30]

    When do changes induce fixes?

    J. ´Sliwerski, T. Zimmermann, and A. Zeller, “When do changes induce fixes?” ACM sigsoft software engineering notes , vol. 30, no. 4, pp. 1–5, 2005

    2005

  23. [31]

    V-szz: automatic identifi- cation of version ranges affected by cve vulnerabilities,

    L. Bao, X. Xia, A. E. Hassan, and X. Yang, “V-szz: automatic identifi- cation of version ranges affected by cve vulnerabilities,” in Proceedings of the 44th International Conference on Software Engineering , 2022, pp. 2352–2364

    2022

  24. [32]

    Repro- ducibility and credibility in empirical software engineering: A case study based on a systematic literature review of the use of the szz algorithm,

    G. Rodr ´ıguez-P´erez, G. Robles, and J. M. Gonz ´alez-Barahona, “Repro- ducibility and credibility in empirical software engineering: A case study based on a systematic literature review of the use of the szz algorithm,” Information and Software Technology , vol. 99, pp. 164–...

    2018

  25. [33]

    A large-scale empirical study on vulnerability distri- bution within projects and the lessons learned,

    B. Liu, G. Meng, W. Zou, Q. Gong, F. Li, M. Lin, D. Sun, W. Huo, and C. Zhang, “A large-scale empirical study on vulnerability distri- bution within projects and the lessons learned,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering , 2020, ...

    2020

  26. [34]

    Evaluating szz implementations through a developer- informed oracle,

    G. Rosa, L. Pascarella, S. Scalabrino, R. Tufano, G. Bavota, M. Lanza, and R. Oliveto, “Evaluating szz implementations through a developer- informed oracle,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 2021, pp. 436–447

    2021

  27. [35]

    Evaluating szz implementations: An empirical study on the linux kernel,

    Y . Lyu, H. J. Kang, R. Widyasari, J. Lawall, and D. Lo, “Evaluating szz implementations: An empirical study on the linux kernel,” IEEE Transactions on Software Engineering , 2024

    2024

  28. [36]

    Data quality for software vulnerability datasets,

    R. Croft, M. A. Babar, and M. M. Kholoosi, “Data quality for software vulnerability datasets,” in 2023 IEEE/ACM 45th International Confer- ence on Software Engineering (ICSE) . IEEE, 2023, pp. 121–133

    2023

  29. [37]

    FFmpeg Official Website,

    FFmpeg Project, “FFmpeg Official Website,” 2024, accessed: 2024-03-

    2024

  30. [38]

    Available: https://ffmpeg.org

    [Online]. Available: https://ffmpeg.org

  31. [39]

    FFmpeg Trac Wiki,

    ——, “FFmpeg Trac Wiki,” 2024, accessed: 2024-03-12. [Online]. Available: https://trac.ffmpeg.org

    2024

  32. [40]

    Linux Kernel Official Website,

    Linux Kernel Organization, “Linux Kernel Official Website,” 2024, accessed: 2024-03-12. [Online]. Available: https://kernel.org

    2024

  33. [41]

    Linux Kernel CVE Documentation,

    ——, “Linux Kernel CVE Documentation,” 2024, accessed: 2024-03-12. [Online]. Available: https://docs.kernel.org/process/cve.html

    2024

  34. [42]

    Automatic iden- tification of bug-introducing changes,

    S. Kim, T. Zimmermann, K. Pan, E. James Jr et al. , “Automatic iden- tification of bug-introducing changes,” in 21st IEEE/ACM international conference on automated software engineering (ASE’06) . IEEE, 2006, pp. 81–90

    2006

  35. [43]

    Are fix-inducing changes a moving target? a longitudinal case study of just-in-time defect prediction,

    S. McIntosh and Y . Kamei, “Are fix-inducing changes a moving target? a longitudinal case study of just-in-time defect prediction,” in Proceedings of the 40th international conference on software engineering , 2018, pp. 560–560

    2018

  36. [44]

    The importance of accounting for real-world labelling when predicting software vulnerabilities,

    M. Jimenez, R. Rwemalika, M. Papadakis, F. Sarro, Y . Le Traon, and M. Harman, “The importance of accounting for real-world labelling when predicting software vulnerabilities,” in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Sy...

    2019

  37. [45]

    Noisy label learning for security defects,

    R. Croft, M. A. Babar, and H. Chen, “Noisy label learning for security defects,” in Proceedings of the 19th International Conference on Mining Software Repositories, 2022, pp. 435–447

    2022

  38. [46]

    National Vulnerability Database (NVD),

    National Institute of Standards and Technology (NIST), “National Vulnerability Database (NVD),” 2024, accessed: 2024-03-12. [Online]. Available: https://nvd.nist.gov

    2024

  39. [47]

    CVE-2022-3965,

    National Vulnerability Database, “CVE-2022-3965,” 2022. [Online]. Available: https://www.cve.org/CVERecord?id=CVE-2022-3965

    2022

  40. [48]

    CVE-2023-52461,

    ——, “CVE-2023-52461,” 2023. [Online]. Available: https://nvd.nist. gov/vuln/detail/CVE-2023-52461

    2023

  41. [49]

    Linux guide for submitting patches,

    L. K. Organization, “Linux guide for submitting patches,” 2024, accessed: 2024-03-12. [Online]. Available: https://www.kernel.org/doc/ html/latest/process/submitting-patches.html

    2024

  42. [50]

    Deep just-in-time defect prediction: how far are we?

    Z. Zeng, Y . Zhang, H. Zhang, and L. Zhang, “Deep just-in-time defect prediction: how far are we?” in Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis , 2021, pp. 427–438

    2021

  43. [51]

    Chronos: Time-aware zero-shot identification of libraries from vulnerability reports,

    Y . Lyu, T. Le-Cong, H. J. Kang, R. Widyasari, Z. Zhao, X.-B. D. Le, M. Li, and D. Lo, “Chronos: Time-aware zero-shot identification of libraries from vulnerability reports,” in Proceedings of the 45th IEEE/ACM Internation Conference on Software Engineering , 2023

    2023

  44. [52]

    Systematic literature review on application of learning-based approaches in contin- uous integration,

    A. K. Arani, T. H. M. Le, M. Zahedi, and M. A. Babar, “Systematic literature review on application of learning-based approaches in contin- uous integration,” IEEE Access, 2024

    2024

  45. [53]

    Automated software vulnerabil- ity assessment with concept drift,

    T. H. M. Le, B. Sabir, and M. A. Babar, “Automated software vulnerabil- ity assessment with concept drift,” in2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR) . IEEE, 2019, pp. 371–382

    2019

  46. [54]

    J. paul guilford (1897- 1987),

    B. Fruchter, A. L. Comrey, and W. B. Michael, “J. paul guilford (1897- 1987),” Multivariate behavioral research, vol. 24, no. 1, pp. 3–15, 1989

    1987

  47. [55]

    The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation,

    D. Chicco and G. Jurman, “The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation,” BMC genomics, vol. 21, pp. 1–13, 2020

    2020

  48. [56]

    On the performance of matthews correlation coefficient (mcc) for imbalanced dataset,

    Q. Zhu, “On the performance of matthews correlation coefficient (mcc) for imbalanced dataset,” Pattern Recognition Letters , vol. 136, pp. 71– 80, 2020

    2020

  49. [57]

    On the use of fine-grained vulnera- ble code statements for software vulnerability assessment models,

    T. H. M. Le and M. A. Babar, “On the use of fine-grained vulnera- ble code statements for software vulnerability assessment models,” in Proceedings of the 19th International Conference on Mining Software Repositories, 2022, pp. 621–633

    2022

  50. [58]

    A large-scale empirical study of just-in-time quality assurance,

    Y . Kamei, E. Shihab, B. Adams, A. E. Hassan, A. Mockus, A. Sinha, and N. Ubayashi, “A large-scale empirical study of just-in-time quality assurance,” IEEE Transactions on Software Engineering , vol. 39, no. 6, pp. 757–773, 2012. 12

    2012

  51. [59]

    Tlel: A two-layer ensemble learning approach for just-in-time defect prediction,

    X. Yang, D. Lo, X. Xia, and J. Sun, “Tlel: A two-layer ensemble learning approach for just-in-time defect prediction,” Information and Software Technology, vol. 87, pp. 206–220, 2017

    2017

  52. [60]

    Deepjit: an end-to-end deep learning framework for just-in-time defect prediction,

    T. Hoang, H. K. Dam, Y . Kamei, D. Lo, and N. Ubayashi, “Deepjit: an end-to-end deep learning framework for just-in-time defect prediction,” in 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR). IEEE, 2019, pp. 34–45

    2019

  53. [61]

    Simple or complex? together for a more accurate just-in-time defect predictor,

    X. Zhou, D. Han, and D. Lo, “Simple or complex? together for a more accurate just-in-time defect predictor,” in Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, 2022, pp. 229–240

    2022

  54. [62]

    Modeling and discover- ing vulnerabilities with code property graphs,

    F. Yamaguchi, N. Golde, D. Arp, and K. Rieck, “Modeling and discover- ing vulnerabilities with code property graphs,” in 2014 IEEE symposium on security and privacy . IEEE, 2014, pp. 590–604

    2014

  55. [63]

    Available: https://github.com/joernio/joern

    Joernio, “Joern.” [Online]. Available: https://github.com/joernio/joern

  56. [64]

    Deep learning for source code modeling and generation: Models, applications, and challenges,

    T. H. Le, H. Chen, and M. A. Babar, “Deep learning for source code modeling and generation: Models, applications, and challenges,” ACM Computing Surveys (CSUR) , vol. 53, no. 3, pp. 1–38, 2020

    2020

  57. [65]

    Codebert: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al., “Codebert: A pre-trained model for programming and natural languages,” arXiv preprint arXiv:2002.08155 , 2020

    2002 arXiv

  58. [66]

    Predicting vulnerable software components,

    S. Neuhaus, T. Zimmermann, C. Holler, and A. Zeller, “Predicting vulnerable software components,” in Proceedings of the 14th ACM conference on Computer and communications security , 2007, pp. 529– 540

    2007

  59. [67]

    Can traditional fault prediction models be used for vulnerability prediction?

    Y . Shin and L. Williams, “Can traditional fault prediction models be used for vulnerability prediction?” Empirical Software Engineering , vol. 18, pp. 25–59, 2013

    2013

  60. [68]

    The devil is in the tails: How long-tailed code distributions impact large language mod- els,

    X. Zhou, K. Kim, B. Xu, J. Liu, D. Han, and D. Lo, “The devil is in the tails: How long-tailed code distributions impact large language mod- els,” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2023, pp. 40–52

    2023

  61. [69]

    Vgx: Large-scale sample generation for boosting learning-based software vul- nerability analyses,

    Y . Nong, R. Fang, G. Yi, K. Zhao, X. Luo, F. Chen, and H. Cai, “Vgx: Large-scale sample generation for boosting learning-based software vul- nerability analyses,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 2024, pp. 1–13

    2024

  62. [70]

    Vulgen: Realistic vulnerability generation via pattern mining and deep learning,

    Y . Nong, Y . Ou, M. Pradel, F. Chen, and H. Cai, “Vulgen: Realistic vulnerability generation via pattern mining and deep learning,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 2527–2539

    2023

  63. [71]

    Mitigating data imbalance for software vulnerability assessment: Does data augmentation help?

    T. H. M. Le and M. Ali Babar, “Mitigating data imbalance for software vulnerability assessment: Does data augmentation help?” in Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement , 2024, pp. 119–130

    2024

  64. [72]

    Towards reliable evaluation of neural program repair with natural robustness testing,

    T. Le-Cong, T.-D. Nguyen, B. Le, and T. Murray, “Towards reliable evaluation of neural program repair with natural robustness testing,” ACM Trans. Softw. Eng. Methodol. , Feb. 2025, just Accepted. [Online]. Available: https://doi.org/10.1145/3716167

    2025 doi

  65. [73]

    Are latent vulnerabilities hidden gems for software vulnerability prediction? an empirical study,

    T. H. M. Le, X. Du, and M. A. Babar, “Are latent vulnerabilities hidden gems for software vulnerability prediction? an empirical study,” in Proceedings of the 21st International Conference on Mining Software Repositories, 2024, pp. 716–727

    2024

  66. [74]

    Deepcva: Automated commit-level vulnerability assessment with deep multi-task learning,

    T. H. M. Le, D. Hin, R. Croft, and M. A. Babar, “Deepcva: Automated commit-level vulnerability assessment with deep multi-task learning,” in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2021, pp. 717–729

    2021

  67. [75]

    A survey on data-driven software vulnerability assessment and prioritization,

    T. H. M. Le, H. Chen, and M. A. Babar, “A survey on data-driven software vulnerability assessment and prioritization,” ACM Computing Surveys, vol. 55, no. 5, pp. 1–39, 2022

    2022

  68. [76]

    A survey of learning- based automated program repair,

    Q. Zhang, C. Fang, Y . Ma, W. Sun, and Z. Chen, “A survey of learning- based automated program repair,” ACM Transactions on Software Engi- neering and Methodology , vol. 33, no. 2, pp. 1–69, 2023. 13

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.