Pith. sign in

REVIEW 4 major objections 6 minor 56 references

This paper shows that deep-learning and LLM vulnerability detectors, strong on standard benchmarks, fail to distinguish vulnerable code in a small set of newly disclosed Linux kernel vulnerabilities.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 17:07 UTC pith:BUHWLJJY

load-bearing objection The VentiVul dataset and the negative result are worth a referee, but Finding 8's 'superior ability' rests on at-chance counts and should be tempered before publication. the 4 major comments →

arxiv 2512.10485 v2 pith:BUHWLJJY submitted 2025-12-11 cs.CR cs.LGcs.SE

From Lab to Reality: A Practical Evaluation of Deep Learning Models and LLMs for Vulnerability Detection

Chaomeng Lu , Bert Lagaisse This is my paper
classification cs.CR cs.LGcs.SE
keywords vulnerability detectiondeep learninglarge language modelsout-of-distribution evaluationLinux kernelcode representationfunction-pair evaluationbenchmark realism
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Deep-learning and LLM-based vulnerability detectors that post strong F1 scores on standard benchmarks (Juliet, Devign, BigVul, ICVul) stop distinguishing vulnerable from non-vulnerable code when tested on VentiVul, a new dataset of 20 real Linux-kernel vulnerabilities fixed in May 2025. The best Whole-File F1 among eight trained model variants is 8.13 and among four LLMs 4.9; in the Function-Pair setting, most models cannot even assign a higher risk score to a function than to its own patched version. The paper argues this happens because current models learn dataset-specific correlations—noisy labels, class imbalance, and surface patterns—rather than generalizable vulnerability semantics. Representation analysis with t-SNE and centroid distances shows vulnerable and non-vulnerable functions overlap heavily, and cross-dataset tests confirm the drop. If the paper is right, benchmark results overstate real-world readiness, and evaluation practice needs to incorporate temporal out-of-distribution data and patch-level reasoning.

Core claim

On a time-wise out-of-distribution dataset (VentiVul) of 20 recent Linux kernel CVEs, the authors find that both representative DL models (ReVeal and LineVul) and four pretrained LLMs (Claude 3.5 Sonnet, GPT-o3-mini, GPT-4o, and GPT-5) fail to reliably distinguish vulnerable from non-vulnerable code. Under Whole-File evaluation, the best F1 among DL variants is 8.13 and among LLMs 4.9; under Function-Pair evaluation, at most 6 of 25 before/after-fix pairs are correctly differentiated (GPT-5), with most models scoring zero. Representation analysis using t-SNE and centroid distances shows heavy overlap between vulnerable and non-vulnerable functions, and cross-dataset experiments show sharp pe

What carries the argument

VentiVul, a manually curated out-of-distribution test set of 20 Linux-kernel CVEs fixed in May 2025, containing 25 vulnerable/patched function pairs plus 835 unrelated functions from the same files. The paper's central evaluation devices are the Whole-File setting (classify every function in a file) and the Function-Pair setting (compare each before-fix function with its after-fix counterpart; a correct judgment requires the model to score the vulnerable version higher than the patched one). The pairing isolates whether a model captures the semantic change of a fix, which is the minimal ability needed to spot new vulnerabilities. t-SNE and centroid distance serve as the representation-qualit

Load-bearing premise

The RQ3 LLM conclusions rest on treating a single hand-written prompt with unstated decoding settings, run once per model, as a fair and stable measure of each LLM's vulnerability-detection ability; prompt phrasing and sampling parameters are known to change LLM outputs, and no sensitivity analysis shows they would not.

What would settle it

Re-run the same VentiVul Function-Pair evaluation with several prompt phrasings and decoding settings (e.g., temperature 0 vs 0.7, top-p variation, repeated sampling) for GPT-4o and GPT-5; if these models then differentiate substantially more than 6 of 25 pairs, or show large variance across runs, the reported failure is an artifact of the specific prompt rather than a stable property of the models.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Benchmark scores on Juliet, Devign, BigVul, and ICVul do not transfer to time-wise out-of-distribution code; deployment-oriented evaluation changes the reported ranking of models.
  • Dataset label quality and balance, not raw volume, dominate cross-dataset generalization; BigVul's noisy labels inflate recall but produce near-zero transferability.
  • Function-Pair evaluation is a cheap, interpretable way to test whether a detector understands the effect of a security patch; current DL models mostly fail it, suggesting they rely on superficial cues.
  • LLMs show relatively stronger patch-sensitivity (GPT-4o and GPT-5 differentiate 5-6 of 25 pairs) despite low recall, pointing to patch-level reasoning as a promising direction rather than whole-file binary classification.
  • Current representation learning—graph-based and token-based—does not yield separable vulnerable/non-vulnerable clusters, so improving representations, not just architectures, is the bottleneck.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because VentiVul contains only 20 CVEs and 25 pairs from one kernel, the quantitative F1 values are noisy; the paper's durable contribution is the evaluation protocol (temporal split plus before/after pairing), not the precise numbers.
  • A natural extension is to use VentiVul-style Function-Pair evaluation as a deployment filter: a detector that cannot tell a function from its own patch is unlikely to catch new bugs, regardless of benchmark F1.
  • Prompt sensitivity is untested; varying prompts, temperatures, and repeated sampling could move LLM results substantially, so the 'LLMs fail' result should be read as 'under this one zero-shot protocol,' not as a law.
  • The finding that distances between different vulnerability types sometimes exceed distances between vulnerable and non-vulnerable code suggests that CWE-specific or root-cause-conditioned models may be a more informative target than a single binary classifier.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper evaluates two widely used vulnerability detection models (ReVeal and LineVul) on four benchmarks (Juliet, Devign, BigVul, ICVul) and, together with four commercial LLMs, on a newly curated out-of-distribution dataset VentiVul of 20 Linux kernel CVEs fixed in May 2025. Three research questions are addressed: (RQ1) whether learned code representations separate vulnerable and non-vulnerable functions, (RQ2) which dataset/model factors affect within- and cross-dataset performance, and (RQ3) whether DL models and LLMs detect recent, previously unseen vulnerabilities in a Whole-File and Function-Pair setting. The main reported findings are that embeddings show limited separation, cross-dataset performance drops sharply, most models have very low F1 on VentiVul, and GPT-4o/GPT-5 differentiate 5-6 of 25 vulnerable/patched pairs, which the paper interprets as superior patch-level reasoning. The paper argues that benchmark results overestimate real-world applicability.

Significance. If the negative results are reliable, the paper adds to a growing body of evidence (Real-Vul, PRIMEVUL) that benchmark-trained vulnerability detectors fail on temporally held-out data. The authors contribute a carefully curated, manually inspected VentiVul dataset, a reproducible framework with code and data release, and a Function-Pair evaluation mode that is more realistic than simple function-level classification. The Whole-File analysis is also useful in exposing all-negative/all-positive prediction biases. The central negative claims are plausible and consistent with prior work. However, the quantitative support is weaker than the narrative: no confidence intervals or significance tests, t-SNE-based distances used as quantitative evidence, a single prompt/run for LLMs, and a positive LLM result that is statistically indistinguishable from chance. These issues are localized but load-bearing for the positive 'promise' claim and for several RQ1/RQ3 statements.

major comments (4)
  1. [Section 4 (RQ3), Table 7, Finding 8] The positive LLM claim is not supported by the reported counts. GPT-4o and GPT-5 differentiate 5 and 6 of 25 pairs. Under any independent-label null, the expected number of differentiated pairs is at most 25 * 0.25 = 6.25, so the observed values are at or below the maximum chance expectation. No confidence interval, null model, or significance test is reported. The conclusion that GPT-4o/GPT-5 exhibit 'superior ability to discern fine-grained semantic changes' and the sentence in the conclusion that 'LLMs hold promise in understanding patch-level changes' therefore need either a proper statistical test (e.g., exact binomial test with a stated null, or permutation test) or should be removed/qualified. The Table 6 F1 numbers still support the negative OOD result; this issue concerns the paper's only positive LLM claim.
  2. [Section 3.2, Table 3] LLM results are based on a single hand-written prompt per setting with no decoding parameters (temperature, top-p, repetition penalty, max tokens) and one run per model. LLM outputs are highly sensitive to prompt wording and sampling. Tables 6-7 and Findings 7-8 therefore may reflect one arbitrary configuration rather than model ability. Please report exact API/model version and decoding settings, run multiple seeds, and report variance or use a fixed deterministic setting with justification. Without this, the RQ3 comparisons among LLMs and the relative claim about GPT-4o/GPT-5 are not reproducible.
  3. [Section 3.3.1, Eq. (1)] Centroid distance is computed after t-SNE dimensionality reduction and min-max normalization. t-SNE is stochastic and does not preserve pairwise distances; hyperparameters (perplexity, learning rate, iterations) are not reported. Under these conditions D_centroid is not a meaningful quantitative measure of class separability. This quantity underlies Findings 1-3. Please either replace or augment it with a distance-preserving or embedding-stable measure (e.g., centroids in the original feature space, or silhouette/other intrinsic metrics), report sensitivity to t-SNE hyperparameters, or explicitly label the centroid distances as illustrative rather than quantitative.
  4. [Section 4, RQ2/RQ3] The paper's comparative conclusions rest on point estimates from a single training/evaluation run on small data (25 pairs/20 CVEs, with four datasets). For example, the statement that some models 'transfer better' is based on F1 differences that are not accompanied by variance or significance information, and models with F1<20 are excluded from ranking to avoid trivial results. Please add uncertainty quantification (multiple seeds/LLM runs, or at least exact binomial confidence intervals for the categorical VentiVul counts) and justify the exclusion rule. This does not undermine the large drop in Table 6, but it is needed before claiming model or dataset ordering.
minor comments (6)
  1. [Section 2.1, Table 1] ICVul is the authors' own dataset and its 92% label accuracy is the result of the authors' manual validation of 50 samples. Given that ICVul is later used to argue that clean labels lead to transferable representations (Finding 5), an independent audit or a clearer statement of provenance and potential conflict would strengthen the paper.
  2. [Table 7] The definitions of 'Vul Detected' (out of 20 CVEs) and 'Vul Funcs Detected' (out of 25 pairs) should be stated in the caption or text, because some CVEs contribute multiple function pairs; otherwise the denominators are confusing.
  3. [Figure 5] A 'non-decrease' conflates unchanged scores and increased scores; plotting these separately (or counting increases) would help interpret the Pairs-Differentiated results.
  4. [Section 3.2/Table 3] The Whole-File prompt asks for 'method(s)' and an output format 'like ...'; it is unclear how free-form outputs are parsed into function names and how ambiguous cases are handled. Add the parsing/validation procedure.
  5. [Throughout] The model name 'GPT-o3-mini' is used inconsistently (also rendered as GPT-o3-mini in the abstract; OpenAI branding is 'o3-mini').
  6. [Section 7, Related Work] The discussion could better contrast VentiVul with the closely related Real-Vul [6] and PRIMEVUL [13] datasets in terms of temporal separation, manual curation, and the Function-Pair evaluation mode.

Circularity Check

1 steps flagged

No derivation-level circularity; the negative VentiVul result is externally grounded. Minor self-citation/self-validation attaches to the ICVul benchmark-quality claim, but the central conclusion does not reduce to it.

specific steps
  1. other [Section 2.1 (Table 1) and Section 3.1.2; reference [24]]
    "For ICVul, we manually evaluated label accuracy by randomly sampling 50 vulnerable functions, identifying their corresponding patch commits, and validating labels based on commit messages and code changes. ... Although ICVul is a newly published dataset, it stands out for its relatively high vulnerability ratio (41%) and high label accuracy (92%) ..."

    ICVul is the authors' own prior dataset [24]. Its 'high-quality'/92% label-accuracy characterization, used to select it as a benchmark and to support RQ2 conclusions about transferability, is justified by the authors' own manual validation rather than by any independent external check. This is a self-referential evidence chain, but it is minor: the paper's central VentiVul negative result relies on externally anchored CVE fix commits and independent model evaluations, not on ICVul quality.

full rationale

The paper is an empirical evaluation, not a derivation. There is no equation in which an output is identical to an input by construction, no fitted parameter renamed as a prediction, and no uniqueness theorem imported from the authors' prior work. The RQ3 metrics (Whole-File F1, Pairs Differentiated) are direct observed counts; the low F1 values and near-chance pair-differentiation counts are measurements, not outputs of a fitted model. The self-constructed VentiVul is time-wise out-of-distribution and anchored to CVE fix commits, so evaluating models on it is a legitimate external test even though the labels were hand-built by the team. The only noteworthy self-reference is the use of ICVul [24], the authors' own dataset, for which the paper itself supplies the 92% label-accuracy estimate; this affects the benchmark-quality framing but is not load-bearing for the main negative conclusion. The skeptics' point about GPT-4o/GPT-5 'superior ability' being at random-guessing levels is a statistical-support concern, not a circularity: the counts do not reduce to the conclusion by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 7 axioms · 0 invented entities

No new theoretical entities or fitted model parameters are introduced. The free parameters are evaluation choices (t-SNE settings, classification threshold, LLM decoding) that are not reported or are chosen by default; they influence the quantitative results. The axioms are domain assumptions about dataset representativeness, label validity, and metric validity.

free parameters (3)
  • t-SNE hyperparameters (perplexity, learning rate, iterations) = not reported
    Centroid distances in Eq. 1 are computed on 2D t-SNE coordinates, which depend strongly on these settings; the paper does not state them, so the RQ1 separability numbers are not robustly reproducible.
  • Classification threshold (0.5) = 0.5
    Figure 5 and Table 7 use a fixed 0.5 threshold to define 'detected' and 'differentiated'; a different threshold would change the Pairs Differentiated counts, which drive the LLM comparison.
  • LLM decoding parameters (temperature, top-p, max tokens) = not reported
    Single-run LLM evaluations without reported decoding settings; outputs are stochastic, so results in Tables 6-7 might vary run-to-run.
axioms (7)
  • domain assumption VentiVul labels are correct: pre-fix functions are vulnerable; post-fix functions and unrelated same-file functions are non-vulnerable.
    Section 2.2: manually curated from CVE fix commits; any label errors directly bias all RQ3 metrics.
  • domain assumption The 20 Linux kernel CVEs from May 2025 are representative of novel real-world vulnerabilities.
    Section 2.2: convenience sample selected from one month and one project; no power analysis, so quantitative conclusions may not generalize.
  • ad hoc to paper Centroid distance computed in t-SNE 2D space is a meaningful quantitative measure of class separability.
    Section 3.3.1 Eq. 1; t-SNE is stochastic and does not preserve global distances; the paper acknowledges parameter sensitivity in Section 6.
  • ad hoc to paper Single-run LLM responses under one hand-written prompt are stable and representative.
    Section 3.2 Table 3; no repeated runs or decoding details are reported, so variance is unknown.
  • ad hoc to paper Excluding models with F1<20 from ranking in RQ2 does not bias conclusions.
    Section 4 RQ2: 'Models with F1-scores below 20 were excluded from ranking to avoid trivial results'; this post-hoc rule shapes the cross-dataset rankings.
  • domain assumption The label accuracy values in Table 1 for Juliet/Devign/BigVul, taken from prior studies [8,11], are reliable.
    These numbers underlie the label-quality discussion (Finding 5); if BigVul's true accuracy is at the low end (25%), the interpretation of its behavior changes.
  • domain assumption Function-level and function-pair test settings are the right operationalization of real-world vulnerability detection.
    Whole-File and Function-Pair modes (Section 3.2) assume that detecting the vulnerable function in a file is the relevant deployment task; real workflows might use other granularities.

pith-pipeline@v1.3.0-alltime-deepseek · 16522 in / 16529 out tokens · 159807 ms · 2026-08-03T17:07:11.580307+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of From Lab to Reality: A Practical Evaluation of Deep Learning Models and LLMs for Vulnerability Detection." pith.science (2026). https://pith.science/paper/BUHWLJJY

@misc{pith2026251210485,
  author       = {Pith},
  title        = {Pith review of: From Lab to Reality: A Practical Evaluation of Deep Learning Models and LLMs for Vulnerability Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BUHWLJJY}},
  note         = {Machine review of arXiv:2512.10485}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Vulnerability detection methods based on deep learning (DL) have shown strong performance on benchmark datasets, yet their real-world effectiveness remains underexplored. Recent work suggests that both graph neural network (GNN)-based and transformer-based models, including large language models (LLMs), yield promising results when evaluated on curated benchmark datasets. These datasets are typically characterized by consistent data distributions and heuristic or partially noisy labels. In this study, we systematically evaluate two representative DL models-ReVeal and LineVul-across four representative datasets: Juliet, Devign, BigVul, and ICVul. Each model is trained independently on each respective dataset, and their code representations are analyzed using t-SNE to uncover vulnerability related patterns. To assess realistic applicability, we deploy these models along with four pretrained LLMs, Claude 3.5 Sonnet, GPT-o3-mini, GPT-4o, and GPT-5 on a curated dataset, VentiVul, comprising 20 recently (May 2025) fixed vulnerabilities from the Linux kernel. Our experiments reveal that current models struggle to distinguish vulnerable from non-vulnerable code in representation space and generalize poorly across datasets with differing distributions. When evaluated on VentiVul, our newly constructed time-wise out-of-distribution dataset, performance drops sharply, with most models failing to detect vulnerabilities reliably. These results expose a persistent gap between academic benchmarks and real-world deployment, emphasizing the value of our deployment-oriented evaluation framework and the need for more robust code representations and higher-quality datasets.

Figures

Figures reproduced from arXiv: 2512.10485 by Bert Lagaisse, Chaomeng Lu.

Figure 1
Figure 1. Figure 1: Examples of code Modifications in selected CVE [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: A deployment-oriented framework for evaluating vulnerability detection models [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of feature space visualizations and cluster analysis on four benchmark datasets using (a) GNN and (b) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Centroid distance comparison between top 5 vul [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Model predictions on before-fix and after-fix functions in VentiVul [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 2 canonical work pages

  1. [1]

    Anthropic. 2024. Claude 3.5 Sonnet. Available at https://www.anthropic.com

    2024

  2. [2]

    Guru Bhandari, Amara Naseer, and Leon Moonen. 2021. CVEfixes: automated collection of vulnerabilities and their fixes from open-source software. InPro- ceedings of the 17th International Conference on Predictive Models and Data An- alytics in Software Engineering(Athens, Greece)(PROMISE 2021). ACM, 30–39. From Lab to Reality: A Practical Evaluation of Dee...

    arXiv 2021

  3. [3]

    Lorenzo Binosi, Gregorio Barzasi, Michele Carminati, Stefano Zanero, and Mario Polino. 2024. The Illusion of Randomness: An Empirical Analysis of Address Space Layout Randomization Implementations. InProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security(Salt Lake City, UT, USA)(CCS ’24). ACM, 1360–1374. doi:10.1145/36586...

    arXiv 2024

  4. [4]

    Tim Boland and Paul E Black. 2012. Juliet 1. 1 C/C++ and java test suite.Computer 45, 10 (2012), 88–90

    2012

  5. [5]

    Beatrice Casey, Joanna C. S. Santos, and George Perry. 2025. A Survey of Source Code Representations for Machine Learning-Based Cybersecurity Tasks.ACM Comput. Surv.57, 8, Article 217 (April 2025), 41 pages. doi:10.1145/3721977

    work page doi:10.1145/3721977 2025

  6. [6]

    Partha Chakraborty, Krishna Kanth Arumugam, Mahmoud Alfadel, Meiyappan Nagappan, and Shane McIntosh. 2024. Revisiting the Performance of Deep Learning-Based Vulnerability Detection on Realistic Datasets.IEEE Transactions on Software Engineering50, 8 (2024), 2163–2177. doi:10.1109/TSE.2024.3423712

    arXiv 2024

  7. [7]

    Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. 2021. Deep learning based vulnerability detection: Are we there yet?IEEE Transactions on Software Engineering48, 9 (2021), 3280–3296

    2021

  8. [8]

    Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David Wagner

  9. [9]

    Xiao Cheng, Haoyu Wang, Jiayi Hua, Guoai Xu, and Yulei Sui. 2021. DeepWukong: Statically Detecting Software Vulnerabilities Using Deep Graph Neural Network. ACM Trans. Softw. Eng. Methodol.30, 3, Article 38 (April 2021), 33 pages. doi:10. 1145/3436877

    2021

  10. [10]

    Roland Croft, M Ali Babar, and Huaming Chen. 2022. Noisy label learning for security defects. InProceedings of the 19th International Conference on Mining Software Repositories. 435–447

    2022

  11. [11]

    Roland Croft, M Ali Babar, and M Mehdi Kholoosi. 2023. Data quality for software vulnerability datasets. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 121–133

    2023

  12. [12]

    Roland Croft, Yongzheng Xie, and Muhammad Ali Babar. 2023. Data Preparation for Software Vulnerability Prediction: A Systematic Literature Review.IEEE Transactions on Software Engineering49, 3 (2023), 1044–1063. doi:10.1109/TSE. 2022.3171202

    arXiv 2023

  13. [13]

    Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. 2024. Vulnerability detection with code language models: How far are we?arXiv preprint arXiv:2403.18624(2024)

    Pith/arXiv arXiv 2024

  14. [14]

    Jiahao Fan, Yi Li, Shaohua Wang, and Tien N. Nguyen. 2020. A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries. InProceedings of the 17th International Conference on Mining Software Repositories(Seoul, Republic of Korea)(MSR ’20). ACM, 508–512. doi:10.1145/3379597.3387501

    arXiv 2020

  15. [15]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages.arXiv preprint arXiv:2002.08155 (2020)

    Pith/arXiv arXiv 2020

  16. [16]

    Michael Fu, Van Nguyen, Chakkrit Tantithamthavorn, Dinh Phung, and Trung Le. 2024. Vision Transformer Inspired Automated Vulnerability Repair. 33, 3, Article 78 (March 2024), 29 pages. doi:10.1145/3632746

    work page doi:10.1145/3632746 2024

  17. [17]

    Michael Fu and Chakkrit Tantithamthavorn. 2022. LineVul: a transformer-based line-level vulnerability prediction. InProceedings of the 19th International Confer- ence on Mining Software Repositories(Pittsburgh, Pennsylvania)(MSR ’22). ACM, 608–620. doi:10.1145/3524842.3528452

    arXiv 2022

  18. [18]

    Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. 2020. Graphcodebert: Pre-training code representations with data flow.arXiv preprint arXiv:2009.08366 (2020)

    Pith/arXiv arXiv 2020

  19. [19]

    Hazim Hanif and Sergio Maffeis. 2022. VulBERTa: Simplified Source Code Pre- Training for Vulnerability Detection. In2022 International Joint Conference on Neural Networks (IJCNN). 1–8. doi:10.1109/IJCNN55064.2022.9892280

    arXiv 2022

  20. [20]

    Ali Babar

    David Hin, Andrey Kan, Huaming Chen, and M. Ali Babar. 2022. LineVD: statement-level vulnerability detection using graph neural networks. InProceed- ings of the 19th International Conference on Mining Software Repositories(Pitts- burgh, Pennsylvania)(MSR ’22). ACM, 596–607. doi:10.1145/3524842.3527949

    arXiv 2022

  21. [21]

    Matthieu Jimenez, Renaud Rwemalika, Mike Papadakis, Federica Sarro, Yves Le Traon, and Mark Harman. 2019. The importance of accounting for real-world labelling when predicting software vulnerabilities. InProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 695–705

    2019

  22. [22]

    Yi Li, Shaohua Wang, and Tien N. Nguyen. 2021. Vulnerability detection with fine-grained interpretations. InProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering(Athens, Greece)(ESEC/FSE 2021). ACM, 292–303. doi:10. 1145/3468264.3468597

    arXiv 2021

  23. [23]

    Guanjun Lin, Wei Xiao, Leo Yu Zhang, Shang Gao, Yonghang Tai, and Jun Zhang

  24. [24]

    Chaomeng Lu, Tianyu Li, Toon Dehaene, and Bert Lagaisse. 2025. ICVul: A Well- labeled C/C++ Vulnerability Dataset with Comprehensive Metadata and VCCs. In2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). 154–158. doi:10.1109/MSR66628.2025.00034

    arXiv 2025

  25. [25]

    Guilong Lu, Xiaolin Ju, Xiang Chen, Wenlong Pei, and Zhilong Cai. 2024. GRACE: Empowering LLM-based software vulnerability detection with graph structure and in-context learning.Journal of Systems and Software212 (2024), 112031

    2024

  26. [26]

    Alejandro Mazuera-Rozo, Anamaria Mojica-Hanke, Mario Linares-Vásquez, and Gabriele Bavota. 2021. Shallow or Deep? An Empirical Study on Detecting Vul- nerabilities using Deep Learning. In2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC). 276–287. doi:10.1109/ICPC52881.2021.00034

    arXiv 2021

  27. [27]

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems26 (2013)

    2013

  28. [28]

    Stephan Neuhaus and Thomas Zimmermann. 2009. The beauty and the beast: Vulnerabilities in red hat’s packages. InProceedings of the 2009 USENIX Annual Technical Conference (USENIX ATC). 383–396

    2009

  29. [29]

    Chao Ni, Liyu Shen, Xiaohu Yang, Yan Zhu, and Shaohua Wang. 2024. MegaVul: A C/C++ Vulnerability Dataset with Comprehensive Code Representations. In Proceedings of the 21st International Conference on Mining Software Repositories (Lisbon, Portugal)(MSR ’24). ACM, 738–742. doi:10.1145/3643991.3644886

    arXiv 2024

  30. [30]

    Georgios Nikitopoulos, Konstantina Dritsa, Panos Louridas, and Dimitris Mitropoulos. 2021. CrossVul: a cross-language vulnerability dataset with commit data. InProceedings of the 29th ACM Joint Meeting on European Software Engi- neering Conference and Symposium on the Foundations of Software Engineering (Athens, Greece)(ESEC/FSE 2021). ACM, 1565–1569. doi...

    arXiv 2021

  31. [31]

    2025.NIST Software Assurance Reference Dataset

    National Institute of Standards and Technology. 2025.NIST Software Assurance Reference Dataset. Retrieved June 16, 2025 from https://samate.nist.gov/SARD

    2025

  32. [32]

    OpenAI. 2025. GPT-o3-mini, GPT-4o, GPT-5. https://openai.com. Available at https://openai.com

    2025

  33. [33]

    Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 1532–1543

    2014

  34. [34]

    Fangcheng Qiu, Zhongxin Liu, Xing Hu, Xin Xia, Gang Chen, and Xinyu Wang

  35. [35]

    Adriana Sejfia, Satyaki Das, Saad Shafiq, and Nenad Medvidović. 2024. To- ward Improved Deep Learning-based Vulnerability Detection. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering(Lisbon, Portugal)(ICSE ’24). ACM, Article 62, 12 pages. doi:10.1145/3597503.3608141

    arXiv 2024

  36. [36]

    Nima Shiri Harzevili, Alvine Boaye Belle, Junjie Wang, Song Wang, Zhen Ming (Jack) Jiang, and Nachiappan Nagappan. 2024. A Systematic Literature Review on Automated Software Vulnerability Detection Using Machine Learning. ACM Comput. Surv.57, 3, Article 55 (Nov. 2024), 36 pages. doi:10.1145/3699711

    doi:10.1145/3699711 2024

  37. [38]

    Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Haijun Wang, Zhengzi Xu, Xiaofei Xie, and Yang Liu. 2024. GPTScan: Detecting Logic Vulnerabilities in Smart Con- tracts by Combining GPT with Program Analysis. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering(Lisbon, Portugal)(ICSE ’24). ACM, New York, NY, USA, Article 166, 13 p...

    arXiv 2024

  38. [39]

    Gaigai Tang, Lianxiao Meng, Huiqiang Wang, Shuangyin Ren, Qiang Wang, Lin Yang, and Weipeng Cao. 2020. A Comparative Study of Neural Network Techniques for Automatic Software Vulnerability Detection. In2020 International Symposium on Theoretical Aspects of Software Engineering (TASE). 1–8. doi:10. 1109/TASE49443.2020.00010

    arXiv 2020

  39. [40]

    Chandra Thapa, Seung Ick Jang, Muhammad Ejaz Ahmed, Seyit Camtepe, Josef Pieprzyk, and Surya Nepal. 2022. Transformer-Based Language Models for Software Vulnerability Detection(ACSAC ’22). ACM, 481–496. doi:10.1145/ 3564625.3567985

    arXiv 2022

  40. [42]

    Zhang, and Qing Liao

    Xin-Cheng Wen, Yupan Chen, Cuiyun Gao, Hongyu Zhang, Jie M. Zhang, and Qing Liao. 2023. Vulnerability Detection with Graph Simplification and Enhanced Graph Representation Learning. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 2275–2286. doi:10.1109/ICSE48619.2023.00191

    arXiv 2023

  41. [43]

    Xin-Cheng Wen, Cuiyun Gao, Feng Luo, Haoyu Wang, Ge Li, and Qing Liao

  42. [44]

    Hua Yan, Yulei Sui, Shiping Chen, and Jingling Xue. 2017. Machine-learning- guided typestate analysis for static use-after-free detection. InProceedings of the Chaomeng Lu and Bert Lagaisse 33rd Annual Computer Security Applications Conference. 42–54

    2017

  43. [45]

    Xu Yang, Shaowei Wang, Yi Li, and Shaohua Wang. 2023. Does data sampling improve deep learning-based vulnerability detection? yeas! and nays!. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2287–2298

    2023

  44. [46]

    Chenyuan Zhang, Hao Liu, Jiutian Zeng, Kejing Yang, Yuhong Li, and Hui Li

  45. [47]

    LIVABLE: Exploring Long-Tailed Classification of Software Vulnerability Types.IEEE Transactions on Software Engineering50, 6 (2024), 1325–1339. doi:10. 1109/TSE.2024.3382361

    arXiv 2024

  46. [48]

    Wei Zheng, Jialiang Gao, Xiaoxue Wu, Fengyu Liu, Yuxing Xun, Guoliang Liu, and Xiang Chen. 2020. The impact factors on the performance of machine learning-based vulnerability detection: A comparative study.Journal of Systems and Software168 (2020), 110659

    2020

  47. [49]

    Yunhui Zheng, Saurabh Pujar, Burn Lewis, Luca Buratti, Edward Epstein, Bo Yang, Jim Laredo, Alessandro Morari, and Zhong Su. 2021. D2A: A Dataset Built for AI-Based Vulnerability Detection Methods Using Differential Analysis. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). 111–120. doi:...

    arXiv 2021

  48. [50]

    Liyi Zhou, Xihan Xiong, Jens Ernstberger, Stefanos Chaliasos, Zhipeng Wang, Ye Wang, Kaihua Qin, Roger Wattenhofer, Dawn Song, and Arthur Gervais. 2023. SoK: Decentralized Finance (DeFi) Attacks. In2023 IEEE Symposium on Security and Privacy (SP). 2444–2461. doi:10.1109/SP46215.2023.10179435

    arXiv 2023

  49. [51]

    In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings(Lisbon, Portugal)(ICSE-Companion ’24)

    Prompt-Enhanced Software Vulnerability Detection Using ChatGPT. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings(Lisbon, Portugal)(ICSE-Companion ’24). ACM, 276–277. doi:10.1145/3639478.3643065

    arXiv 2024

  50. [52]

    Junwei Zhang, Zhongxin Liu, Xing Hu, Xin Xia, and Shanping Li. 2023. Vulnera- bility Detection by Learning From Syntax-Based Execution Paths of Code.IEEE Transactions on Software Engineering49, 8 (2023), 4196–4212. doi:10.1109/TSE. 2023.3286586

    arXiv 2023

  51. [53]

    Yaqin Zhou, Shangqing Liu, Jing Kai Siow, Xianfeng Du, and Yang Liu. 2019. De- Vign: Effective Vulnerability identification by Learning Comprehensive Program Semantics via Graph Neural Networks.arXiv (Cornell University)32 (9 2019), 10197–10207. https://arxiv.org/pdf/1909.03496

    Pith/arXiv arXiv 2019

  52. [56]

    Xin Zhou, Sicong Cao, Xiaobing Sun, and David Lo. 2025. Large Language Model for Vulnerability Detection and Repair: Literature Review and the Road Ahead.ACM Trans. Softw. Eng. Methodol.34, 5, Article 145 (May 2025), 31 pages. doi:10.1145/3708522

    doi:10.1145/3708522 2025

  53. [57]

    Xin Zhou, Ting Zhang, and David Lo. 2024. Large Language Model for Vul- nerability Detection: Emerging Results and Future Directions. InProceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results(Lisbon, Portugal)(ICSE-NIER’24). ACM, 47–51. doi:10.1145/3639476.3639762

    arXiv 2024

  54. [2021]

    Deep neural-based vulnerability discovery demystified: data, model and performance.Neural Computing and Applications33, 20 (2021), 13287–13300

    2021

  55. [2023]

    InProceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses

    Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection. InProceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses. 654–668

  56. [2024]

    IEEE Transactions on Software Engineering50, 8 (2024), 2178–2199

    Vulnerability Detection via Multiple-Graph-Based Code Representation. IEEE Transactions on Software Engineering50, 8 (2024), 2178–2199. doi:10.1109/ TSE.2024.3427815

    arXiv 2024