Pith. sign in

REVIEW 3 major objections 5 minor 62 references

It Only Gets Worse: Revisiting DL-Based Vulnerability Detectors from a Practical Perspective

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DL vulnerability detectors fail their own CWE claims, new evaluation shows

desk verdict Valuable evaluation study with a genuinely new hidden-factor observation, but the flagship F1 claim is undermined by an impossible row in Table IV and the RQ5 design has a same-sample circularity. read the letter →

arxiv 2507.09529 v1 pith:EYKREKMU submitted 2025-07-13 cs.SE

Yunqian Wang , Xiaohong Li , Yao Zhang , Yuekang Li , Zhiping Zhou , Ruitao Feng This is my paper
classification cs.SE
keywords vulnerabilitydetectiondeeplearningCWEpre-trainedmodelsscratch-trainedhiddenfactorsmodelevaluationreal-worldeffectiveness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that state-of-the-art deep learning vulnerability detectors are far less reliable in practice than their reported benchmarks suggest. Across seven detectors, including both scratch-trained and pre-trained models, the authors find low consistency within each detector's declared CWE coverage, poor precision and recall on real-world vulnerability datasets, and sharp drops in performance on newly disclosed 2024 CVEs. The paper's central positive discovery is that hidden code features beyond the CWE label—external function calls, dynamic variable values, indirect assignment, and control flow complexity—strongly influence detection outcomes. Removing or simplifying any one of these factors improved recall for all seven detectors, with the largest gains being a 35% recall increase and a 13.8% F1-score increase, suggesting that CWE-only sample selection masks the true drivers of detection success.

What carries the argument

The central mechanism is the hidden-factor dataset: five subsets built from 100 vulnerable and 100 non-vulnerable real-world samples, where the baseline is compared against versions that systematically remove indirect assignment, eliminate external function calls, replace dynamic variable values with static ones, or simplify control flow structures. These modifications isolate each factor while leaving the CWE label and vulnerability status unchanged, allowing the study to attribute changes in recall and F1-score directly to code-feature variation.

What would settle it

Run the same seven detectors on a new set of samples where a controlled transformation removes, say, external function calls while keeping the buffer size, data flow, and control-flow branching logically identical; if the reported recall and F1 improvements do not reproduce on this independent set, the hidden-factor claim is not robust.

Watch

Extended reading notes

Core claim

The paper establishes that current DL-based vulnerability detectors do not behave as their reported performance implies: they show inconsistent accuracy across the CWE categories they claim to cover, achieve limited precision and recall on realistic code, and largely fail to generalize to vulnerabilities disclosed in 2024. It further shows that pre-trained-based models are not universally superior to scratch-trained models; each paradigm has distinct strengths, with pre-trained models better on semantic vulnerabilities like CWE-20 and CWE-89 and scratch-trained models better on structured ones like CWE-119. Most importantly, the paper identifies four hidden factors—indirect assignment, external function calls, dynamic variable values, and control flow structures—that significantly affect detection performance beyond the CWE label. In controlled experiments, modifying any single one of these factors improved recall for all seven evaluated detectors, and six detectors also achieved higher F1 scores.

Load-bearing premise

The hidden-factor experiments assume that manually subtracting each factor from the baseline samples changes only that factor, leaving the vulnerability label and all other code characteristics intact; if the edits also alter code length, token distribution, or sample difficulty, the measured recall and F1 changes are confounded.

Editorial extensions

If this is right

  • Detector evaluations should report performance not only by CWE category but also by code-feature profile, since samples in the same CWE with different hidden factors can yield wildly different detection results.
  • Training datasets should incorporate hidden-factor diversity so models do not rely on superficial cues such as the presence of external calls or dynamic values rather than the actual vulnerability pattern.
  • Scratch-trained and pre-trained-based models should be selected or combined according to whether the target vulnerabilities are structured (e.g., buffer overflows) or semantic (e.g., input validation, SQL injection), rather than assuming pre-trained models are always better.
  • Scalability tests on newly disclosed CVEs should become a standard evaluation axis, because current detectors show that training-set familiarity, not model quality, often drives apparent accuracy.
  • Detector developers can check whether gains reported in hidden-factor experiments translate into real-world improvements by deploying the modified data as a diagnostic suite before release.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The hidden-factor results imply that current benchmarks overestimate detector capability because they mix code-feature distributions unevenly across CWE classes; a detector that performs well on one CWE may simply have been tested on samples with favorable external-call or control-flow structures.
  • A testable extension would be to build a diagnostic benchmark that holds CWE constant and varies the four factors systematically, then measure whether the same detectors retain the reported recall improvements on a larger, independent sample set.
  • The paper's finding that pre-trained models degrade less on 2024 vulnerabilities suggests that representation breadth, not task-specific training, is the main driver of generalization; this could be probed by comparing a pre-trained model fine-tuned with hidden-factor-balanced data against a scratch-trained model on the same test set.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes VulTegra, an evaluation framework for deep-learning-based vulnerability detectors, and applies it to seven detectors (SySeVR, VulDeeLocator, DeepWuKong, VulCNN, LineVul, PILOT, PDBert). It investigates five research questions: consistency of detectors across declared CWE categories (RQ1), real-world effectiveness (RQ2), scalability to newly disclosed vulnerabilities (RQ3), comparison of scratch-trained versus pre-trained-based models (RQ4), and the influence of four hidden code factors—indirect assignment, external function calls, dynamic variable values, and control flow structure—beyond CWE labels (RQ5). The paper reports low consistency across CWEs, limited real-world precision/recall, poor scalability to 2024 CVEs, distinct strengths for the two model families, and that modifying a single hidden factor improves recall on all seven detectors and F1 on six, with the largest gains being a 35% recall improvement and a 13.8% F1 improvement. The main stated contributions are the framework, the first comparative study of scratch-trained versus pre-trained-based detectors, and the identification of hidden factors for evaluation and dataset construction.

Significance. If the findings hold, this study provides a useful multi-dimensional, independently reproduced evaluation of state-of-the-art vulnerability detectors and strengthens the case that CWE-only sample selection is insufficient. The negative observations in RQ1–RQ3 align with and extend prior empirical work, and the claimed hidden-factor effects would be a new, actionable direction for dataset design. The paper's strengths include reproducing detectors with their original datasets and code, reporting per-CWE accuracy and real-world metrics transparently, and providing an artifact link (VulTegra). However, the central positive claim (Observation-5) is currently undermined by an arithmetic inconsistency in Table IV and by the design of the RQ5 experiments, which do not establish that the manual modifications isolate individual factors. The scalability conclusion is also based on a very small sample. These issues are fixable, but they require recomputation, additional controls, and tempering of the claims.

major comments (3)
  1. [Table IV / §IV-E] Every LineVul F1 entry in Table IV is arithmetically inconsistent with the adjacent precision and recall columns. For example, the No-factor row reports Precision=0.667 and Recall=0.020, which gives F1=2PR/(P+R)≈0.039, not 0.575. The same issue appears in Factor-1 (0.607 and 0.170 give 0.266, not 0.566), Factor-2 (0.804 and 0.370 give 0.507, not 0.713), Factor-3 (0.793 and 0.230 give 0.356, not 0.673), and Factor-4 (0.400 and 0.280 give 0.329, not 0.400). Since the headline '13.8% F1-score increase' is obtained as 0.713−0.575, that specific claim is unsupported. The authors must recompute all F1 values, correct Observation-5 and the abstract/introduction, and re-examine which detectors actually achieve higher F1 scores under each factor.
  2. [§III-C, §III-E, §IV-E] The RQ5 hidden-factor experiment conflates factor discovery with factor validation. Section III-C states that the four factors were identified by manually analyzing over 500 samples, while Section III-E constructs the hidden-factor test set from 100 vulnerable and 100 non-vulnerable samples with no stated independence from the discovery pool. Section IV-E then attributes recall and F1 changes to individual factors, assuming that each manual modification changes only the target factor. This assumption is not verified: replacing dynamic values with static values, removing external function calls, and simplifying control flow necessarily change token distribution, code length, and possibly the vulnerability's semantic difficulty. Without a held-out construction protocol, automated transformations with equivalence checks, or control datasets, Observation-5's causal claims about individual hidden factors are not established.
  3. [§IV-C / Figure 5] The scalability conclusion rests on only 20 newly disclosed CVEs across 12 CWE categories. With n=20, the detector-level counts (e.g., SySeVR 2/20, PDBert 11/20) have wide 95% binomial confidence intervals that overlap substantially; the paper reports neither intervals nor statistical tests. Consequently, the claim in Observation-3 that PDBert and PILOT show 'relatively stable' generalization compared with scratch-trained models is not statistically supported. The authors should either expand the new-CVE dataset or temper the generalization claims and report confidence intervals.
minor comments (5)
  1. [§I / Observation-1] The introduction states that 'consistency among the seven detectors remains below 80%,' but this aggregate figure is not derived in RQ1; please add the computation or report per-detector consistency rates explicitly.
  2. [§IV-A] The consistency threshold of 90% accuracy in RQ1 is arbitrary. Because CWE subsets are imbalanced, accuracy alone can be misleading; consider reporting balanced accuracy, precision/recall, or confidence intervals, and perform a sensitivity analysis around the 90% threshold.
  3. [§III-D] The statement that detectors were 'reproduced using their original datasets and code' would be stronger if version hashes, environment specifications, and reproduction scripts were included in the VulTegra artifact.
  4. [Figure 5] Figure 5 currently shows detection counts but lacks axis labels, per-detector sample sizes, and error bars or confidence intervals; please add these to make the figure self-contained.
  5. [§IV-C] The phrase '20 CVE vulnerabilities' is redundant; consider '20 CVEs' or '20 newly disclosed vulnerabilities.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper's claims are empirical and independently evaluated, though Table IV contains internal arithmetic inconsistencies.

full rationale

The paper's derivation chain is empirical rather than definitional. RQ5 identifies four hidden factors from manual analysis of misclassified samples, then constructs modified datasets and measures detector responses, which is a hypothesis-then-experiment structure and not a reduction by construction. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled via citation. The only self-citation is the framework website ([17]), which is not load-bearing. The main validity concerns are (a) the RQ5 test set's independence from the discovery pool is not stated, which could cause selection bias but is not a circular reduction, and (b) Table IV's F1 values for LineVul are arithmetically inconsistent with the reported precision and recall (e.g., P=0.667, R=0.020 implies F1≈0.039, not 0.575), undermining the claimed 13.8% F1 improvement. These are correctness/reproducibility issues, not circularity. The paper is self-contained against external detectors and datasets, so no circularity score is warranted.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper's conclusions rest on domain assumptions about label quality, faithful reproduction, representativeness of small samples, and isolation of hidden factors in manually edited code, plus a hand-chosen consistency threshold. There are no fitted model parameters in the mathematical sense, and no new physical or conceptual entities are postulated.

free parameters (1)
  • Consistency threshold = 90% accuracy
    A detector is called 'consistent' for a CWE only if its accuracy exceeds 90%; the threshold is hand-picked with no sensitivity analysis, and RQ1 conclusions depend on it.
assumptions (4)
  • domain assumption CWE labels in the datasets (NVD, SARD, Devign, Reveal, Fan) are accurate and sufficient ground truth for vulnerability classification.
    Used throughout RQ1 and dataset construction; mislabeled samples would distort per-CWE accuracy and the reported consistency rates.
  • domain assumption The reproduced detectors faithfully match the original published models and training procedures.
    Section III-D states detectors were reproduced using original datasets and code, but no reproduction artifacts or verification are supplied.
  • ad hoc to paper Manual modification of 100 vulnerable and 100 non-vulnerable samples isolates each hidden factor without changing the label or other confounding features.
    Section III-E and Section IV-E; the central RQ5 result depends on this isolation, and no validation of edit equivalence is provided.
  • domain assumption The 20 selected 2024 CVEs are representative of newly emerged vulnerabilities.
    Section IV-C; generalization claims rest on a tiny convenience sample of vulnerabilities from popular high-fork projects.

how reviews work

0 comments
Cite this review

Pith. "Pith review of It Only Gets Worse: Revisiting DL-Based Vulnerability Detectors from a Practical Perspective." pith.science (2026). https://pith.science/paper/EYKREKMU

@misc{pith2026250709529,
  author       = {Pith},
  title        = {Pith review of: It Only Gets Worse: Revisiting DL-Based Vulnerability Detectors from a Practical Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EYKREKMU}},
  note         = {Machine review of arXiv:2507.09529}
}
read the original abstract

With the growing threat of software vulnerabilities, deep learning (DL)-based detectors have gained popularity for vulnerability detection. However, doubts remain regarding their consistency within declared CWE ranges, real-world effectiveness, and applicability across scenarios. These issues may lead to unreliable detection, high false positives/negatives, and poor adaptability to emerging vulnerabilities. A comprehensive analysis is needed to uncover critical factors affecting detection and guide improvements in model design and deployment. In this paper, we present VulTegra, a novel evaluation framework that conducts a multidimensional comparison of scratch-trained and pre-trained-based DL models for vulnerability detection. VulTegra reveals that state-of-the-art (SOTA) detectors still suffer from low consistency, limited real-world capabilities, and scalability challenges. Contrary to common belief, pre-trained models are not consistently better than scratch-trained models but exhibit distinct strengths in specific contexts.Importantly, our study exposes the limitations of relying solely on CWE-based classification and identifies key factors that significantly affect model performance. Experimental results show that adjusting just one such factor consistently improves recall across all seven evaluated detectors, with six also achieving better F1 scores. Our findings provide deeper insights into model behavior and emphasize the need to consider both vulnerability types and inherent code features for effective detection.

Figures

Figures reproduced from arXiv: 2507.09529 by the authors.

Figure 1
Figure 1. Contrasting Performance of Scratch-Trained Model and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. provides an overview of the framework em￾ployed in our study to comprehensively evaluate and com￾pare DL-based vulnerability detectors, with a particular focus on scratch-trained models and pre-trained-based models. The overarching goal of this study is to produce a deep and systematic understanding of the detection capabilities of state￾of-the-art detectors, compare their strengths and weaknesses across multiple di… view at source ↗
Figure 4
Figure 4. Hidden Factors Affecting Detection Performance [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Evaluation Results for Scalability Result [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

62 extracted references · 51 canonical work pages

  1. [1]

    Common Vulnerabilities and Exposures (CVE),

    MITRE Corporation, “Common Vulnerabilities and Exposures (CVE),” https://cve.mitre.org, accessed: March 12, 2025

    work page 2025

  2. [2]

    Large scale characterization of software vulnerability life cycles,

    M. Shahzad, M. Z. Shafiq, and A. X. Liu, “Large scale characterization of software vulnerability life cycles,” IEEE Transactions on Dependable and Secure Computing , vol. 17, no. 4, pp. 730–744, 2019

    work page 2019

  3. [3]

    Savi: Static-analysis vulnerability indicator,

    J. Walden and M. Doyle, “Savi: Static-analysis vulnerability indicator,” IEEE Security & Privacy , vol. 10, no. 3, pp. 32–39, 2012

    work page 2012

  4. [4]

    Static analysis for security,

    B. Chess and G. McGraw, “Static analysis for security,” IEEE security & privacy , vol. 2, no. 6, pp. 76–79, 2004

    work page 2004

  5. [5]

    Klee: unassisted and automatic generation of high-coverage tests for complex systems programs

    C. Cadar, D. Dunbar, D. R. Engler et al., “Klee: unassisted and automatic generation of high-coverage tests for complex systems programs.” in OSDI, vol. 8, 2008, pp. 209–224

    work page 2008

  6. [6]

    Arbiter: Bridging the static and dynamic divide in vulnerability discovery on binary programs,

    J. Vadayath, M. Eckert, K. Zeng, N. Weideman, G. P. Menon, Y . Fratan- tonio, D. Balzarotti, A. Doup´e, T. Bao, R. Wanget al., “Arbiter: Bridging the static and dynamic divide in vulnerability discovery on binary programs,” in 31st USENIX Security Symposium (USENIX Security 22) , 2022, pp. 413–430

    work page 2022

  7. [7]

    The concept of dynamic analysis,

    T. Ball, “The concept of dynamic analysis,” ACM SIGSOFT Software Engineering Notes , vol. 24, no. 6, pp. 216–234, 1999

    work page 1999

  8. [8]

    Combining static analysis error traces with dynamic symbolic execution (experi- ence paper),

    F. Busse, P. Gharat, C. Cadar, and A. F. Donaldson, “Combining static analysis error traces with dynamic symbolic execution (experi- ence paper),” in Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis , 2022, pp. 568–579

    work page 2022

Show all 62 references
  1. [9]

    Software vulner- ability detection using deep neural networks: a survey,

    G. Lin, S. Wen, Q.-L. Han, J. Zhang, and Y . Xiang, “Software vulner- ability detection using deep neural networks: a survey,” Proceedings of the IEEE , vol. 108, no. 10, pp. 1825–1848, 2020

    2020

  2. [10]

    Deep semantic feature learning for software defect prediction,

    S. Wang, T. Liu, J. Nam, and L. Tan, “Deep semantic feature learning for software defect prediction,” IEEE Transactions on Software Engineering, vol. 46, no. 12, pp. 1267–1293, 2018

    2018

  3. [11]

    Scale: Construct- ing structured natural language comment trees for software vulnerability detection,

    X.-C. Wen, C. Gao, S. Gao, Y . Xiao, and M. R. Lyu, “Scale: Construct- ing structured natural language comment trees for software vulnerability detection,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2024, pp. 235–247

    2024

  4. [12]

    Multitask-based evaluation of open- source llm on software vulnerability,

    X. Yin, C. Ni, and S. Wang, “Multitask-based evaluation of open- source llm on software vulnerability,” IEEE Transactions on Software Engineering, 2024

    2024

  5. [13]

    Revisiting the performance of deep learning-based vulner- ability detection on realistic datasets,

    P. Chakraborty, K. K. Arumugam, M. Alfadel, M. Nagappan, and S. McIntosh, “Revisiting the performance of deep learning-based vulner- ability detection on realistic datasets,” IEEE Transactions on Software Engineering, 2024

    2024

  6. [14]

    Common Weakness Enumeration (CWE),

    MITRE, “Common Weakness Enumeration (CWE),” https://cwe.mitre. org/

  7. [15]

    An empirical study of deep learning models for vulnerability detection,

    B. Steenhoek, M. M. Rahman, R. Jiles, and W. Le, “An empirical study of deep learning models for vulnerability detection,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 2237–2248

    2023

  8. [16]

    Fine-grained commit-level vulnerability type prediction by cwe tree structure,

    S. Pan, L. Bao, X. Xia, D. Lo, and S. Li, “Fine-grained commit-level vulnerability type prediction by cwe tree structure,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 957–969

    2023

  9. [17]

    Vultegra home,

    VulTegra, “Vultegra home,” https://sites.google.com/view/vultegra

  10. [18]

    Combining graph-based learning with automated data collection for code vulnerability detection,

    H. Wang, G. Ye, Z. Tang, S. H. Tan, S. Huang, D. Fang, Y . Feng, L. Bian, and Z. Wang, “Combining graph-based learning with automated data collection for code vulnerability detection,” IEEE Transactions on Information F orensics and Security, vol. 16, pp. 1943–1958, 2020

    1943

  11. [19]

    Vuldeelocator: a deep learning-based fine-grained vulnerability detector,

    Z. Li, D. Zou, S. Xu, Z. Chen, Y . Zhu, and H. Jin, “Vuldeelocator: a deep learning-based fine-grained vulnerability detector,” IEEE Transactions on Dependable and Secure Computing , vol. 19, no. 4, pp. 2821–2837, 2021

    2021

  12. [20]

    Vulnerability detection with fine- grained interpretations,

    Y . Li, S. Wang, and T. N. Nguyen, “Vulnerability detection with fine- grained interpretations,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the F oundations of Software Engineering, 2021, pp. 292–303

    2021

  13. [21]

    Cvefixes: automated collec- tion of vulnerabilities and their fixes from open-source software,

    G. Bhandari, A. Naseer, and L. Moonen, “Cvefixes: automated collec- tion of vulnerabilities and their fixes from open-source software,” in Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering , 2021, pp. 30–39

    2021

  14. [22]

    Codebert: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al., “Codebert: A pre-trained model for programming and natural languages,” arXiv preprint arXiv:2002.08155 , 2020

    2002 arXiv

  15. [23]

    Graphcodebert: Pre-training code repre- sentations with data flow,

    D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu et al. , “Graphcodebert: Pre-training code repre- sentations with data flow,” arXiv preprint arXiv:2009.08366 , 2020

    2009 arXiv

  16. [24]

    Unixcoder: Unified cross-modal pre-training for code representation,

    D. Guo, S. Lu, N. Duan, Y . Wang, M. Zhou, and J. Yin, “Unixcoder: Unified cross-modal pre-training for code representation,” arXiv preprint arXiv:2203.03850, 2022

    2022 arXiv

  17. [25]

    From generalist to specialist: Exploring cwe-specific vulnerability detection,

    S. A. Atiiq, C. Gehrmann, K. Dahl ´en, and K. Khalil, “From generalist to specialist: Exploring cwe-specific vulnerability detection,” arXiv preprint arXiv:2408.02329, 2024

    2024 arXiv

  18. [26]

    Sysevr: A framework for using deep learning to detect software vulnerabilities,

    Z. Li, D. Zou, S. Xu, H. Jin, Y . Zhu, and Z. Chen, “Sysevr: A framework for using deep learning to detect software vulnerabilities,” IEEE Transactions on Dependable and Secure Computing , vol. 19, no. 4, pp. 2244–2258, 2021

    2021

  19. [27]

    Deepwukong: Statically detecting software vulnerabilities using deep graph neural network,

    X. Cheng, H. Wang, J. Hua, G. Xu, and Y . Sui, “Deepwukong: Statically detecting software vulnerabilities using deep graph neural network,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 30, no. 3, pp. 1–33, 2021

    2021

  20. [28]

    Vulcnn: An image-inspired scalable vulnerability detection system,

    Y . Wu, D. Zou, S. Dou, W. Yang, D. Xu, and H. Jin, “Vulcnn: An image-inspired scalable vulnerability detection system,” in Proceedings of the 44th International Conference on Software Engineering , 2022, pp. 2365–2376

    2022

  21. [29]

    Linevul: A transformer-based line- level vulnerability prediction,

    M. Fu and C. Tantithamthavorn, “Linevul: A transformer-based line- level vulnerability prediction,” in Proceedings of the 19th International Conference on Mining Software Repositories , 2022, pp. 608–620

    2022

  22. [30]

    When less is enough: Positive and unlabeled learning model for vulnerability detec- tion,

    X.-C. Wen, X. Wang, C. Gao, S. Wang, Y . Liu, and Z. Gu, “When less is enough: Positive and unlabeled learning model for vulnerability detec- tion,” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2023, pp. 345–357

    2023

  23. [31]

    Pre-training by predicting program dependencies for vulnerability analysis tasks,

    Z. Liu, Z. Tang, J. Zhang, X. Xia, and X. Yang, “Pre-training by predicting program dependencies for vulnerability analysis tasks,” in Proceedings of the IEEE/ACM 46th International Conference on Soft- ware Engineering, 2024, pp. 1–13

    2024

  24. [32]

    Vgx: Large-scale sample generation for boosting learning-based software vul- nerability analyses,

    Y . Nong, R. Fang, G. Yi, K. Zhao, X. Luo, F. Chen, and H. Cai, “Vgx: Large-scale sample generation for boosting learning-based software vul- nerability analyses,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 2024, pp. 1–13

    2024

  25. [33]

    Vulnerability detection via multiple-graph-based code representation,

    F. Qiu, Z. Liu, X. Hu, X. Xia, G. Chen, and X. Wang, “Vulnerability detection via multiple-graph-based code representation,” IEEE Transac- tions on Software Engineering , 2024

    2024

  26. [34]

    Vuleval: Towards repository-level evaluation of software vulnerability detection,

    X.-C. Wen, X. Wang, Y . Chen, R. Hu, D. Lo, and C. Gao, “Vuleval: Towards repository-level evaluation of software vulnerability detection,” arXiv preprint arXiv:2404.15596 , 2024

    2024 arXiv

  27. [35]

    Does data sampling improve deep learning-based vulnerability detection? yeas! and nays!

    X. Yang, S. Wang, Y . Li, and S. Wang, “Does data sampling improve deep learning-based vulnerability detection? yeas! and nays!” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 2287–2298

    2023

  28. [36]

    Interpreters for gnn-based vulnerability detection: Are we there yet?

    Y . Hu, S. Wang, W. Li, J. Peng, Y . Wu, D. Zou, and H. Jin, “Interpreters for gnn-based vulnerability detection: Are we there yet?” in Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2023, pp. 1407–1419

    2023

  29. [37]

    Path-sensitive code embedding via contrastive learning for software vulnerability detection,

    X. Cheng, G. Zhang, H. Wang, and Y . Sui, “Path-sensitive code embedding via contrastive learning for software vulnerability detection,” in Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis , 2022, pp. 519–531

    2022

  30. [38]

    National Vul- nerability Database (NVD),

    National Institute of Standards and Technology (NIST), “National Vul- nerability Database (NVD),” https://nvd.nist.gov/

  31. [39]

    Devign: Effective vul- nerability identification by learning comprehensive program semantics via graph neural networks,

    Y . Zhou, S. Liu, J. Siow, X. Du, and Y . Liu, “Devign: Effective vul- nerability identification by learning comprehensive program semantics via graph neural networks,” Advances in neural information processing systems, vol. 32, 2019

    2019

  32. [40]

    Deep learning based vulnerability detection: Are we there yet?

    S. Chakraborty, R. Krishna, Y . Ding, and B. Ray, “Deep learning based vulnerability detection: Are we there yet?” IEEE Transactions on Software Engineering , vol. 48, no. 9, pp. 3280–3296, 2021

    2021

  33. [41]

    Ac/c++ code vulnerability dataset with code changes and cve summaries,

    J. Fan, Y . Li, S. Wang, and T. N. Nguyen, “Ac/c++ code vulnerability dataset with code changes and cve summaries,” in Proceedings of the 17th international conference on mining software repositories , 2020, pp. 508–512

    2020

  34. [42]

    SARD Dataset,

    National Institute of Standards and Technology (NIST), “SARD Dataset,” https://samate.nist.gov/SRD/index.php

  35. [43]

    CWE TOP 25,

    MITRE, “CWE TOP 25,” https://cwe.mitre.org/top25/archive/2024/ 2024 cwe top25.html

    2024

  36. [44]

    Cppcheck,

    Cppcheck team, “Cppcheck,” https://cppcheck.sourceforge.io/, 2021

    2021

  37. [45]

    SonarQube - Continuous Code Quality,

    SonarSource, “SonarQube - Continuous Code Quality,” https://www. sonarsource.com/products/sonarqube/, 2021

    2021

  38. [46]

    OW ASP Dependency Check,

    Jeremy Long and OW ASP, “OW ASP Dependency Check,” https://owasp. org/www-project-dependency-check/, 2021

    2021

  39. [47]

    A large-scale study of usability criteria addressed by static analysis tools,

    M. Nachtigall, M. Schlichtig, and E. Bodden, “A large-scale study of usability criteria addressed by static analysis tools,” in Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis , 2022, pp. 532–543

    2022

  40. [48]

    ” false negative-that one is going to kill you

    A. S. Ami, K. Moran, D. Poshyvanyk, and A. Nadkarni, “” false negative-that one is going to kill you”: Understanding industry perspec- tives of static analysis based security testing,” in 2024 IEEE Symposium on Security and Privacy (SP) . IEEE, 2024, pp. 3979–3997

    2024

  41. [49]

    An empirical study on the effectiveness of static c code analyzers for vulnerability detection,

    S. Lipp, S. Banescu, and A. Pretschner, “An empirical study on the effectiveness of static c code analyzers for vulnerability detection,” in Proceedings of the 31st ACM SIGSOFT international symposium on software testing and analysis , 2022, pp. 544–555

    2022

  42. [50]

    Mvd: memory- related vulnerability detection based on flow-sensitive graph neural networks,

    S. Cao, X. Sun, L. Bo, R. Wu, B. Li, and C. Tao, “Mvd: memory- related vulnerability detection based on flow-sensitive graph neural networks,” in Proceedings of the 44th international conference on software engineering, 2022, pp. 1456–1468

    2022

  43. [51]

    {MVP}: Detecting vulnerabilities using {Patch-Enhanced} vulnerability signatures,

    Y . Xiao, B. Chen, C. Yu, Z. Xu, Z. Yuan, F. Li, B. Liu, Y . Liu, W. Huo, W. Zou et al. , “ {MVP}: Detecting vulnerabilities using {Patch-Enhanced} vulnerability signatures,” in 29th USENIX Security Symposium (USENIX Security 20) , 2020, pp. 1165–1182

    2020

  44. [52]

    µ vuldeepecker: A deep learning-based system for multiclass vulnerability detection,

    D. Zou, S. Wang, S. Xu, Z. Li, and H. Jin, “ µ vuldeepecker: A deep learning-based system for multiclass vulnerability detection,” IEEE Transactions on Dependable and Secure Computing , vol. 18, no. 5, pp. 2224–2236, 2019

    2019

  45. [53]

    Regvd: Revisiting graph neural networks for vulnerability detection,

    V .-A. Nguyen, D. Q. Nguyen, V . Nguyen, T. Le, Q. H. Tran, and D. Phung, “Regvd: Revisiting graph neural networks for vulnerability detection,” in Proceedings of the ACM/IEEE 44th International Con- ference on Software Engineering: Companion Proceedings , 2022, pp. 178–182

    2022

  46. [54]

    Deepvd: Toward class-separation features for neural network vulnera- bility detection,

    W. Wang, T. N. Nguyen, S. Wang, Y . Li, J. Zhang, and A. Yadavally, “Deepvd: Toward class-separation features for neural network vulnera- bility detection,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 2249–2261

    2023

  47. [55]

    Learning to detect memory-related vulnerabilities,

    S. Cao, X. Sun, L. Bo, R. Wu, B. Li, X. Wu, C. Tao, T. Zhang, and W. Liu, “Learning to detect memory-related vulnerabilities,” ACM Transactions on Software Engineering and Methodology , vol. 33, no. 2, pp. 1–35, 2023

    2023

  48. [56]

    Learning program semantics for vulnerability detection via vulnerability-specific inter-procedural slicing,

    B. Wu, S. Liu, Y . Xiao, Z. Li, J. Sun, and S.-W. Lin, “Learning program semantics for vulnerability detection via vulnerability-specific inter-procedural slicing,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the F oundations ...

    2023

  49. [57]

    Unified pre-training for program understanding and generation,

    W. U. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Unified pre-training for program understanding and generation,” arXiv preprint arXiv:2103.06333, 2021

    2021 arXiv

  50. [58]

    Stagedvulbert: Multi-granular vulnerability detection with a novel pre-trained code model,

    Y . Jiang, Y . Zhang, X. Su, C. Treude, and T. Wang, “Stagedvulbert: Multi-granular vulnerability detection with a novel pre-trained code model,” IEEE Transactions on Software Engineering , 2024

    2024

  51. [59]

    Vulnerability detection by learning from syntax-based execution paths of code,

    J. Zhang, Z. Liu, X. Hu, X. Xia, and S. Li, “Vulnerability detection by learning from syntax-based execution paths of code,” IEEE Transactions on Software Engineering , vol. 49, no. 8, pp. 4196–4212, 2023

    2023

  52. [60]

    Combining structured static code information and dynamic symbolic traces for software vulnerability prediction,

    H. Wang, Z. Tang, S. H. Tan, J. Wang, Y . Liu, H. Fang, C. Xia, and Z. Wang, “Combining structured static code information and dynamic symbolic traces for software vulnerability prediction,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering ,...

    2024

  53. [61]

    Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection,

    Y . Chen, Z. Ding, L. Alowain, X. Chen, and D. Wagner, “Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection,” in Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses , 2023, pp. 654–668

    2023

  54. [62]

    Vulnerability detection with code language models: How far are we?

    Y . Ding, Y . Fu, O. Ibrahim, C. Sitawarin, X. Chen, B. Alomair, D. Wagner, B. Ray, and Y . Chen, “Vulnerability detection with code language models: How far are we?” arXiv preprint arXiv:2403.18624 , 2024

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.