Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

Investigating Large Language Models for Code Vulnerability Detection: An Experimental Study

T0 review · 4 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuned open-source LLMs are effective code-vulnerability detectors, especially on long code samples, with class imbalance the decisive performance factor.

desk verdict A useful open benchmark for fine-tuned LLMs in vulnerability detection, but its headline causal claims about class imbalance and length are not supported by its own tables. read the letter →

arxiv 2412.18260 v2 pith:FTPJ57JP submitted 2024-12-24 cs.CL

Xuefeng Jiang , Lvhua Wu , Sheng Sun , Jia Li , Jingjing Xue , Yuwei Wang , Tingting Wu , Min Liu This is my paper · ORCID
classification cs.CL
keywords codevulnerabilitydetectionlargelanguagemodelsfine-tuningLoRAclassimbalancelongsamplesempiricalbenchmarkLlama
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether fine-tuned open-source large language models can detect code vulnerabilities, and answers with a five-dataset, nine-model benchmark. Its central assertion is that LoRA-fine-tuned Llama-series models are effective detectors, with a particular advantage on long code samples (over 512 tokens) that older graph-based and medium-size sequence models cannot handle. The paper further claims that the proportion of vulnerable samples in the training set plays a decisive role: model F1 collapses on imbalanced datasets and jumps when the positive-sample ratio reaches roughly 30 percent, while code length matters much less. A sympathetic reader would care because the finding reorients practical effort toward dataset balance rather than model size, and it makes long-code vulnerability detection feasible with LLMs.

What carries the argument

The argument is carried by a controlled benchmark protocol rather than a single identity. Five C/C++ function-level datasets are each split at 512 tokens into short and long subsets, and nine models are trained in one unified codebase: three graph-based models, two medium-size sequence models, and four Llama-series LLMs fine-tuned with low-rank adaptation (LoRA) using an instruction prompt that asks the model to classify the code. The load-bearing controls are the resampling experiments that fix dataset size at 25,000 and vary the positive-sample ratio in steps from 10% to 50%, and the length experiment that mixes all datasets into eight equal-size length intervals from 0 to 1024 tokens at 20% positive ratio, isolating length from imbalance. F1 is the primary metric, with precision, recall, accuracy, and false positive rate as supporting metrics.

What would settle it

Have two independent security experts re-label a random sample of, say, 300 functions from each of ReVeal, Draper, BigVul, and DiverseVul, recompute F1 for the four LLMs on the corrected labels, and check whether the class-imbalance ranking and the long-sample advantage survive; if they flatten or reverse, the headline findings are artifacts of label noise.

Watch

Extended reading notes

Core claim

The paper's central claim is empirical: fine-tuned LLMs are strong code-vulnerability detectors, especially on long samples, and the single most decisive factor in their performance is class imbalance. On long samples the four Llama-series models clearly outperform the two models that can be evaluated there, while on short samples the medium-size sequence models generally achieve higher F1 on imbalanced datasets. Controlled resampling shows F1 and recall rise sharply once vulnerable samples reach about 30% of training data, and a length-controlled experiment shows F1 declines mildly as code grows longer, far less than the imbalance effect. The paper also finds LLMs produce low false positive rates, making them more usable as triage tools.

Load-bearing premise

The benchmark scores models against dataset labels as ground truth, but four of the five datasets were labeled by automatic tools, patches, or commits rather than experts, and the paper itself calls the Draper labels of unknown and less investigated quality, so if many labels are wrong, every reported performance difference partly measures label noise rather than detection ability.

Editorial extensions

If this is right

  • LoRA-fine-tuned Llama-series models make long-sample code vulnerability detection practical, since most prior models are limited to 512 tokens and lose on long code.
  • Balancing a vulnerability dataset to at least 30% positive samples improves F1 and recall sharply for both CodeBERT and Llama-3.1, so dataset curation, not only model choice, drives results.
  • On short, imbalanced samples the medium-size sequence models still lead, meaning LLMs are not uniformly superior and should be selected per sample-length regime.
  • Small false positive rates make fine-tuned LLMs more reliable as pre-screening tools for security reviewers.
  • Larger LoRA rank improves F1 when compute allows, and a scaling factor near twice the rank performs best, giving concrete tuning guidance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If class imbalance is as decisive as reported, simple fixes such as oversampling vulnerable functions or class-weighted loss on existing imbalanced corpora should produce large F1 gains, a directly testable extension.
  • Because the paper excludes samples over 1024 tokens, the finding that length has only mild negative impact may not extend to very long functions; testing the same protocol on 1K-4K token code would show whether the length effect bends upward.
  • The label-noise caveat cuts both ways: on auto-labeled datasets, reported F1 differences may partly measure annotation quality, so future benchmarks should publish noise estimates next to scores to separate model capability from data quality.
  • A practical pipeline suggested by the data is to pair short-sample specialist models with long-sample LLMs, using code length as a routing signal.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper presents an empirical study of fine-tuned LLMs (Llama-2-7B, CodeLlama-7B, Llama-3-8B, Llama-3.1-8B) for code vulnerability detection, comparing them against three graph-based models and two medium-size sequence models on five C/C++ datasets split into short (≤512 tokens) and long (512–1024 tokens) subsets. It reports accuracy, precision, recall, F1, and FPR, and conducts resampling experiments (Section VI-C), length-controlled experiments (Section VI-D), and a LoRA hyperparameter sensitivity study (Section VI-E). The manuscript claims that class imbalance is the decisive factor limiting CVD performance, that medium-size sequence models generally outperform LLMs on short samples, and that LLMs show exceptional potential on long samples; all code and preprocessed data are released at the stated repositories.

Significance. If the findings were fully supported, the paper would be a useful benchmark contribution: it integrates nine models across five datasets in a unified codebase, evaluates recent Llama-3/3.1 models in a setting rarely covered (samples longer than 512 tokens), and appropriately uses F1 and FPR rather than accuracy for imbalanced data. The open-sourced code and datasets are a concrete asset for the community, and the decision to report FPR alongside F1 is a strength. However, the paper's headline findings are partially contradicted by its own Table III and by the controlled length experiment in Section VI-D, so the significance currently rests on the benchmark and reproducibility value rather than on the stated conclusions.

major comments (4)
  1. [§VI-B, Finding 2 and Table III] Finding 2 states that medium-size sequence models generally outperform LLMs on short samples, but Table III shows the opposite on BigVul short samples: CodeBERT, UniXcoder, and GraphCodeBERT achieve F1 of 12.60%, 11.76%, and 16.00% respectively, while Llama-2-7B, Llama-3-8B, and Llama-3.1-8B achieve 87.00%, 83.50%, and 81.48%. The finding needs to be qualified, and this quarter of the short-sample evaluation matrix requires an explanation rather than a general statement that LLM recall is significantly lower.
  2. [§VI-B, Finding 3 and §VI-D] Finding 3 claims exceptional LLM performance on long samples based on Table III, but for every dataset the long-sample subset has a higher vulnerable ratio than the short-sample subset (Table II), so the long/short comparison is confounded with class balance. The paper acknowledges this in Section VI-D, but the replacement experiment mixes all five datasets before splitting by length, which confounds length with dataset composition (datasets differ in difficulty, label quality, and positive ratio). Moreover, Figure 5 shows F1 decreasing as sample length increases, which contradicts Finding 3 rather than supporting it.
  3. [§VI-B, concluding paragraph] The claim that 'the proportion of positive samples in the dataset plays a decisive role in the CVD performance of trained models' is not supported by the cross-dataset evidence in Table III: BigVul short samples have the second-lowest positive ratio (4.46%) yet yield the best LLM F1 scores in the paper (81.48–87.00), while Devign short samples with 44.08% positive ratio yield only 47.86–61.49 for LLMs. The within-dataset resampling experiments in Section VI-C support a positive-ratio effect, but they cover only two datasets and two models, so the general 'decisive role' conclusion should be restricted to within-dataset comparisons or replaced by a dataset-level analysis.
  4. [§V-B, §V-C, §VII] The manuscript repeatedly acknowledges that only Devign has expert-curated labels, that Draper's static-analyzer labels are of 'unknown and less investigated' quality with 'low' accuracy, and that several datasets contain noisy or anomalous labels (Section V-B, Section V-C, Section VII). Because all F1/precision/recall numbers are computed against these labels as ground truth, the reported performance differences among models and datasets may partly measure label noise rather than detection capability. The paper should either provide a robustness analysis (e.g., evaluation on a clean or filtered subset, or label-noise sensitivity) or explicitly frame the findings as conditional on dataset label quality; as written, this is a significant threat to the validity of the quantitative claims.
minor comments (7)
  1. [§VI-C heading] The heading 'Analysis on Datasets with Varying Postive Sample Ratios' contains a typo: 'Postive' should be 'Positive'.
  2. [Figure captions] The captions for Figures 3 and 4 use 'Varing' instead of 'Varying'.
  3. [§VI-D heading] The heading 'Analysis on Analysis on Datasets with Varying Sample Lengths' has a duplicated phrase; it should read 'Analysis on Datasets with Varying Sample Lengths'.
  4. [§V-C and Figure 2] Section V-C states that the Alpaca instruction template is 'illustrated in Figure 1', but the prompt template appears in Figure 2; the cross-reference should be checked.
  5. [§VI-A] The sentence 'the model is trained for 5 epoch' should read '5 epochs'.
  6. [Figures 3–7] In the submitted manuscript, Figures 3–7 appear garbled (the plot regions show repeated token-like path strings instead of visible graphs), making it impossible to verify the qualitative trends discussed in Sections VI-C through VI-E from the text; the figures must be legible in the final version.
  7. [References] References [41] and [50] are duplicate entries for Clark et al. (ELECTRA); one should be removed.

Circularity Check

0 steps flagged · score 1.0 of 10

Empirical measurement study; reported results are observations from held-out test splits, and the only self-citations appear in future-work pointers, so no load-bearing circularity is present.

full rationale

This is an experimental benchmark paper, not a derivation, so the main circularity patterns do not apply. All performance numbers in Table III are measured on test splits defined by an "8:1:1" train/validation/test partition (Section VI-A), so no fitted parameter is renamed as a prediction. The paper's key claims, such as "the proportion of positive samples in the dataset plays a decisive role in the CVD performance of trained models" (Section VI-B), are empirical interpretations of the observed table, not quantities reconstructed from their own inputs. The most plausible concern—that the long-sample advantage in the main experiments is confounded with positive-sample ratio—is explicitly acknowledged by the authors: "we cannot determine definitively whether sample length or positive sample ratio is the significant factor of influencing model's performance" (Section VI-D), and they then run a separate controlled experiment rather than treating the confounded observation as a derivation. Similarly, the paper notes that four of five datasets rely on auto-labelers, patches, or commits and that label quality "is unknown and less investigated" (Section V-B), but this is a label-validity limitation disclosed in the text, not a circular step. The only self-citations are in the future-work section, e.g., references [46]–[48] on noisy-label robust training and [74]–[75] on long-tailed learning; these are pointers to planned directions and do not ground any experimental finding or design choice. Because the central results are open, externally reproducible measurements with code and datasets released, the appropriate finding is no significant circularity, with a minor deduction only for the presence of non-load-bearing self-citations.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims depend on choices about data preprocessing, label trust, and experiment design rather than on derived parameters. These are listed below; the most consequential is treating noisy labels as ground truth.

free parameters (4)
  • short-sample subsampling cap = 25,000 samples
    Section V-C caps short subsets at 25,000 samples for Draper, BigVul, and DiverseVul to control cost; this changes positive ratios and which examples are trained on.
  • sequence length cutoffs = 512 and 1024 tokens
    Section V-C uses 512 as the short/long boundary (RoBERTa limit) and 1024 as the upper cutoff, discarding longer samples; the long-sample findings depend on this cutoff.
  • LoRA rank and scaling factor = rank 16, alpha 32
    Main experiments fix LoRA rank 16 and alpha 32, chosen by hand from VulLLM practice; sensitivity analysis in Section VI-E shows F1 varies with these settings.
  • positive-ratio levels in resampling = 10%, 20%, 30%, 40%, 50%
    Section VI-C resamples Draper and DiverseVul to these ratios to demonstrate imbalance effects; the '30% threshold' conclusion depends on these chosen levels.
assumptions (4)
  • domain assumption Dataset labels are treated as ground truth when computing F1, precision, recall, and FPR.
    Section V-B acknowledges label quality is 'unknown' for Draper and that other datasets use auto-labelers and commit heuristics, yet all metrics in Table III are computed against these labels.
  • ad hoc to paper Samples longer than 1024 tokens can be excluded without invalidating long-sample conclusions.
    Section V-C excludes >1024-token samples due to cost and length variation; the 'LLMs excel on long samples' finding is defined on the 512-1024 window only.
  • domain assumption RoBERTa-based models cannot be evaluated on long samples without reinitializing positional encodings.
    Section V-A and VI-B state CodeBERT, GraphCodeBERT, and UniXcoder are limited to 512 tokens; this motivates why long-sample comparisons exclude them.
  • domain assumption Equal-size samples from a mixture of five datasets control for length effects in the length study.
    Section VI-D mixes all datasets to build 10,000-sample intervals at 20% positive ratio, but dataset composition per interval is not reported, so length and dataset identity are potentially confounded.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Investigating Large Language Models for Code Vulnerability Detection: An Experimental Study." pith.science (2026). https://pith.science/paper/FTPJ57JP

@misc{pith2026241218260,
  author       = {Pith},
  title        = {Pith review of: Investigating Large Language Models for Code Vulnerability Detection: An Experimental Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FTPJ57JP}},
  note         = {Machine review of arXiv:2412.18260}
}
read the original abstract

Code vulnerability detection (CVD) is essential for addressing and preventing system security issues, playing a crucial role in ensuring software security. Previous learning-based vulnerability detection methods rely on either fine-tuning medium-size sequence models or training smaller neural networks from scratch. Recent advancements in large pre-trained language models (LLMs) have showcased remarkable capabilities in various code intelligence tasks including code understanding and generation. However, the effectiveness of LLMs in detecting code vulnerabilities is largely under-explored. This work aims to investigate the gap by fine-tuning LLMs for the CVD task, involving four widely-used open-source LLMs. We also implement other five previous graph-based or medium-size sequence models for comparison. Experiments are conducted on five commonly-used CVD datasets, including both the part of short samples and long samples. In addition, we conduct quantitative experiments to investigate the class imbalance issue and the model's performance on samples of different lengths, which are rarely studied in previous works. To better facilitate communities, we open-source all codes and resources of this study in https://github.com/SakiRinn/LLM4CVD and https://huggingface.co/datasets/xuefen/VulResource.

Figures

Figures reproduced from arXiv: 2412.18260 by the authors.

Figure 1
Figure 1. Processing procedures for sequence-based models and graph-based models. We use simple naive tokenizer in this figure as an illustrative example. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Prompt template for large language models. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Metrics on Varing Positive Sample Ratio on the DiverseVul [22] Dataset. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Metrics on Varing Positive Sample Ratio on the Draper [26] Dataset. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Metrics on Varying Code Sequence Length. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Sensitivity Study on LoRA Rank 4 8 16 32 64 50 55 60 65 70 75 80 Accuracy (%) 4 8 16 32 64 40 50 60 70 80 90 Precision (%) 4 8 16 32 64 30 35 40 45 50 55 60 65 70 75 Recall (%) 4 8 16 32 64 0 5 10 15 20 25 30 35 40 45 FPR (%) 4 8 16 32 64 35 40 45 50 55 60 65 70 F1-Sco…
Figure 7
Figure 7. Figure 7: As the scaling factor increases, the F1-score first rises [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code

    cs.CR 2025-06 conditional novelty 5.0 of 10

    A 558-task benchmark with SAST and LLM judging shows state-of-the-art LLMs generate vulnerability-free code only about 37% of the time in zero-shot settings.

Reference graph

Works this paper leans on

77 extracted references · 56 canonical work pages · cited by 1 Pith paper

  1. [1]

    An investigation of quality issues in vulnerability detection datasets

    Guo Y , Bettaieb S. An investigation of quality issues in vulnerability detection datasets. 2023 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW). IEEE, 2023

    work page 2023

  2. [2]

    Devign: Effective vulnerability identifi- cation by learning comprehensive program semantics via graph neural networks

    Zhou Y , Liu S, Siow J, et al. Devign: Effective vulnerability identifi- cation by learning comprehensive program semantics via graph neural networks. Advances in neural information processing systems, 2019

    work page 2019

  3. [3]

    Codebert: A pre-trained model for programming and natural languages

    Feng Z, Guo D, Tang D, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155, 2020

    arXiv 2002

  4. [4]

    The Devil is in the Tails: How Long- Tailed Code Distributions Impact Large Language Models

    Zhout X, Kim K, Xu B, et al. The Devil is in the Tails: How Long- Tailed Code Distributions Impact Large Language Models. The 38th IEEE/ACM International Conference on Automated Software Engineer- ing (ASE). IEEE, 2023: 40-52

    work page 2023

  5. [5]

    Large language model for vulnerability detec- tion: Emerging results and future directions

    Zhou X, Zhang T, Lo D. Large language model for vulnerability detec- tion: Emerging results and future directions. Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results. 2024. 9https://github.com/soarsmu/ChatGPT-VulDetection

    work page 2024

  6. [6]

    StagedVulBERT: Multi-Granular Vul- nerability Detection with a Novel Pre-trained Code Model[J]

    Jiang Y , Zhang Y , Su X, et al. StagedVulBERT: Multi-Granular Vul- nerability Detection with a Novel Pre-trained Code Model[J]. IEEE Transactions on Software Engineering, 2024

    work page 2024

  7. [7]

    Learning from noisy labels with deep neural networks: A survey[J]

    Song H, Kim M, Park D, et al. Learning from noisy labels with deep neural networks: A survey[J]. IEEE transactions on neural networks and learning systems, 2022, 34(11): 8135-8153

    work page 2022

  8. [8]

    Finding software vulnerabilities by smart fuzzing

    Bekrar S, Bekrar C, Groz R, et al. Finding software vulnerabilities by smart fuzzing. Fourth IEEE International Conference on Software Testing, Verification and Validation. IEEE, 2011: 427-430

    work page 2011

Show all 77 references
  1. [9]

    Class imbalance should not throw you off balance: Choosing the right classifiers and performance metrics for brain decoding with imbalanced data[J]

    Th ¨olke P, Mantilla-Ramos Y J, Abdelhedi H, et al. Class imbalance should not throw you off balance: Choosing the right classifiers and performance metrics for brain decoding with imbalanced data[J]. Neu- roImage, 2023

    2023

  2. [10]

    A comprehensive survey on graph neural networks

    Wu Z, Pan S, Chen F, et al. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 2020, 32(1): 4-24

    2020

  3. [11]

    Bayling: Bridging cross-lingual alignment and instruction following through interactive translation for large language models[J]

    Zhang S, Fang Q, Zhang Z, et al. Bayling: Bridging cross-lingual alignment and instruction following through interactive translation for large language models[J]. arXiv preprint arXiv:2306.10968, 2023

    2023 arXiv

  4. [12]

    The rise of software vulnerability: Taxonomy of software vulnerabilities detection and machine learning approaches

    Hanif H, Nasir M H N M, Ab Razak M F, et al. The rise of software vulnerability: Taxonomy of software vulnerabilities detection and machine learning approaches. Journal of Network and Computer Applications, 2021, 179: 103009

    2021

  5. [13]

    Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network

    Sherstinsky A. Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network. Physica D: Nonlinear Phe- nomena, 2020, 404: 132306

    2020

  6. [14]

    Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit

    Wan Y , Bi Z, He Y , et al. Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit. ACM Computing Surveys, 2024

    2024

  7. [15]

    GPT-3: Its nature, scope, limits, and conse- quences[J]

    Floridi L, Chiriatti M. GPT-3: Its nature, scope, limits, and conse- quences[J]. Minds and Machines, 2020, 30: 681-694

    2020

  8. [16]

    Drivelm: Driving with graph visual question answering

    Sima C, Renz K, Chitta K, et al. Drivelm: Driving with graph visual question answering. European Conference on Computer Vision, 2024

    2024

  9. [17]

    Generalization-Enhanced Code Vulnera- bility Detection via Multi-Task Instruction Fine-Tuning

    Du X, Wen M, Zhu J, et al. Generalization-Enhanced Code Vulnera- bility Detection via Multi-Task Instruction Fine-Tuning. arXiv preprint arXiv:2406.03718, https://arxiv.org/abs/2406.03718, June. 2024

    2024 arXiv

  10. [18]

    The falcon series of open language models[J]

    Almazrouei E, Alobeidli H, Alshamsi A, et al. The falcon series of open language models[J]. arXiv preprint arXiv:2311.16867, 2023

    2023 arXiv

  11. [19]

    How far have we gone in vulnerability detection using large language models

    Gao Z, Wang H, Zhou Y , et al. How far have we gone in vulnerability detection using large language models. arXiv preprint arXiv:2311.12420, https://arxiv.org/abs/2311.12420, 2023

    2023 arXiv

  12. [20]

    Pre-trained model-based automated software vulnerability repair: How far are we?

    Zhang Q, Fang C, Yu B, et al. Pre-trained model-based automated software vulnerability repair: How far are we?. IEEE Transactions on Dependable and Secure Computing, 2023

    2023

  13. [21]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022

    2022

  14. [22]

    Chen Y , Ding Z, Alowain L, et al. Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection.Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses. 2023: 654-668

    2023

  15. [23]

    A C/C++ code vulnerability dataset with code changes and CVE summaries

    Fan J, Li Y , Wang S, et al. A C/C++ code vulnerability dataset with code changes and CVE summaries. Proceedings of the 17th International Conference on Mining Software Repositories. 2020

    2020

  16. [24]

    Deep learning based vulner- ability detection: Are we there yet?

    Chakraborty S, Krishna R, Ding Y , et al. Deep learning based vulner- ability detection: Are we there yet?. IEEE Transactions on Software Engineering, 2021

    2021

  17. [25]

    D2a: A dataset built for ai- based vulnerability detection methods using differential analysis

    Zheng Y , Pujar S, Lewis B, et al. D2a: A dataset built for ai- based vulnerability detection methods using differential analysis. 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 2021

    2021

  18. [26]

    Draper VDISC Dataset - Vulnerability Detection in Source Code

    Louis Kim, Rebecca Russell. Draper VDISC Dataset - Vulnerability Detection in Source Code. https://osf.io/d45bw/

  19. [27]

    Pytorch: An imperative style, high-performance deep learning library

    Paszke A, Gross S, Massa F, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 2019, 32

    2019

  20. [28]

    Code Llama: Open foundation models for code

    Roziere B, Gehring J, Gloeckle F, et al. Code Llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  21. [29]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron H, Martin L, Stone K, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  22. [30]

    G, Lukasz K, and Illia P

    Vaswani, A, Noam S, Niki P, Jakob U, Llion J, AidanN. G, Lukasz K, and Illia P. Attention Is All You Need. Neural Information Processing Systems,Neural Information Processing Systems, June, 2017

    2017

  23. [31]

    W., Lee, K., & Toutanova, K

    Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. . BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 12

    2018 arXiv

  24. [32]

    Code vulnerability detection based on deep sequence and graph models: A survey

    Wu B, Zou F. Code vulnerability detection based on deep sequence and graph models: A survey. Security and Communication Networks, 2022, 2022(1): 1176898

    2022

  25. [33]

    Empirical evaluation of gated recurrent neural networks on sequence modeling

    Chung J, Gulcehre C, Cho K H, et al. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014

    2014 arXiv

  26. [34]

    A review of recurrent neural networks: LSTM cells and network architectures

    Yu Y , Si X, Hu C, et al. A review of recurrent neural networks: LSTM cells and network architectures. Neural computation, 2019, 31(7): 1235- 1270

    2019

  27. [35]

    The Llama 3 herd of models

    Dubey A, Jauhri A, Pandey A, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  28. [36]

    L2 regularization versus batch and weight normaliza- tion

    Van Laarhoven T. L2 regularization versus batch and weight normaliza- tion. arXiv preprint arXiv:1706.05350, 2017

    2017 arXiv

  29. [37]

    Chain-of-thought prompting of large language models for discovering and fixing software vulnerabilities

    Nong Y , Aldeen M, Cheng L, et al. Chain-of-thought prompting of large language models for discovering and fixing software vulnerabilities. arXiv preprint arXiv:2402.17230, 2024

    2024 arXiv

  30. [38]

    ReGVD: Revisiting graph neural networks for vulnerability detection

    Nguyen V A, Nguyen D Q, Nguyen V , et al. ReGVD: Revisiting graph neural networks for vulnerability detection. Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings. 2022

    2022

  31. [39]

    Vulnerability detection by learning from syntax-based execution paths of code

    Zhang J, Liu Z, Hu X, et al. Vulnerability detection by learning from syntax-based execution paths of code. IEEE Transactions on Software Engineering, 2023, 49(8): 4196-4212

    2023

  32. [40]

    Gpt-4 technical report

    Achiam J, Adler S, Agarwal S, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  33. [42]

    Graphcodebert: Pre-training code represen- tations with data flow

    Guo D, Ren S, Lu S, et al. Graphcodebert: Pre-training code represen- tations with data flow. arXiv preprint arXiv:2009.08366, 2020

    2009 arXiv

  34. [43]

    Investigating the effectiveness of BPE: The power of shorter sequences

    Gall ´e M. Investigating the effectiveness of BPE: The power of shorter sequences. Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 2019

    2019

  35. [44]

    Data quality for software vulnerability datasets

    Croft R, Babar M A, Kholoosi M M. Data quality for software vulnerability datasets. 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023

    2023

  36. [45]

    Evaluating the data inconsistency of open- source vulnerability repositories

    Jiang Y , Jeusfeld M, Ding J. Evaluating the data inconsistency of open- source vulnerability repositories. Proceedings of the 16th International Conference on Availability, Reliability and Security. 2021: 1-10

    2021

  37. [46]

    Tackling Noisy Clients in Federated Learning with End-to-end Label Correction

    Jiang X, Sun S, Li J, et al. Tackling Noisy Clients in Federated Learning with End-to-end Label Correction. arXiv preprint arXiv:2408.04301, 2024

    2024 arXiv

  38. [47]

    Towards federated learning against noisy labels via local self-regularization[C]//Proceedings of the 31st ACM International Conference on Information & Knowledge Management

    Jiang X, Sun S, Wang Y , et al. Towards federated learning against noisy labels via local self-regularization[C]//Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 2022: 862-873

    2022

  39. [48]

    FNBench: Benchmarking Robust Federated Learning against Noisy Labels

    Jiang X, Li J, Wu N, et al. FNBench: Benchmarking Robust Federated Learning against Noisy Labels. Authorea Preprints, 2024

    2024

  40. [49]

    UniXcoder: Unified Cross-Modal Pre- training for Code Representation

    Guo D, Lu S, Duan N, et al. UniXcoder: Unified Cross-Modal Pre- training for Code Representation. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2022

    2022

  41. [50]

    Electra: Pre-training text encoders as discriminators rather than generators

    Clark K. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020

    2003 arXiv

  42. [51]

    DP-CCL: A Supervised Con- trastive Learning Approach Using CodeBERT Model in Software Defect Prediction

    Sahar S, Younas M, Khan M M, et al. DP-CCL: A Supervised Con- trastive Learning Approach Using CodeBERT Model in Software Defect Prediction. IEEE Access, 2024

    2024

  43. [52]

    Sysevr: A framework for using deep learning to detect software vulnerabilities

    Li Z, Zou D, Xu S, et al. Sysevr: A framework for using deep learning to detect software vulnerabilities. IEEE Transactions on Dependable and Secure Computing, 2021, 19(4): 2244-2258

    2021

  44. [53]

    Vuldeepecker: A deep learning-based system for vulnerability detection

    Li Z, Zou D, Xu S, et al. Vuldeepecker: A deep learning-based system for vulnerability detection. Network and Distributed Systems Security (NDSS) Symposium, 2018

    2018

  45. [54]

    MegaVul: AC/C++ Vulnerability Dataset with Comprehensive Code Representations

    Ni C, Shen L, Yang X, et al. MegaVul: AC/C++ Vulnerability Dataset with Comprehensive Code Representations. 2024 IEEE/ACM 21st In- ternational Conference on Mining Software Repositories (MSR). IEEE, 2024: 738-742

    2024

  46. [55]

    Zheng Q, Xia X, Zou X, et al. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval- x[C]//Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023: 5673-5684

    2023

  47. [56]

    Cursor - The AI code editor, https://www.cursor.com/, accessed in Nov, 2024

    2024

  48. [57]

    Alpaca: A strong, replicable instruction-following model[J]

    Taori R, Gulrajani I, Zhang T, et al. Alpaca: A strong, replicable instruction-following model[J]. Stanford Center for Research on Founda- tion Models. https://crfm. stanford. edu/2023/03/13/alpaca. html, 2023, 3(6): 7

    2023

  49. [58]

    On the robustness of code generation techniques: An empirical study on github copi- lot[C]//2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)

    Mastropaolo A, Pascarella L, Guglielmi E, et al. On the robustness of code generation techniques: An empirical study on github copi- lot[C]//2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023: 2149-2160

    2023

  50. [59]

    Alpaca: A strong, replicable instruction-following model[J]

    Taori R, Gulrajani I, Zhang T, et al. Alpaca: A strong, replicable instruction-following model[J]. Stanford Center for Research on Foun- dation Models. https://crfm.stanford.edu/2023/03/13/alpaca. html, 2023, 3(6): 7

    2023

  51. [60]

    Cgems: A metric model for automatic code generation using gpt-3[J]

    Narasimhan A, Rao K P A V . Cgems: A metric model for automatic code generation using gpt-3[J]. arXiv preprint arXiv:2108.10168, 2021

    2021 arXiv

  52. [61]

    Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT[J]

    XIA C, ZHANG L. Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT[J]. 2023

    2023

  53. [62]

    Modeling and Dis- covering Vulnerabilities with Code Property Graphs[C/OL]//2014 IEEE Symposium on Security and Privacy, San Jose, CA

    Y AMAGUCHI F, GOLDE N, ARP D, et al. Modeling and Dis- covering Vulnerabilities with Code Property Graphs[C/OL]//2014 IEEE Symposium on Security and Privacy, San Jose, CA. 2014. http://dx.doi.org/10.1109/sp.2014.44. DOI:10.1109/sp.2014.44

    2014 doi

  54. [63]

    VulChecker: Graph-based Vul- nerability Localization in Source Code[C]//32nd USENIX Security Symposium (USENIX Security 23)

    Mirsky Y , Macon G, Brown M, et al. VulChecker: Graph-based Vul- nerability Localization in Source Code[C]//32nd USENIX Security Symposium (USENIX Security 23). 2023: 6557-6574

    2023

  55. [64]

    An empirical study of deep learning models for vulnerability detection[C]//2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)

    Steenhoek B, Rahman M M, Jiles R, et al. An empirical study of deep learning models for vulnerability detection[C]//2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023: 2237-2248

    2023

  56. [65]

    Distributed representations of words and phrases and their compositionality[J]

    Mikolov T, Sutskever I, Chen K, et al. Distributed representations of words and phrases and their compositionality[J]. Advances in neural information processing systems, 2013, 26

    2013

  57. [66]

    https://github.com/joernio/joern/, accessed in Nov, 2024

    Joern. https://github.com/joernio/joern/, accessed in Nov, 2024

    2024

  58. [67]

    Fair text-to-image diffusion via fair mapping[J]

    Li J, Hu L, Zhang J, et al. Fair text-to-image diffusion via fair mapping[J]. arXiv preprint arXiv:2311.17695, 2023

    2023 arXiv

  59. [68]

    Direct preference optimization: Your language model is secretly a reward model[J]

    Rafailov R, Sharma A, Mitchell E, et al. Direct preference optimization: Your language model is secretly a reward model[J]. Advances in Neural Information Processing Systems, 2024, 36

    2024

  60. [69]

    Roberta: A robustly optimized bert pretraining approach[J]

    Liu Y . Roberta: A robustly optimized bert pretraining approach[J]. arXiv preprint arXiv:1907.11692, 2019, 364

    1907 arXiv

  61. [70]

    Integrating static and dynamic analysis for de- tecting vulnerabilities[C]//30th Annual International Computer Software and Applications Conference (COMPSAC’06)

    Aggarwal A, Jalote P. Integrating static and dynamic analysis for de- tecting vulnerabilities[C]//30th Annual International Computer Software and Applications Conference (COMPSAC’06). IEEE, 2006, 1: 343-350

    2006

  62. [71]

    A systematic literature review of actionable alert identification techniques for automated static code analysis[J]

    Heckman S, Williams L. A systematic literature review of actionable alert identification techniques for automated static code analysis[J]. Information and Software Technology, 2011, 53(4): 363-387

    2011

  63. [72]

    The case for 4-bit precision: k-bit inference scaling laws[C]//International Conference on Machine Learning

    Dettmers T, Zettlemoyer L. The case for 4-bit precision: k-bit inference scaling laws[C]//International Conference on Machine Learning. PMLR, 2023: 7750-7774

    2023

  64. [73]

    Qlora: Efficient finetuning of quantized llms[J]

    Dettmers T, Pagnoni A, Holtzman A, et al. Qlora: Efficient finetuning of quantized llms[J]. Advances in Neural Information Processing Systems, 2024, 36

    2024

  65. [74]

    FedLF: Adaptive Logit Adjustment and Fea- ture Optimization in Federated Long-Tailed Learning[J]

    Lu X, Li P, Jiang X. FedLF: Adaptive Logit Adjustment and Fea- ture Optimization in Federated Long-Tailed Learning[J]. arXiv preprint arXiv:2409.12105, 2024

    2024 arXiv

  66. [75]

    Federated Classification Tasks in Long- tailed Data Environments via Classifier Representation Adjustment and Calibration[J]

    Li X, Sun S, Liu M, et al. Federated Classification Tasks in Long- tailed Data Environments via Classifier Representation Adjustment and Calibration[J]. Authorea Preprints, 2023

    2023

  67. [76]

    SMOTE: synthetic minority over-sampling technique[J]

    Chawla N V , Bowyer K W, Hall L O, et al. SMOTE: synthetic minority over-sampling technique[J]. Journal of artificial intelligence research, 2002, 16: 321-357

    2002

  68. [77]

    DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence[J]

    Guo D, Zhu Q, Yang D, et al. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence[J]. arXiv preprint arXiv:2401.14196, 2024

    2024 arXiv

  69. [78]

    Mistral 7B[J]

    Jiang A Q, Sablayrolles A, Mensch A, et al. Mistral 7B[J]. arXiv preprint arXiv:2310.06825, 2023. Xuefeng Jiang is currently a Ph.D. candidate with the Institute of Computing Technology, Chinese Academy of Sciences. Before that, he received his bachelor’s degree with honors in...

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.