REVIEW 4 major objections 6 minor
From Lab to Reality: A Practical Evaluation of Deep Learning Models and LLMs for Vulnerability Detection
T0 review · 4 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read This paper shows that deep-learning and LLM vulnerability detectors, strong on standard benchmarks, fail to distinguish vulnerable code in a small set of newly disclosed Linux kernel vulnerabilities.
desk verdict The VentiVul dataset and the negative result are worth a referee, but Finding 8's 'superior ability' rests on at-chance counts and should be tempered before publication. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
VentiVul, a manually curated out-of-distribution test set of 20 Linux-kernel CVEs fixed in May 2025, containing 25 vulnerable/patched function pairs plus 835 unrelated functions from the same files. The paper's central evaluation devices are the Whole-File setting (classify every function in a file) and the Function-Pair setting (compare each before-fix function with its after-fix counterpart; a correct judgment requires the model to score the vulnerable version higher than the patched one). The pairing isolates whether a model captures the semantic change of a fix, which is the minimal ability needed to spot new vulnerabilities. t-SNE and centroid distance serve as the representation-qualit
What would settle it
Re-run the same VentiVul Function-Pair evaluation with several prompt phrasings and decoding settings (e.g., temperature 0 vs 0.7, top-p variation, repeated sampling) for GPT-4o and GPT-5; if these models then differentiate substantially more than 6 of 25 pairs, or show large variance across runs, the reported failure is an artifact of the specific prompt rather than a stable property of the models.
Extended reading notes
Core claim
On a time-wise out-of-distribution dataset (VentiVul) of 20 recent Linux kernel CVEs, the authors find that both representative DL models (ReVeal and LineVul) and four pretrained LLMs (Claude 3.5 Sonnet, GPT-o3-mini, GPT-4o, and GPT-5) fail to reliably distinguish vulnerable from non-vulnerable code. Under Whole-File evaluation, the best F1 among DL variants is 8.13 and among LLMs 4.9; under Function-Pair evaluation, at most 6 of 25 before/after-fix pairs are correctly differentiated (GPT-5), with most models scoring zero. Representation analysis using t-SNE and centroid distances shows heavy overlap between vulnerable and non-vulnerable functions, and cross-dataset experiments show sharp pe
Load-bearing premise
The RQ3 LLM conclusions rest on treating a single hand-written prompt with unstated decoding settings, run once per model, as a fair and stable measure of each LLM's vulnerability-detection ability; prompt phrasing and sampling parameters are known to change LLM outputs, and no sensitivity analysis shows they would not.
Editorial extensions
If this is right
- Benchmark scores on Juliet, Devign, BigVul, and ICVul do not transfer to time-wise out-of-distribution code; deployment-oriented evaluation changes the reported ranking of models.
- Dataset label quality and balance, not raw volume, dominate cross-dataset generalization; BigVul's noisy labels inflate recall but produce near-zero transferability.
- Function-Pair evaluation is a cheap, interpretable way to test whether a detector understands the effect of a security patch; current DL models mostly fail it, suggesting they rely on superficial cues.
- LLMs show relatively stronger patch-sensitivity (GPT-4o and GPT-5 differentiate 5-6 of 25 pairs) despite low recall, pointing to patch-level reasoning as a promising direction rather than whole-file binary classification.
- Current representation learning—graph-based and token-based—does not yield separable vulnerable/non-vulnerable clusters, so improving representations, not just architectures, is the bottleneck.
Reading between the lines
- Because VentiVul contains only 20 CVEs and 25 pairs from one kernel, the quantitative F1 values are noisy; the paper's durable contribution is the evaluation protocol (temporal split plus before/after pairing), not the precise numbers.
- A natural extension is to use VentiVul-style Function-Pair evaluation as a deployment filter: a detector that cannot tell a function from its own patch is unlikely to catch new bugs, regardless of benchmark F1.
- Prompt sensitivity is untested; varying prompts, temperatures, and repeated sampling could move LLM results substantially, so the 'LLMs fail' result should be read as 'under this one zero-shot protocol,' not as a law.
- The finding that distances between different vulnerability types sometimes exceed distances between vulnerable and non-vulnerable code suggests that CWE-specific or root-cause-conditioned models may be a more informative target than a single binary classifier.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper evaluates two widely used vulnerability detection models (ReVeal and LineVul) on four benchmarks (Juliet, Devign, BigVul, ICVul) and, together with four commercial LLMs, on a newly curated out-of-distribution dataset VentiVul of 20 Linux kernel CVEs fixed in May 2025. Three research questions are addressed: (RQ1) whether learned code representations separate vulnerable and non-vulnerable functions, (RQ2) which dataset/model factors affect within- and cross-dataset performance, and (RQ3) whether DL models and LLMs detect recent, previously unseen vulnerabilities in a Whole-File and Function-Pair setting. The main reported findings are that embeddings show limited separation, cross-dataset performance drops sharply, most models have very low F1 on VentiVul, and GPT-4o/GPT-5 differentiate 5-6 of 25 vulnerable/patched pairs, which the paper interprets as superior patch-level reasoning. The paper argues that benchmark results overestimate real-world applicability.
Significance. If the negative results are reliable, the paper adds to a growing body of evidence (Real-Vul, PRIMEVUL) that benchmark-trained vulnerability detectors fail on temporally held-out data. The authors contribute a carefully curated, manually inspected VentiVul dataset, a reproducible framework with code and data release, and a Function-Pair evaluation mode that is more realistic than simple function-level classification. The Whole-File analysis is also useful in exposing all-negative/all-positive prediction biases. The central negative claims are plausible and consistent with prior work. However, the quantitative support is weaker than the narrative: no confidence intervals or significance tests, t-SNE-based distances used as quantitative evidence, a single prompt/run for LLMs, and a positive LLM result that is statistically indistinguishable from chance. These issues are localized but load-bearing for the positive 'promise' claim and for several RQ1/RQ3 statements.
major comments (4)
- [Section 4 (RQ3), Table 7, Finding 8] The positive LLM claim is not supported by the reported counts. GPT-4o and GPT-5 differentiate 5 and 6 of 25 pairs. Under any independent-label null, the expected number of differentiated pairs is at most 25 * 0.25 = 6.25, so the observed values are at or below the maximum chance expectation. No confidence interval, null model, or significance test is reported. The conclusion that GPT-4o/GPT-5 exhibit 'superior ability to discern fine-grained semantic changes' and the sentence in the conclusion that 'LLMs hold promise in understanding patch-level changes' therefore need either a proper statistical test (e.g., exact binomial test with a stated null, or permutation test) or should be removed/qualified. The Table 6 F1 numbers still support the negative OOD result; this issue concerns the paper's only positive LLM claim.
- [Section 3.2, Table 3] LLM results are based on a single hand-written prompt per setting with no decoding parameters (temperature, top-p, repetition penalty, max tokens) and one run per model. LLM outputs are highly sensitive to prompt wording and sampling. Tables 6-7 and Findings 7-8 therefore may reflect one arbitrary configuration rather than model ability. Please report exact API/model version and decoding settings, run multiple seeds, and report variance or use a fixed deterministic setting with justification. Without this, the RQ3 comparisons among LLMs and the relative claim about GPT-4o/GPT-5 are not reproducible.
- [Section 3.3.1, Eq. (1)] Centroid distance is computed after t-SNE dimensionality reduction and min-max normalization. t-SNE is stochastic and does not preserve pairwise distances; hyperparameters (perplexity, learning rate, iterations) are not reported. Under these conditions D_centroid is not a meaningful quantitative measure of class separability. This quantity underlies Findings 1-3. Please either replace or augment it with a distance-preserving or embedding-stable measure (e.g., centroids in the original feature space, or silhouette/other intrinsic metrics), report sensitivity to t-SNE hyperparameters, or explicitly label the centroid distances as illustrative rather than quantitative.
- [Section 4, RQ2/RQ3] The paper's comparative conclusions rest on point estimates from a single training/evaluation run on small data (25 pairs/20 CVEs, with four datasets). For example, the statement that some models 'transfer better' is based on F1 differences that are not accompanied by variance or significance information, and models with F1<20 are excluded from ranking to avoid trivial results. Please add uncertainty quantification (multiple seeds/LLM runs, or at least exact binomial confidence intervals for the categorical VentiVul counts) and justify the exclusion rule. This does not undermine the large drop in Table 6, but it is needed before claiming model or dataset ordering.
minor comments (6)
- [Section 2.1, Table 1] ICVul is the authors' own dataset and its 92% label accuracy is the result of the authors' manual validation of 50 samples. Given that ICVul is later used to argue that clean labels lead to transferable representations (Finding 5), an independent audit or a clearer statement of provenance and potential conflict would strengthen the paper.
- [Table 7] The definitions of 'Vul Detected' (out of 20 CVEs) and 'Vul Funcs Detected' (out of 25 pairs) should be stated in the caption or text, because some CVEs contribute multiple function pairs; otherwise the denominators are confusing.
- [Figure 5] A 'non-decrease' conflates unchanged scores and increased scores; plotting these separately (or counting increases) would help interpret the Pairs-Differentiated results.
- [Section 3.2/Table 3] The Whole-File prompt asks for 'method(s)' and an output format 'like ...'; it is unclear how free-form outputs are parsed into function names and how ambiguous cases are handled. Add the parsing/validation procedure.
- [Throughout] The model name 'GPT-o3-mini' is used inconsistently (also rendered as GPT-o3-mini in the abstract; OpenAI branding is 'o3-mini').
- [Section 7, Related Work] The discussion could better contrast VentiVul with the closely related Real-Vul [6] and PRIMEVUL [13] datasets in terms of temporal separation, manual curation, and the Function-Pair evaluation mode.
Circularity Check
No derivation-level circularity; the negative VentiVul result is externally grounded. Minor self-citation/self-validation attaches to the ICVul benchmark-quality claim, but the central conclusion does not reduce to it.
-
other
[Section 2.1 (Table 1) and Section 3.1.2; reference [24]]
"For ICVul, we manually evaluated label accuracy by randomly sampling 50 vulnerable functions, identifying their corresponding patch commits, and validating labels based on commit messages and code changes. ... Although ICVul is a newly published dataset, it stands out for its relatively high vulnerability ratio (41%) and high label accuracy (92%) ..."
ICVul is the authors' own prior dataset [24]. Its 'high-quality'/92% label-accuracy characterization, used to select it as a benchmark and to support RQ2 conclusions about transferability, is justified by the authors' own manual validation rather than by any independent external check. This is a self-referential evidence chain, but it is minor: the paper's central VentiVul negative result relies on externally anchored CVE fix commits and independent model evaluations, not on ICVul quality.
full rationale
The paper is an empirical evaluation, not a derivation. There is no equation in which an output is identical to an input by construction, no fitted parameter renamed as a prediction, and no uniqueness theorem imported from the authors' prior work. The RQ3 metrics (Whole-File F1, Pairs Differentiated) are direct observed counts; the low F1 values and near-chance pair-differentiation counts are measurements, not outputs of a fitted model. The self-constructed VentiVul is time-wise out-of-distribution and anchored to CVE fix commits, so evaluating models on it is a legitimate external test even though the labels were hand-built by the team. The only noteworthy self-reference is the use of ICVul [24], the authors' own dataset, for which the paper itself supplies the 92% label-accuracy estimate; this affects the benchmark-quality framing but is not load-bearing for the main negative conclusion. The skeptics' point about GPT-4o/GPT-5 'superior ability' being at random-guessing levels is a statistical-support concern, not a circularity: the counts do not reduce to the conclusion by construction.
Assumptions & free parameters
free parameters (3)
- t-SNE hyperparameters (perplexity, learning rate, iterations) =
not reported
- Classification threshold (0.5) =
0.5
- LLM decoding parameters (temperature, top-p, max tokens) =
not reported
assumptions (7)
- domain assumption VentiVul labels are correct: pre-fix functions are vulnerable; post-fix functions and unrelated same-file functions are non-vulnerable.
- domain assumption The 20 Linux kernel CVEs from May 2025 are representative of novel real-world vulnerabilities.
- ad hoc to paper Centroid distance computed in t-SNE 2D space is a meaningful quantitative measure of class separability.
- ad hoc to paper Single-run LLM responses under one hand-written prompt are stable and representative.
- ad hoc to paper Excluding models with F1<20 from ranking in RQ2 does not bias conclusions.
- domain assumption The label accuracy values in Table 1 for Juliet/Devign/BigVul, taken from prior studies [8,11], are reliable.
- domain assumption Function-level and function-pair test settings are the right operationalization of real-world vulnerability detection.
Cite this review
Pith. "Pith review of From Lab to Reality: A Practical Evaluation of Deep Learning Models and LLMs for Vulnerability Detection." pith.science (2026). https://pith.science/paper/BUHWLJJY
@misc{pith2026251210485,
author = {Pith},
title = {Pith review of: From Lab to Reality: A Practical Evaluation of Deep Learning Models and LLMs for Vulnerability Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/BUHWLJJY}},
note = {Machine review of arXiv:2512.10485}
}
read the original abstract
Vulnerability detection methods based on deep learning (DL) have shown strong performance on benchmark datasets, yet their real-world effectiveness remains underexplored. Recent work suggests that graph neural network-based and transformer-based models, including large language models (LLMs), yield promising results when evaluated on curated benchmark datasets. These datasets are typically characterized by similar data distributions and may contain synthetic samples, heuristic labels, or labeling noise. In this study, we systematically evaluate four representative DL models---Devign, ReVeal, LineVul, and VulBERTa---across four representative datasets: Juliet, Devign, BigVul, and ICVul. Each model is trained independently on each dataset, and the graph-based and CodeBERT representations adopted by these models are analyzed using t-SNE and centroid distance to examine vulnerability-related patterns. To assess realistic applicability, we further evaluate trained ReVeal and LineVul models, along with four open-weight LLMs, on VentiVul, our newly constructed temporally separated out-of-distribution (OOD) dataset comprising 200 recent vulnerabilities from Linux and Chromium. Our experiments reveal that current representation methods struggle to distinguish vulnerable from non-vulnerable code and that trained models generalize poorly across datasets with differing distributions and characteristics. When evaluated on VentiVul, performance drops sharply, with most models failing to detect vulnerabilities reliably or distinguish vulnerable functions from their patched counterparts. These results expose a persistent gap between academic benchmarks and real-world deployment, emphasizing the value of our deployment-oriented evaluation framework and the need for more robust code representations, higher-quality datasets, and evaluation methods that account for vulnerability-fixing changes.
Figures
Figures from the paper (2 more)
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.