{"id":"230cb122-24dc-452c-8cb3-99de082854bb","arxiv_id":"2512.10485","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Deep learning and LLM-based vulnerability detectors generalize poorly to recent out-of-distribution Linux kernel vulnerabilities, with near-chance performance at distinguishing pre-fix from post-fix code.","lead":"An evaluation of two deep-learning vulnerability detectors and four large language models shows they perform well on standard benchmarks but mostly fail on a new set of 20 recent Linux kernel vulnerabilities. The authors built a small, manually curated dataset (VentiVul) and two test modes to expose this lab-to-real-world gap.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Finding 8's 'superior ability' is based on Pairs-Differentiated counts that are at the random-guessing level; the negative core survives, but the only positive LLM claim is unsupported.","rationale":"The reader's formal weakest assumption was prompt/decoding sensitivity, which is a real limitation for the LLM portion. But the more sharply load-bearing problem is internal: the paper's only positive LLM result, Finding 8, is based on counts that are at the level of random guessing. A single binomial/permutation test would settle whether the 'superior ability' claim survives; on the reported numbers it will not. This reinforces rather than changes the CONDITIONAL verdict: the core negative result on VentiVul is credible and consistent with prior RealVul/PRIMEVUL work, so no rejection is warranted, but Finding 8 and the 'promise in patch-level reasoning' conclusion should be removed or substantially tempered. Agreement is partial because the reader's rationale mentions near-chance counts, but their weakest_assumption emphasizes prompt sensitivity rather than the missing null model for Pairs-Differentiated.","tokens_in":16897,"tokens_out":14474,"duration_ms":155498,"concrete_test":"For GPT-4o and GPT-5, compute the exact null distribution of Pairs-Differentiated under the model that each of the 25 pairs is differentiated with probability 0.25 (independent 50/50 labels for the before-fix and after-fix versions), and report the binomial p-value for the observed counts (5 and 6). As a robustness check, run 10,000 permutations that randomly swap the before/after status within each pair and recompute Pairs-Differentiated. If the p-value is not small (on the reported numbers it will be near 0.5), Finding 8 cannot be sustained; the paper should either drop the 'superior ability' claim or report a metric with a real chance baseline, such as the area under a before/after score ROC curve.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Finding 8 and the conclusion that LLMs 'exhibit superior ability to discern fine-grained semantic changes' rest on Table 7's Pairs-Differentiated counts: GPT-4o scores 5 and GPT-5 scores 6 out of 25. No null model or confidence interval is reported. Under any independent-label null, the expected number of differentiated pairs is 25 * p_before * (1 - p_after), which is maximized at p_before = p_after = 0.5, giving 25 * 0.25 = 6.25. The observed 5-6 is therefore at or below the maximum possible chance expectation. With GPT-5's observed marginals (14 before-fix positives among 25), the independence expectation is close to 6 as well. This is not merely a missing error bar: the paper explicitly interprets these near-chance counts as evidence that GPT-4o/GPT-5 can 'separate vulnerable functions from their fixed counterparts' and uses this to temper the negative conclusion ('LLMs hold promise in understanding patch-level changes'). The central VentiVul negative result does not depend on this finding — Table 6's F1 values (max 8.13 for DL, 4.9 for LLMs) stand separately — but Finding 8 and the 'promise' sentence are unsupported by the paper's own numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper evaluates two widely used vulnerability detection models (ReVeal and LineVul) on four benchmarks (Juliet, Devign, BigVul, ICVul) and, together with four commercial LLMs, on a newly curated out-of-distribution dataset VentiVul of 20 Linux kernel CVEs fixed in May 2025. Three research questions are addressed: (RQ1) whether learned code representations separate vulnerable and non-vulnerable functions, (RQ2) which dataset/model factors affect within- and cross-dataset performance, and (RQ3) whether DL models and LLMs detect recent, previously unseen vulnerabilities in a Whole-File and Function-Pair setting. The main reported findings are that embeddings show limited separation, cross-dataset performance drops sharply, most models have very low F1 on VentiVul, and GPT-4o/GPT-5 differentiate 5-6 of 25 vulnerable/patched pairs, which the paper interprets as superior patch-level reasoning. The paper argues that benchmark results overestimate real-world applicability.","tokens_in":17243,"tokens_out":8452,"duration_ms":86821,"significance":"If the negative results are reliable, the paper adds to a growing body of evidence (Real-Vul, PRIMEVUL) that benchmark-trained vulnerability detectors fail on temporally held-out data. The authors contribute a carefully curated, manually inspected VentiVul dataset, a reproducible framework with code and data release, and a Function-Pair evaluation mode that is more realistic than simple function-level classification. The Whole-File analysis is also useful in exposing all-negative/all-positive prediction biases. The central negative claims are plausible and consistent with prior work. However, the quantitative support is weaker than the narrative: no confidence intervals or significance tests, t-SNE-based distances used as quantitative evidence, a single prompt/run for LLMs, and a positive LLM result that is statistically indistinguishable from chance. These issues are localized but load-bearing for the positive 'promise' claim and for several RQ1/RQ3 statements.","major_comments":[{"comment":"The positive LLM claim is not supported by the reported counts. GPT-4o and GPT-5 differentiate 5 and 6 of 25 pairs. Under any independent-label null, the expected number of differentiated pairs is at most 25 * 0.25 = 6.25, so the observed values are at or below the maximum chance expectation. No confidence interval, null model, or significance test is reported. The conclusion that GPT-4o/GPT-5 exhibit 'superior ability to discern fine-grained semantic changes' and the sentence in the conclusion that 'LLMs hold promise in understanding patch-level changes' therefore need either a proper statistical test (e.g., exact binomial test with a stated null, or permutation test) or should be removed/qualified. The Table 6 F1 numbers still support the negative OOD result; this issue concerns the paper's only positive LLM claim.","section":"Section 4 (RQ3), Table 7, Finding 8"},{"comment":"LLM results are based on a single hand-written prompt per setting with no decoding parameters (temperature, top-p, repetition penalty, max tokens) and one run per model. LLM outputs are highly sensitive to prompt wording and sampling. Tables 6-7 and Findings 7-8 therefore may reflect one arbitrary configuration rather than model ability. Please report exact API/model version and decoding settings, run multiple seeds, and report variance or use a fixed deterministic setting with justification. Without this, the RQ3 comparisons among LLMs and the relative claim about GPT-4o/GPT-5 are not reproducible.","section":"Section 3.2, Table 3"},{"comment":"Centroid distance is computed after t-SNE dimensionality reduction and min-max normalization. t-SNE is stochastic and does not preserve pairwise distances; hyperparameters (perplexity, learning rate, iterations) are not reported. Under these conditions D_centroid is not a meaningful quantitative measure of class separability. This quantity underlies Findings 1-3. Please either replace or augment it with a distance-preserving or embedding-stable measure (e.g., centroids in the original feature space, or silhouette/other intrinsic metrics), report sensitivity to t-SNE hyperparameters, or explicitly label the centroid distances as illustrative rather than quantitative.","section":"Section 3.3.1, Eq. (1)"},{"comment":"The paper's comparative conclusions rest on point estimates from a single training/evaluation run on small data (25 pairs/20 CVEs, with four datasets). For example, the statement that some models 'transfer better' is based on F1 differences that are not accompanied by variance or significance information, and models with F1<20 are excluded from ranking to avoid trivial results. Please add uncertainty quantification (multiple seeds/LLM runs, or at least exact binomial confidence intervals for the categorical VentiVul counts) and justify the exclusion rule. This does not undermine the large drop in Table 6, but it is needed before claiming model or dataset ordering.","section":"Section 4, RQ2/RQ3"}],"minor_comments":[{"comment":"ICVul is the authors' own dataset and its 92% label accuracy is the result of the authors' manual validation of 50 samples. Given that ICVul is later used to argue that clean labels lead to transferable representations (Finding 5), an independent audit or a clearer statement of provenance and potential conflict would strengthen the paper.","section":"Section 2.1, Table 1"},{"comment":"The definitions of 'Vul Detected' (out of 20 CVEs) and 'Vul Funcs Detected' (out of 25 pairs) should be stated in the caption or text, because some CVEs contribute multiple function pairs; otherwise the denominators are confusing.","section":"Table 7"},{"comment":"A 'non-decrease' conflates unchanged scores and increased scores; plotting these separately (or counting increases) would help interpret the Pairs-Differentiated results.","section":"Figure 5"},{"comment":"The Whole-File prompt asks for 'method(s)' and an output format 'like ...'; it is unclear how free-form outputs are parsed into function names and how ambiguous cases are handled. Add the parsing/validation procedure.","section":"Section 3.2/Table 3"},{"comment":"The model name 'GPT-o3-mini' is used inconsistently (also rendered as GPT-o3-mini in the abstract; OpenAI branding is 'o3-mini').","section":"Throughout"},{"comment":"The discussion could better contrast VentiVul with the closely related Real-Vul [6] and PRIMEVUL [13] datasets in terms of temporal separation, manual curation, and the Function-Pair evaluation mode.","section":"Section 7, Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper is stronger as a negative empirical result than as a positive LLM finding. I would advise the editor that Finding 8 and the 'LLMs hold promise' sentence must either be statistically supported or removed/qualified. The authors' connection to ICVul is disclosed but should be handled carefully in the revision. The negative VentiVul result does not depend on the unsupported positive claim, so a revision within the manuscript's scope is feasible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the VentiVul dataset and the Whole-File/Function-Pair evaluation setup are genuinely useful additions, and the main negative result — DL models and LLMs fail to reliably detect recent Linux kernel CVEs — is credible and consistent with RealVul and PRIMEVUL. The paper's one positive claim, Finding 8, is not supported by its own numbers.\n\nWhat's actually new: the dataset is the main thing. Twenty May-2025 Linux CVEs, manually curated, with 25 before/after-fix function pairs and 835 unrelated functions from the same files. That is a real time-wise OOD test. The Function-Pair mode is a nice way to ask whether a detector can distinguish vulnerable from patched code. The cross-dataset results in Tables 4 and 5 show the expected drop and support the paper's central narrative.\n\nThe stress-test note is right. In Table 7, GPT-4o and GPT-5 differentiate 5 and 6 pairs out of 25. Under any reasonable independent-label null, the expected number is around 6.25. Calling that \"superior ability to discern fine-grained semantic changes\" is interpreting noise as signal. This is not a missing confidence interval; it's a claim that contradicts the paper's own table. The conclusion sentence that \"LLMs hold promise in understanding patch-level changes\" should be cut or rewritten.\n\nOther soft spots, in rough order of severity. First, the LLM evaluation uses one hand-written prompt per setting, no decoding parameters reported, one run per model. That is load-bearing for RQ3 and invites the obvious sensitivity objection. Second, t-SNE centroid distances are used as a quantitative separability measure even though the paper itself notes they can vary under different parameter settings; treat those as qualitative. Third, ICVul is the authors' own dataset and their manual label validation is used to argue for its quality — not fatal, but reviewers should ask for independent evidence or a clearer acknowledgment. The 20-CVE sample is small, but that's not a flaw for a stress-test dataset; it does mean confidence intervals would help mostly for the positive claims.\n\nThe negative core does not depend on any of this. Whole-File F1 values max out at 8.13 for DL models and 4.9 for LLMs because models predict nearly everything as non-vulnerable. That is the takeaway readers should remember.\n\nRecommendation: send to peer review. The paper deserves a serious referee, but the revision should be substantial — fix Finding 8, add caveats about LLM settings and a chance baseline, and downgrade the t-SNE distance claims. With those changes it would be a useful contribution.","headline":"The VentiVul dataset and the negative result are worth a referee, but Finding 8's 'superior ability' rests on at-chance counts and should be tempered before publication.","tokens_in":17719,"tokens_out":2235,"would_cite":true,"duration_ms":24688,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that deep-learning and LLM vulnerability detectors, strong on standard benchmarks, fail to distinguish vulnerable code in a small set of newly disclosed Linux kernel vulnerabilities.","keywords":["vulnerability detection","deep learning","large language models","out-of-distribution evaluation","Linux kernel","code representation","function-pair evaluation","benchmark realism"],"falsifier":"Re-run the same VentiVul Function-Pair evaluation with several prompt phrasings and decoding settings (e.g., temperature 0 vs 0.7, top-p variation, repeated sampling) for GPT-4o and GPT-5; if these models then differentiate substantially more than 6 of 25 pairs, or show large variance across runs, the reported failure is an artifact of the specific prompt rather than a stable property of the models.","tokens_in":1460,"feed_emoji":"🐛","tokens_out":1382,"duration_ms":63839,"temperature":0.7,"pith_summary":"Deep-learning and LLM-based vulnerability detectors that post strong F1 scores on standard benchmarks (Juliet, Devign, BigVul, ICVul) stop distinguishing vulnerable from non-vulnerable code when tested on VentiVul, a new dataset of 20 real Linux-kernel vulnerabilities fixed in May 2025. The best Whole-File F1 among eight trained model variants is 8.13 and among four LLMs 4.9; in the Function-Pair setting, most models cannot even assign a higher risk score to a function than to its own patched version. The paper argues this happens because current models learn dataset-specific correlations—noisy labels, class imbalance, and surface patterns—rather than generalizable vulnerability semantics. Representation analysis with t-SNE and centroid distances shows vulnerable and non-vulnerable functions overlap heavily, and cross-dataset tests confirm the drop. If the paper is right, benchmark results overstate real-world readiness, and evaluation practice needs to incorporate temporal out-of-distribution data and patch-level reasoning.","feed_headline":"Fresh Linux bugs expose benchmark-trained vulnerability detectors","feed_subtitle":"Best Whole-File F1 was 8.1 for DL models and 4.9 for LLMs on 20 fresh CVEs.","key_machinery":"VentiVul, a manually curated out-of-distribution test set of 20 Linux-kernel CVEs fixed in May 2025, containing 25 vulnerable/patched function pairs plus 835 unrelated functions from the same files. The paper's central evaluation devices are the Whole-File setting (classify every function in a file) and the Function-Pair setting (compare each before-fix function with its after-fix counterpart; a correct judgment requires the model to score the vulnerable version higher than the patched one). The pairing isolates whether a model captures the semantic change of a fix, which is the minimal ability needed to spot new vulnerabilities. t-SNE and centroid distance serve as the representation-qualit","core_discovery":"On a time-wise out-of-distribution dataset (VentiVul) of 20 recent Linux kernel CVEs, the authors find that both representative DL models (ReVeal and LineVul) and four pretrained LLMs (Claude 3.5 Sonnet, GPT-o3-mini, GPT-4o, and GPT-5) fail to reliably distinguish vulnerable from non-vulnerable code. Under Whole-File evaluation, the best F1 among DL variants is 8.13 and among LLMs 4.9; under Function-Pair evaluation, at most 6 of 25 before/after-fix pairs are correctly differentiated (GPT-5), with most models scoring zero. Representation analysis using t-SNE and centroid distances shows heavy overlap between vulnerable and non-vulnerable functions, and cross-dataset experiments show sharp pe","pith_inferences":["Because VentiVul contains only 20 CVEs and 25 pairs from one kernel, the quantitative F1 values are noisy; the paper's durable contribution is the evaluation protocol (temporal split plus before/after pairing), not the precise numbers.","A natural extension is to use VentiVul-style Function-Pair evaluation as a deployment filter: a detector that cannot tell a function from its own patch is unlikely to catch new bugs, regardless of benchmark F1.","Prompt sensitivity is untested; varying prompts, temperatures, and repeated sampling could move LLM results substantially, so the 'LLMs fail' result should be read as 'under this one zero-shot protocol,' not as a law.","The finding that distances between different vulnerability types sometimes exceed distances between vulnerable and non-vulnerable code suggests that CWE-specific or root-cause-conditioned models may be a more informative target than a single binary classifier."],"forward_implications":["Benchmark scores on Juliet, Devign, BigVul, and ICVul do not transfer to time-wise out-of-distribution code; deployment-oriented evaluation changes the reported ranking of models.","Dataset label quality and balance, not raw volume, dominate cross-dataset generalization; BigVul's noisy labels inflate recall but produce near-zero transferability.","Function-Pair evaluation is a cheap, interpretable way to test whether a detector understands the effect of a security patch; current DL models mostly fail it, suggesting they rely on superficial cues.","LLMs show relatively stronger patch-sensitivity (GPT-4o and GPT-5 differentiate 5-6 of 25 pairs) despite low recall, pointing to patch-level reasoning as a promising direction rather than whole-file binary classification.","Current representation learning—graph-based and token-based—does not yield separable vulnerable/non-vulnerable clusters, so improving representations, not just architectures, is the bottleneck."],"fun_headline_variants":["DL and LLM detectors stumble on real Linux kernel exploits","Fresh Linux CVEs expose benchmark-trained detector blind spots","Best F1 8.1: DL models fail on time-shifted vulnerability set","20 recent kernel CVEs reveal fragility of trained detector models","Model performance craters on new real-world vulnerability dataset"],"cache_read_input_tokens":18944,"weakest_assumption_plain":"The RQ3 LLM conclusions rest on treating a single hand-written prompt with unstated decoding settings, run once per model, as a fair and stable measure of each LLM's vulnerability-detection ability; prompt phrasing and sampling parameters are known to change LLM outputs, and no sensitivity analysis shows they would not.","fun_headline_variants_meta":{"raw":{"variants":["DL and LLM detectors stumble on real Linux kernel exploits","Fresh Linux CVEs expose benchmark-trained detector blind spots","Best F1 8.1: DL models fail on time-shifted vulnerability set","20 recent kernel CVEs reveal fragility of trained detector models","Model performance craters on new real-world vulnerability dataset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1385,"prompt_tokens":851,"completion_tokens":534,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":449}},"tokens_in":595,"tokens_out":534,"duration_ms":6724,"temperature":1.0,"reasoning_tokens":449,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T17:07:11.580307+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same VentiVul Function-Pair evaluation with several prompt phrasings and decoding settings (e.g., temperature 0 vs 0.7, top-p variation, repeated sampling) for GPT-4o and GPT-5; if these models then differentiate substantially more than 6 of 25 pairs, or show large variance across runs, the reported failure is an artifact of the specific prompt rather than a stable property of the models.","supporting_citations":[],"review_version":1}