{"id":"e50be329-def1-4d0d-932b-edf789821c47","arxiv_id":"2412.18260","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"In a five-dataset benchmark, fine-tuned Llama-series LLMs match or beat smaller code models on some datasets, excel on long samples, yet class imbalance remains the dominant factor in detection performance.","lead":"This paper fine-tunes four open-source Llama models with LoRA to detect code vulnerabilities and compares them against five older graph-based and sequence models across five C/C++ datasets. It also measures how class imbalance and code length affect detection performance, and releases a unified codebase and preprocessed datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table III contradicts the 'positive ratio is decisive' claim: low-ratio BigVul gives the highest LLM F1, and the long-sample advantage is confounded with positive ratio.","rationale":"Good-faith reading: this is a useful, reproducible benchmark with appropriate F1/FPR metrics and open code, and it explicitly admits label-quality limitations. The problem is that the headline findings are causal claims drawn from uncontrolled cross-dataset comparisons. The BigVul anomaly is not a subtle regression artifact; it is visible in Table III: the dataset with the second-lowest positive ratio produces the highest LLM F1 scores, and Devign, the balanced dataset, sits in the middle. Similarly, the long-sample analysis compares long and short subsets that differ in positive ratio, and the paper's own controlled length experiment reverses Finding 3. The label-noise issue raised by the reader is real, but even taking all labels as ground truth, the causal conclusions fail. The benchmark retains value as a reproducible resource, so I keep the conditional verdict rather than rejecting; the authors must either run matched-ratio comparisons or substantially weaken the 'decisive' and 'exceptionally well on long samples' claims.","tokens_in":26087,"tokens_out":7574,"duration_ms":66828,"concrete_test":"Retrain one LLM (e.g., Llama-3.1-8B) on Devign and BigVul short-sample subsets downsampled to the same positive ratio (e.g., 10%), using identical preprocessing, LoRA hyperparameters, and train/validation/test splits. If BigVul still outperforms Devign by a large margin, dataset identity or label quality, not positive-sample ratio, drives the reported differences.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table III contradicts the central causal claim. On short samples, BigVul has a 4.46% positive ratio, the second-lowest of the five datasets, yet Llama-2-7B achieves 87.00 F1 and Llama-3.1-8B achieves 81.48 F1, the best LLM results in the table. Devign, at 44.08% positive, yields only 47.86-61.49 F1 across LLMs. If positive-sample ratio were decisive, Devign should dominate. Finding 3 is similarly unsupported: for every dataset, the long-sample subset has a higher positive ratio than the short-sample subset (Table II), so any long-sample advantage in Table III is confounded with class balance; Section VI-D acknowledges this, but its controlled length experiment mixes all five datasets, so length is confounded with dataset composition. That experiment also shows F1 decreasing as length increases, contradicting Finding 3. The paper's own numbers therefore fail to isolate the claimed causal factors; label noise (the reader's concern) is one possible confound among several, but the contradiction is visible from Table III alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical study of fine-tuned LLMs (Llama-2-7B, CodeLlama-7B, Llama-3-8B, Llama-3.1-8B) for code vulnerability detection, comparing them against three graph-based models and two medium-size sequence models on five C/C++ datasets split into short (≤512 tokens) and long (512–1024 tokens) subsets. It reports accuracy, precision, recall, F1, and FPR, and conducts resampling experiments (Section VI-C), length-controlled experiments (Section VI-D), and a LoRA hyperparameter sensitivity study (Section VI-E). The manuscript claims that class imbalance is the decisive factor limiting CVD performance, that medium-size sequence models generally outperform LLMs on short samples, and that LLMs show exceptional potential on long samples; all code and preprocessed data are released at the stated repositories.","tokens_in":26336,"tokens_out":5940,"duration_ms":48348,"significance":"If the findings were fully supported, the paper would be a useful benchmark contribution: it integrates nine models across five datasets in a unified codebase, evaluates recent Llama-3/3.1 models in a setting rarely covered (samples longer than 512 tokens), and appropriately uses F1 and FPR rather than accuracy for imbalanced data. The open-sourced code and datasets are a concrete asset for the community, and the decision to report FPR alongside F1 is a strength. However, the paper's headline findings are partially contradicted by its own Table III and by the controlled length experiment in Section VI-D, so the significance currently rests on the benchmark and reproducibility value rather than on the stated conclusions.","major_comments":[{"comment":"Finding 2 states that medium-size sequence models generally outperform LLMs on short samples, but Table III shows the opposite on BigVul short samples: CodeBERT, UniXcoder, and GraphCodeBERT achieve F1 of 12.60%, 11.76%, and 16.00% respectively, while Llama-2-7B, Llama-3-8B, and Llama-3.1-8B achieve 87.00%, 83.50%, and 81.48%. The finding needs to be qualified, and this quarter of the short-sample evaluation matrix requires an explanation rather than a general statement that LLM recall is significantly lower.","section":"§VI-B, Finding 2 and Table III"},{"comment":"Finding 3 claims exceptional LLM performance on long samples based on Table III, but for every dataset the long-sample subset has a higher vulnerable ratio than the short-sample subset (Table II), so the long/short comparison is confounded with class balance. The paper acknowledges this in Section VI-D, but the replacement experiment mixes all five datasets before splitting by length, which confounds length with dataset composition (datasets differ in difficulty, label quality, and positive ratio). Moreover, Figure 5 shows F1 decreasing as sample length increases, which contradicts Finding 3 rather than supporting it.","section":"§VI-B, Finding 3 and §VI-D"},{"comment":"The claim that 'the proportion of positive samples in the dataset plays a decisive role in the CVD performance of trained models' is not supported by the cross-dataset evidence in Table III: BigVul short samples have the second-lowest positive ratio (4.46%) yet yield the best LLM F1 scores in the paper (81.48–87.00), while Devign short samples with 44.08% positive ratio yield only 47.86–61.49 for LLMs. The within-dataset resampling experiments in Section VI-C support a positive-ratio effect, but they cover only two datasets and two models, so the general 'decisive role' conclusion should be restricted to within-dataset comparisons or replaced by a dataset-level analysis.","section":"§VI-B, concluding paragraph"},{"comment":"The manuscript repeatedly acknowledges that only Devign has expert-curated labels, that Draper's static-analyzer labels are of 'unknown and less investigated' quality with 'low' accuracy, and that several datasets contain noisy or anomalous labels (Section V-B, Section V-C, Section VII). Because all F1/precision/recall numbers are computed against these labels as ground truth, the reported performance differences among models and datasets may partly measure label noise rather than detection capability. The paper should either provide a robustness analysis (e.g., evaluation on a clean or filtered subset, or label-noise sensitivity) or explicitly frame the findings as conditional on dataset label quality; as written, this is a significant threat to the validity of the quantitative claims.","section":"§V-B, §V-C, §VII"}],"minor_comments":[{"comment":"The heading 'Analysis on Datasets with Varying Postive Sample Ratios' contains a typo: 'Postive' should be 'Positive'.","section":"§VI-C heading"},{"comment":"The captions for Figures 3 and 4 use 'Varing' instead of 'Varying'.","section":"Figure captions"},{"comment":"The heading 'Analysis on Analysis on Datasets with Varying Sample Lengths' has a duplicated phrase; it should read 'Analysis on Datasets with Varying Sample Lengths'.","section":"§VI-D heading"},{"comment":"Section V-C states that the Alpaca instruction template is 'illustrated in Figure 1', but the prompt template appears in Figure 2; the cross-reference should be checked.","section":"§V-C and Figure 2"},{"comment":"The sentence 'the model is trained for 5 epoch' should read '5 epochs'.","section":"§VI-A"},{"comment":"In the submitted manuscript, Figures 3–7 appear garbled (the plot regions show repeated token-like path strings instead of visible graphs), making it impossible to verify the qualitative trends discussed in Sections VI-C through VI-E from the text; the figures must be legible in the final version.","section":"Figures 3–7"},{"comment":"References [41] and [50] are duplicate entries for Clark et al. (ELECTRA); one should be removed.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main risk is not the paper's empirical framing but the gap between its stated findings and its own data; I would ask the authors to either re-analyze or substantially soften Findings 2 and 3 and the 'decisive role' conclusion. The garbled figures are likely an artifact of the PDF extraction but should be verified. During revision, the open-source repository should be used to independently confirm the BigVul short-sample numbers, since those results drive the contradiction with the class-imbalance claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a genuine consolidation effort: a unified codebase with four LLMs, three graph models, two medium sequence models, five datasets, and open-sourced code and data. That alone is valuable to the CVD community. The metric choice (F1 and FPR) is right for imbalanced data, and the extension of VulLLM to Llama-3/3.1 and the 512-1024 token region is a legitimate step forward. The controlled resampling and length-stratification experiments are the most interesting part, even if their conclusions are muddled.\n\nThe soft spots are real and load-bearing. Finding 1 says class imbalance is decisive, but Table III shows BigVul with a 4.46% positive ratio giving the highest LLM F1 (87.00 for Llama-2) while Devign at 44.08% gives F1s in the 47-61 range. If positive ratio were the decisive factor, Devign should dominate. Finding 3 says LLMs are exceptionally good on long samples, but every long-sample subset has a higher positive ratio than its short-sample counterpart, and their own controlled length experiment (VI-D) shows F1 decreasing as length increases. They acknowledge the confound in VI-D but still state Finding 3 without qualification. That's an internal contradiction the authors need to resolve.\n\nThe preprocessing is also under-documented: the filtering step is vague (\"we find anomalies\"), and the exact subsampling procedure is not fully specified. There is no uncertainty estimation across seeds; each number appears to come from a single run. And the label-noise issue is raised in the text but then ignored in the analysis, which is especially consequential for Draper and DiverseVul where ground truth is acknowledged to be shaky.\n\nAll that said, the resource is worth engaging with. A revised version that compares models within fixed positive-ratio and length strata, reports variance, and documents preprocessing could become a standard testbed. As is, the findings are not fully supported.\n\nSend it to peer review, but expect major revision. The open codebase and breadth justify referee time.","headline":"A useful open benchmark for fine-tuned LLMs in vulnerability detection, but its headline causal claims about class imbalance and length are not supported by its own tables.","tokens_in":26881,"tokens_out":2812,"would_cite":false,"duration_ms":25073,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned open-source LLMs are effective code-vulnerability detectors, especially on long code samples, with class imbalance the decisive performance factor.","keywords":["code vulnerability detection","large language models","fine-tuning","LoRA","class imbalance","long code samples","empirical benchmark","Llama models"],"falsifier":"Have two independent security experts re-label a random sample of, say, 300 functions from each of ReVeal, Draper, BigVul, and DiverseVul, recompute F1 for the four LLMs on the corrected labels, and check whether the class-imbalance ranking and the long-sample advantage survive; if they flatten or reverse, the headline findings are artifacts of label noise.","tokens_in":25878,"feed_emoji":"🛡️","tokens_out":5324,"duration_ms":45265,"temperature":0.7,"pith_summary":"This paper asks whether fine-tuned open-source large language models can detect code vulnerabilities, and answers with a five-dataset, nine-model benchmark. Its central assertion is that LoRA-fine-tuned Llama-series models are effective detectors, with a particular advantage on long code samples (over 512 tokens) that older graph-based and medium-size sequence models cannot handle. The paper further claims that the proportion of vulnerable samples in the training set plays a decisive role: model F1 collapses on imbalanced datasets and jumps when the positive-sample ratio reaches roughly 30 percent, while code length matters much less. A sympathetic reader would care because the finding reorients practical effort toward dataset balance rather than model size, and it makes long-code vulnerability detection feasible with LLMs.","feed_headline":"Class imbalance is the key brake on LLM vulnerability detection","feed_subtitle":"Five-dataset study shows balanced data, not longer models, drives F1 gains in fine-tuned Llama detectors.","key_machinery":"The argument is carried by a controlled benchmark protocol rather than a single identity. Five C/C++ function-level datasets are each split at 512 tokens into short and long subsets, and nine models are trained in one unified codebase: three graph-based models, two medium-size sequence models, and four Llama-series LLMs fine-tuned with low-rank adaptation (LoRA) using an instruction prompt that asks the model to classify the code. The load-bearing controls are the resampling experiments that fix dataset size at 25,000 and vary the positive-sample ratio in steps from 10% to 50%, and the length experiment that mixes all datasets into eight equal-size length intervals from 0 to 1024 tokens at 20% positive ratio, isolating length from imbalance. F1 is the primary metric, with precision, recall, accuracy, and false positive rate as supporting metrics.","core_discovery":"The paper's central claim is empirical: fine-tuned LLMs are strong code-vulnerability detectors, especially on long samples, and the single most decisive factor in their performance is class imbalance. On long samples the four Llama-series models clearly outperform the two models that can be evaluated there, while on short samples the medium-size sequence models generally achieve higher F1 on imbalanced datasets. Controlled resampling shows F1 and recall rise sharply once vulnerable samples reach about 30% of training data, and a length-controlled experiment shows F1 declines mildly as code grows longer, far less than the imbalance effect. The paper also finds LLMs produce low false positive rates, making them more usable as triage tools.","pith_inferences":["If class imbalance is as decisive as reported, simple fixes such as oversampling vulnerable functions or class-weighted loss on existing imbalanced corpora should produce large F1 gains, a directly testable extension.","Because the paper excludes samples over 1024 tokens, the finding that length has only mild negative impact may not extend to very long functions; testing the same protocol on 1K-4K token code would show whether the length effect bends upward.","The label-noise caveat cuts both ways: on auto-labeled datasets, reported F1 differences may partly measure annotation quality, so future benchmarks should publish noise estimates next to scores to separate model capability from data quality.","A practical pipeline suggested by the data is to pair short-sample specialist models with long-sample LLMs, using code length as a routing signal."],"forward_implications":["LoRA-fine-tuned Llama-series models make long-sample code vulnerability detection practical, since most prior models are limited to 512 tokens and lose on long code.","Balancing a vulnerability dataset to at least 30% positive samples improves F1 and recall sharply for both CodeBERT and Llama-3.1, so dataset curation, not only model choice, drives results.","On short, imbalanced samples the medium-size sequence models still lead, meaning LLMs are not uniformly superior and should be selected per sample-length regime.","Small false positive rates make fine-tuned LLMs more reliable as pre-screening tools for security reviewers.","Larger LoRA rank improves F1 when compute allows, and a scaling factor near twice the rank performs best, giving concrete tuning guidance."],"supporting_citations":[{"why":"Supplies the Devign dataset and graph-based model used as a balanced-data baseline and one of the compared architectures.","marker":"[2]"},{"why":"Supplies CodeBERT, the medium-size sequence model that sets top F1 on short imbalanced samples.","marker":"[3]"},{"why":"Provides the prior fine-tuning recipe for attention modules and defines the long-sample gap this study extends.","marker":"[17]"},{"why":"Supplies LoRA, the low-rank adaptation method used to fine-tune all four LLMs.","marker":"[21]"},{"why":"Supplies DiverseVul, the most imbalanced dataset, plus evidence that auto-labeled datasets contain label noise.","marker":"[22]"},{"why":"Supplies BigVul, the CVE-derived dataset where LLMs achieve their highest F1.","marker":"[23]"},{"why":"Supplies ReVeal, a patch and commit labeled dataset used for the short-sample comparison.","marker":"[24]"},{"why":"Supplies Draper, the static-analyzer-labeled dataset used in the resampling and length experiments.","marker":"[26]"},{"why":"Supplies Llama-2-7B, one of the four fine-tuned LLMs.","marker":"[29]"},{"why":"Supplies Llama-3 and Llama-3.1, the newer LLMs this study is among the first to evaluate for vulnerability detection.","marker":"[35]"}],"fun_headline_variants":["Balanced data beats bigger models for LLM vuln detection","Class imbalance limits LLM code vulnerability detection","LLM detectors: long code shines, but imbalance caps F1","Fine-tuned LLMs excel on long code, data balance is key","Resampling raises LLM vulnerability detection F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark scores models against dataset labels as ground truth, but four of the five datasets were labeled by automatic tools, patches, or commits rather than experts, and the paper itself calls the Draper labels of unknown and less investigated quality, so if many labels are wrong, every reported performance difference partly measures label noise rather than detection ability.","fun_headline_variants_meta":{"raw":{"variants":["Balanced data beats bigger models for LLM vuln detection","Class imbalance limits LLM code vulnerability detection","LLM detectors: long code shines, but imbalance caps F1","Fine-tuned LLMs excel on long code, data balance is key","Resampling raises LLM vulnerability detection F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000168,"raw_usage":{"total_tokens":1234,"prompt_tokens":891,"completion_tokens":343,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":261}},"tokens_in":507,"tokens_out":343,"duration_ms":3700,"temperature":1.0,"reasoning_tokens":261,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:52:18.218253+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent security experts re-label a random sample of, say, 300 functions from each of ReVeal, Draper, BigVul, and DiverseVul, recompute F1 for the four LLMs on the corrected labels, and check whether the class-imbalance ranking and the long-sample advantage survive; if they flatten or reverse, the headline findings are artifacts of label noise.","supporting_citations":[{"cited_title":"Devign: Effective vulnerability identifi- cation by learning comprehensive program semantics via graph neural networks","cited_arxiv_id":null,"evidence_quote":"Supplies the Devign dataset and graph-based model used as a balanced-data baseline and one of the compared architectures."},{"cited_title":"Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen","cited_arxiv_id":null,"evidence_quote":"Supplies LoRA, the low-rank adaptation method used to fine-tune all four LLMs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies DiverseVul, the most imbalanced dataset, plus evidence that auto-labeled datasets contain label noise."},{"cited_title":"A C/C++ code vulnerability dataset with code changes and CVE summaries","cited_arxiv_id":null,"evidence_quote":"Supplies BigVul, the CVE-derived dataset where LLMs achieve their highest F1."},{"cited_title":"Deep learning based vulner- ability detection: Are we there yet?","cited_arxiv_id":null,"evidence_quote":"Supplies ReVeal, a patch and commit labeled dataset used for the short-sample comparison."},{"cited_title":"Draper VDISC Dataset - Vulnerability Detection in Source Code","cited_arxiv_id":null,"evidence_quote":"Supplies Draper, the static-analyzer-labeled dataset used in the resampling and length experiments."}],"review_version":1}