{"id":"228395f5-030f-4c6b-a2a3-70c119634b3c","arxiv_id":"2412.15905","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Across six programming languages, fine-tuned language models detect JavaScript vulnerabilities with F1 scores near 70%, much higher than the low scores reported for C/C++, while code complexity metrics show no consistent relationship with detection performance.","lead":"This paper fine-tunes four language models to find security flaws in JavaScript, Java, Python, PHP, Go, and C/C++ code, and reports that JavaScript is much easier to handle than C/C++. It also finds that code complexity does not predict how well the models perform.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The cross-language F1 ranking likely reflects CVEfixes' paired before/after construction and dataset-size/class-balance confounds rather than language-dependent detectability; the paper never shows non-vulnerable samples are not the fixed counterparts of the vulnerable ones.","rationale":"I read the paper as an empirical benchmark whose main contribution is a language-spanning vulnerability detection comparison. The strongest claim is that language models are considerably more effective for JavaScript than for C/C++, which is supported by Table 3's same-dataset C/C++@CVEFixes rows, not only by cross-paper numbers. That same-dataset comparison is real evidence, so I do not think the claim collapses immediately. However, the comparison is only valid if the language subsets are comparable in how labels were constructed and in size and balance. The most load-bearing unresolved question is whether the non-vulnerable samples are simply the fixed counterparts of the vulnerable samples, because CVEfixes is derived from fixing commits. If so, the task can degrade into detecting which side of a diff a function comes from, and the language ranking becomes an artifact of how easily each language's patches can be recognized. The reader's weakest assumption identified exactly this issue, and I agree with it. I also agree with the related confounds of dataset size and class balance, and with the Table 5 copy error, which separately undermines the complexity section. I do not see internal inconsistency in the experimental setup beyond these data-construction and reporting issues, and the authors do provide a public dataset and scripts, which is creditworthy. A conditional verdict is appropriate: the results are publishable as a benchmark only after the pairing is disclosed or removed and the comparison is repeated under controlled subsampling. My read therefore does not change the reader's verdict; it reinforces it.","tokens_in":15324,"tokens_out":4588,"duration_ms":46207,"concrete_test":"Download the released dataset from the GitHub repository; for each language subset, join vulnerable and non-vulnerable samples on CVE ID, commit hash, and file path to compute what fraction of non-vulnerable samples are the exact post-fix counterparts of a vulnerable sample. Then retrain CodeBERT on balanced, size-matched subsets with paired counterparts removed from both train and test, and evaluate on independently sourced non-vulnerable functions. If the JavaScript F1 advantage over C/C++ persists under this matched design, the concern is resolved; if it shrinks or vanishes, the reported ranking is an artifact of dataset construction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that JavaScript is 'considerably better' than C/C++ requires the F1 gap to measure language-dependent detectability under comparable data conditions. That condition is least secure at the dataset-construction step (Section 4.1, Table 1). CVEfixes is built from vulnerability-fixing commits; by construction, the 'non-vulnerable' entries are typically the post-fix versions of the same functions that appear as 'vulnerable' entries. The paper never states that it removed or de-duplicated this pairing. If paired before/after versions survive preprocessing, the binary classifier can succeed by recognizing which side of a diff it is looking at, a much easier and possibly language-specific artifact signal, rather than by learning vulnerability semantics. That would invalidate both the JS-versus-C/C++ ranking and the 'practical detection' framing. Even setting pairing aside, the comparison is confounded by extreme size and balance differences: JavaScript is capped at the first 100,000 entries while Go has 6,283 and C/C++ has 20,060; class balance also differs, with JavaScript roughly 47% vulnerable versus PHP at 17%. No subsampling, propensity weighting, or matched-size control is reported, so larger and more balanced JavaScript data alone could explain its higher F1. A separate issue, Table 5 lists identical complexity values for Java and C/C++, indicating a copy error that undermines the secondary complexity analysis.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a cross-language empirical study of language-model-based vulnerability detection using the CVEfixes dataset. The authors construct language-specific subsets for JavaScript, PHP, Java, Python, Go, and C/C++, fine-tune four LMs (CodeT5, CodeBERT, UnixCoder, DeepSeek-Coder), and report accuracy, F1, precision, recall, and FPR. Their central finding is that JavaScript shows the best detection performance, with 'considerably better and more practical detection capabilities compared to C/C++', while PHP and Go show lower F1 despite high accuracy. As a secondary analysis, they compute Pearson correlations between six code-complexity metrics and per-language F1 scores across all four models, concluding that the correlations are weak and statistically insignificant. The paper also validates some models on external datasets and releases the curated data, scripts, and results.","tokens_in":15610,"tokens_out":4548,"duration_ms":38865,"significance":"If the main finding were robust, this would fill a real gap: prior vulnerability-detection benchmarks for LMs concentrate heavily on C/C++, and a controlled cross-language comparison would be valuable for practitioners and benchmark designers. The authors are transparent about raw numbers, provide model-level metrics, and open-source their dataset and scripts, which supports reproducibility. However, the current evidence does not yet sustain the headline claim because the cross-language comparison is threatened by uncontrolled dataset construction differences and by a likely copy error in the complexity analysis. The significance is therefore contingent on the comparability concerns being resolved; with additional controls, this could become a useful reference study.","major_comments":[{"comment":"The manuscript never states whether the non-vulnerable samples are the fixed counterparts of the vulnerable functions drawn from the same CVEfixes vulnerability-fixing commits. Because CVEfixes is constructed from fixing commits, the non-vulnerable entries are typically post-fix versions of the same functions that appear as vulnerable entries; if such paired before/after versions survive preprocessing, the binary classifier can succeed by recognizing which side of a diff it is looking at rather than by learning vulnerability semantics, and the JS-versus-C/C++ ranking would be invalidated. Please clarify whether such pairs were identified and separated across the train/test split, and report a control experiment with paired samples removed or with train and test sets constrained to disjoint commits.","section":"Section 4.1 (Data Preprocessing), Table 1"},{"comment":"The cross-language F1 comparison is confounded by dataset size and class balance: JavaScript has 100,000 samples (46,802 vulnerable) and C/C++ has 20,060 (8,299 vulnerable), while PHP has only 4,758 vulnerable samples out of 28,257 total. The manuscript states that these sizes reflect the NVD distribution, but that does not remove the confound for the claim that language-dependent detectability differs. Without matched-size subsamples, propensity weighting, or a direct demonstration that the larger and more balanced JavaScript data alone does not explain the gap, the conclusion that JavaScript yields 'considerably better and more practical detection' is not supported by the reported experiments.","section":"Table 1 and Section 5.2"},{"comment":"The Java and C/C++ rows of Table 5 are identical for all six complexity metrics (Token Length 274.79, Halstead Volume 879.30, Halstead Difficulty 10.81, Halstead Effort 16462.39, Cyclomatic Complexity 3.27, NLOC 20.50), which indicates a copy error. Since Section 5.3 uses these per-language means to compute the Pearson correlations reported in Table 6, the complexity analysis and its conclusion of weak, statistically insignificant correlation are not reliable until the table is corrected and the correlations are recomputed.","section":"Table 5 and Section 5.3"}],"minor_comments":[{"comment":"The description 'divide the data into training and testing sets based on the commit timestamps using stratified sampling' is ambiguous: stratified sampling preserves class proportions, while a temporal split orders by commit date; please clarify how the two procedures are combined and whether the stratification is applied within time intervals.","section":"Section 4.1"},{"comment":"The cap of 100,000 JavaScript entries is reported only in a footnote without justification; please explain why the first 100,000 entries were chosen and discuss the potential impact on representativeness and comparability with other language subsets.","section":"Table 1 footnote"},{"comment":"The sentence 'The major issue with the detection results reported from the PrimeVul [11] and DiverseVul [7] is is the low F1 scores' contains a duplicated 'is' and a subject-verb agreement error; it should be rephrased.","section":"Section 2.2"},{"comment":"The sentence beginning 'Furthermore, the vulnerability types diversity, each with its own unique characteristics...' is grammatically incomplete, and the citation [2] is to the authors' own work only; please rephrase and provide broader citations for the diversity of vulnerability types.","section":"Section 2.2"},{"comment":"The sentence 'This indicates a better ability to detect vulnerabilities with LLM for JavaScript compared to C/C++' overstates the evidence given the dataset-size and pairing confounds discussed above; please condition this conclusion on the comparability analysis or temper the wording.","section":"Section 5.2"},{"comment":"Figure 1 is referenced in the text but does not appear in the provided manuscript; please ensure the figure is included and legible, since it is the sole support for the claim about language distribution in CVEfixes.","section":"Section 4.1, Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant gap and the authors provide useful artifacts, including a cleaned dataset and experimental scripts. My main concern is that the headline claim about JavaScript versus C/C++ depends on data-construction choices that are not disclosed (paired fixing-commit samples) and on uncontrolled dataset-size and class-balance differences. These are fixable with additional experiments, so the paper does not appear to contain a fundamental methodological flaw beyond comparability. The complexity analysis also needs a corrected Table 5 before it can support the secondary conclusion. With these revisions, the paper could be suitable for a venue that accepts empirical benchmark studies."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful part of this paper is the dataset work: they take CVEFixes, split by language, clean duplicates, and release the subsets and scripts. That is a real contribution, and the per-language F1 tables give tool builders a quick reference. The cross-dataset checks for Java, Python, and PHP are also a sensible addition. If I want a rough sense of how fine-tuned CodeBERT or DeepSeek-Coder behaves on JavaScript or Go, this is now a reasonable place to look.\n\nThe problem is the central claim. The paper says JavaScript detection is 'considerably better' than C/C++, but the comparison is confounded in at least two ways that the authors do not address. First, CVEfixes entries come from fixing commits, so each vulnerable function has a paired fixed version. The non-vulnerable samples are likely those fixed versions. The paper only removes exact duplicates, not the before/after pairs. If the pairing survives, the model can succeed by recognizing which side of a diff it sees, which is an artifact, not vulnerability semantics. That alone would weaken the language ranking. Second, the JavaScript subset is capped at the first 100,000 entries while C/C++ has 20,060, and class balance differs (47% vs 41%). No subsampling or matched-size control is reported. Larger, more balanced data can explain higher F1. The authors note the size differences but do not treat them as confounds.\n\nTable 5 also has an apparent copy error: Java and C/C++ have identical complexity values across all six metrics. That undermines the complexity analysis, though it was underpowered anyway with six languages.\n\nNone of this is fatal to the dataset release or the basic per-language results. But the headline conclusion about language-dependent detectability is not supported by the current experiments. A serious revision would need subsampled comparisons, a clear statement about pairing, and a corrected Table 5.\n\nThe paper is worth sending to peer review—the artifact and benchmark are useful—but it should come back with major revision, not be accepted as-is.\n\nBest.","headline":"Useful per-language benchmark, but the headline JavaScript-vs-C/C++ comparison is not controlled enough to support the paper's conclusion.","tokens_in":16086,"tokens_out":2647,"would_cite":false,"duration_ms":22396,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that language models detect vulnerabilities far better in JavaScript and Java than in C/C++, and that code complexity does not explain the gap.","keywords":["vulnerability detection","language models","JavaScript","C/C++","code complexity","CVEFixes","fine-tuning","cross-language"],"falsifier":"Rebuild the JavaScript test set so that every non-vulnerable function is taken from unrelated code instead of from the same fix history as the vulnerable function, then re-fine-tune and re-measure F1; if the JavaScript advantage over C/C++ disappears or drops sharply, the result was an artifact of distinguishing buggy from fixed code rather than detecting vulnerabilities. Alternatively, cap every language at the same training-sample count (for example, the Go count of 6,283) and check whether the ranking survives.","tokens_in":1454,"feed_emoji":"🛡️","tokens_out":3097,"duration_ms":65525,"temperature":0.7,"pith_summary":"This paper asks whether large language models can detect security vulnerabilities in languages other than C/C++, which has been the near-exclusive focus of prior work. By cleaning the CVEFixes database into language-specific subsets for JavaScript, Java, Python, PHP, Go and C/C++, and fine-tuning four language models under identical conditions, the authors find that detection performance varies strongly by language. JavaScript comes out best, with F1 scores near 70 percent, considerably ahead of C/C++, and both JavaScript and Java outperform the C/C++ results reported in earlier studies. The authors also find that standard code-complexity metrics show only weak and statistically insignificant correlations with F1 scores, so complexity does not explain the differences.","feed_headline":"JavaScript beats C/C++ in LLM vulnerability spotting","feed_subtitle":"Across six languages, fine-tuned models score highest on JavaScript; code complexity barely predicts success.","key_machinery":"A controlled cross-language comparison built on a single dataset source: the paper filters CVEFixes into language-specific sets, applies identical preprocessing (duplicate removal, stratified temporal train/test split), fine-tunes the same four models on each language under the same hyperparameters, and then measures F1 and false-positive rate. The language-dependent results are validated on independent datasets for Java, Python and PHP. A second analysis computes Pearson correlations between six code-complexity metrics (token length, Halstead volume, difficulty and effort, cyclomatic complexity, and lines of code) and each model's F1 score across the six languages to test whether complexity explains performance.","core_discovery":"On a cleaned subset of CVEFixes covering six programming languages and four fine-tuned language models (CodeT5, CodeBERT, UniXcoder and DeepSeek-Coder), the paper's central finding is that vulnerability detection performance is strongly language-dependent. JavaScript achieves the best F1 scores (about 70 percent with CodeBERT and UniXcoder), followed by Java, while C/C++ lags with F1 scores in the low 30s on the same data — and the non-C/C++ results are considerably better than the C/C++ figures reported by the DiverseVul and PrimeVul studies. The paper further claims that code complexity, measured by token length, Halstead metrics, cyclomatic complexity and lines of code, correlates only weakly with F1 scores, and that none of the correlations are statistically significant. This is presented as evidence that language characteristics beyond raw complexity drive how well LMs can learn vulnerabilities, and that JavaScript and Java are more practical targets for LM-based detection than the traditionally studied C/C++.","pith_inferences":["A natural explanation the authors do not fully explore is that dataset size drives part of the JavaScript advantage: with 100,000 samples, JavaScript has roughly 12 times the data of Go and Java, so a matched-size experiment would separate language effects from data abundance.","If the non-vulnerable JavaScript samples are the fixed counterparts of the vulnerable functions, the model may learn to spot the diff itself; one testable fix is to build a JavaScript evaluation set where non-vulnerable code is unrelated to the vulnerable function.","The weak complexity correlation could be an artifact of the narrow complexity range in this dataset; applying the same correlation analysis to a more varied, multi-language corpus would tell whether complexity ever matters.","The authors' claim that JS and Java are more practical targets than C/C++ would be strengthened if independent JavaScript and Go datasets existed; building such datasets is a concrete next step for the community."],"forward_implications":["Practitioners should target JavaScript and Java codebases first if they plan to deploy LM-based vulnerability detection today, since those languages give the best detection quality.","The poor C/C++ results across CVEFixes, PrimeVul, DiverseVul and Gen2Spec indicate the problem is not a single dataset artifact but something about the language or its vulnerability patterns.","Because complexity metrics do not predict F1, model developers should not use complexity as a proxy for expected detection difficulty.","The mixed external validation results for Python and PHP mean that language-level conclusions should not be transferred blindly to new data sources.","The strongest JavaScript F1 scores come from the 125M-parameter encoder models, not the larger 1.3B Decoder model, so architecture matters more than raw size for this task."],"supporting_citations":[{"why":"Supplies the full cross-language vulnerability and fix data that the paper filters into language-specific subsets.","marker":"[4]"},{"why":"Provides the DiverseVul C/C++ dataset and results that serve as the primary baseline for the language comparison.","marker":"[7]"},{"why":"Provides the PrimeVul C/C++ dataset and results that show poor real-world LM performance, which the paper contrasts with its JavaScript and Java outcomes.","marker":"[11]"},{"why":"Defines CodeBERT, one of the four fine-tuned models whose per-language F1 scores drive the central comparison.","marker":"[12]"},{"why":"Defines UniXcoder, the model that achieves the best JavaScript F1 score and consistently strong results across languages.","marker":"[13]"},{"why":"Defines DeepSeek-Coder, the largest evaluated model, whose mixed performance supports the claim that model size is not decisive.","marker":"[15]"},{"why":"Defines CodeT5, the smallest evaluated model, which underperforms on most languages and anchors the architecture-vs-size discussion.","marker":"[42]"}],"fun_headline_variants":["JavaScript beats C/C++ in LLM vulnerability detection","LLMs spot more vulnerabilities in JavaScript than in C/C++","Code complexity doesn't explain LLM vulnerability detection gaps","JS and Java lead C/C++ in LLM vulnerability spotting","Why LLMs detect more vulnerabilities in JavaScript than C"],"cache_read_input_tokens":18304,"weakest_assumption_plain":"The central comparison assumes that the language-specific datasets differ only in language, not in how vulnerable and non-vulnerable samples were chosen, so a higher F1 score reflects easier detectability rather than easier data.","fun_headline_variants_meta":{"raw":{"variants":["JavaScript beats C/C++ in LLM vulnerability detection","LLMs spot more vulnerabilities in JavaScript than in C/C++","Code complexity doesn't explain LLM vulnerability detection gaps","JS and Java lead C/C++ in LLM vulnerability spotting","Why LLMs detect more vulnerabilities in JavaScript than C"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000778,"raw_usage":{"total_tokens":3434,"prompt_tokens":931,"completion_tokens":2503,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":2423}},"tokens_in":547,"tokens_out":2503,"duration_ms":15403,"temperature":1.0,"reasoning_tokens":2423,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:58:46.883858+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rebuild the JavaScript test set so that every non-vulnerable function is taken from unrelated code instead of from the same fix history as the vulnerable function, then re-fine-tune and re-measure F1; if the JavaScript advantage over C/C++ disappears or drops sharply, the result was an artifact of distinguishing buggy from fixed code rather than detecting vulnerabilities. Alternatively, cap every language at the same training-sample count (for example, the Go count of 6,283) and check whether the ranking survives.","supporting_citations":[{"cited_title":"In: Muresan, S., Nakov, P., Villavi- cencio, A","cited_arxiv_id":null,"evidence_quote":"Defines UniXcoder, the model that achieves the best JavaScript F1 score and consistently strong results across languages."}],"review_version":1}