{"id":"9c91ea84-5219-47bd-9d23-9e0e1f4ec5f0","arxiv_id":"2412.19088","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"An under-specified evaluation of five open LLMs on vulnerability and deprecated-code detection reports moderate average scores (60.4 to 67.0) but lacks baselines, data, and reproducibility.","lead":"The paper tests five open-license language models on two software security tasks: detecting coding errors and deprecated code. It concludes the models are not yet reliable enough to replace rule-based security scanners, and sees their best use as an augmented review layer.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TruthfulQA is not a code-vulnerability benchmark, so the reported scores cannot support the claim that LLMs detect vulnerabilities or deprecated code in software supply chains.","rationale":"The reader's weakest assumption and my load-bearing concern are the same: the TruthfulQA benchmark is invalid as a measure of vulnerability detection. The paper's central claim — that LLMs can fortify software supply chain security and potentially replace static/dynamic scanners — depends entirely on the experimental evaluation. That evaluation uses a benchmark designed for truthfulness on open-domain questions, not for identifying security flaws in code. No evidence is given that the two tasks are related, and Table I shows no code-specific metric. Even if all scores were reproduced, they would not tell us whether the models can detect vulnerabilities or deprecated code. This invalidates the empirical foundation of the paper. The absence of a baseline comparison to static scanners, the lack of error bars, and the unreleased dataset all compound the problem, but the benchmark mismatch is the load-bearing issue because it severs the link between the data and the conclusion. The paper's own limitation section acknowledges memory and unfamiliar-pattern issues but does not acknowledge this measurement validity problem. Therefore, the reject verdict is appropriate and should remain unchanged.","tokens_in":9321,"tokens_out":3178,"duration_ms":32791,"concrete_test":"Reconstruct the evaluation by obtaining the exact prompts and dataset from the authors, or independently replicate it: run the same five models on the original TruthfulQA benchmark and separately on a held-out set of 100 known vulnerable or deprecated code snippets from a public dataset such as DiverseVul or CVE-fix commits, using a binary detection task. If detection accuracy on the code snippets is near chance or uncorrelated with the TruthfulQA scores, the evaluation in Section III-A does not measure code-security detection. If the original prompts were TruthfulQA's general-knowledge questions, the paper's central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-A states that the authors use the TruthfulQA benchmark to assess how well LLMs detect security issues, but TruthfulQA is a general-knowledge benchmark for measuring whether models repeat common misconceptions; it contains no source-code vulnerability or deprecated-code detection tasks. The paper provides no mapping or adaptation that would make TruthfulQA scores informative for code-security detection, and no prompts, dataset samples, or labeled ground truth are shown. Table I reports only a 'TQA' score and an 'Avg' score with no definition of what 'Avg' measures and no per-language detection results, despite the discussion claiming language-specific findings. Section III-B describes a dataset of 500 vulnerabilities per language but gives no provenance, labels, or examples, and no code or data are released. Because the dependent variable — vulnerability/deprecated-code detection — is never actually measured, the empirical results cannot support the central claim that LLMs can augment or replace static and dynamic security scanners. This is an internal-validity failure, not a disagreement with community consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates five open-weights LLMs (GPT-2, Gemma 7B, Mistral-7B, OpenLLaMA, Phi2) for detecting code vulnerabilities and deprecated patterns in software supply chains. The methodology applies the TruthfulQA benchmark and a dataset described only as containing 500 vulnerabilities per programming language. Table I reports TQA and 'Avg' scores per model; Section IV discusses per-language performance differences, and the conclusion claims LLMs could augment, not replace, traditional scanners.","tokens_in":9415,"tokens_out":3490,"duration_ms":32894,"significance":"If the results were valid, a systematic comparison of open LLMs for vulnerability detection would be a useful input to software supply chain tooling decisions. The paper also surveys relevant prior work on LLMs for vulnerability repair and detection, which is the strongest part of the manuscript. However, the current evaluation does not establish the central claim: the benchmark does not measure the target task, the dataset is unverifiable, no baseline scanners are compared, and the reported table contradicts the discussion. The paper ships no code, data, or machine-checked artifacts, so the empirical contribution is not reproducible.","major_comments":[{"comment":"The TruthfulQA benchmark is a general-knowledge truthfulness test that contains no source-code vulnerability or deprecated-code detection tasks. The paper provides no adaptation or mapping from TruthfulQA to code security, so the reported TQA scores cannot support the claim that these models detect security issues in software supply chains. This is an internal-validity failure that undermines the empirical basis of the central claim.","section":"Section III-A, Table I"},{"comment":"The dataset is described in one paragraph with no provenance, labeling criteria, ground truth, example instances, or release mechanism. The claim of 500 vulnerabilities or deprecated anti-patterns per programming language cannot be checked, reproduced, or interpreted, and no quality-control process is described. Without this information, the 'Avg' scores in Table I are uninterpretable.","section":"Section III-B"},{"comment":"The discussion states that OpenLLaMA 'indissolubly stands out with the highest overall performance,' but Table I shows GPT-2 with the highest Avg (67.02) and OpenLLaMA at 66.61. The table does not define what 'Avg' aggregates, and no per-language scores are provided despite Section IV making detailed per-language claims (e.g., Mistral weak in Ruby and Perl, GPT-2 weak in JavaScript). The reported evidence does not support the stated comparative findings.","section":"Section IV and Table I"},{"comment":"The paper's hypothesis is that LLMs can 'supplant traditional static and dynamic security scanners,' but the experiments include no comparison against any static or dynamic scanner, and the conclusion retreats to LLMs as 'a valuable augmentation' of existing monitoring services. The central research question about replacement is therefore never tested against the baseline it claims to replace.","section":"Introduction and Conclusion"}],"minor_comments":[{"comment":"The abstract and several passages contain grammatical errors (e.g., 'human errors always looming', 'With autonomous system issues are not just stop') that impede readability.","section":"Abstract and passim"},{"comment":"The paper mentions repeating each prompt 10 times for reliability but reports no variance, confidence intervals, or statistical tests, making it impossible to assess the stability of the reported scores.","section":"Section III-A"},{"comment":"The subsection 'Taxonomy of Supply Chain Security' is listed but contains no text, leaving the structuring of the background incomplete.","section":"Section II-A"},{"comment":"Figure 2 is cited with a caption but is not described in the text, and the figure itself is not included in the full text provided; readers cannot understand what the workflow actually involved.","section":"Figure 2"},{"comment":"Reference [8] is cited in support of using LLMs to fix bugs, but the cited work ('Enterprise Security: The Manager's Defense Guide') does not appear relevant to that claim; please verify and replace the citation.","section":"References"},{"comment":"The prose uses idiosyncratic and imprecise wording ('indissolubly', 'scotches', 'perpetuates') that obscures the intended technical meaning and should be revised for clarity.","section":"Section IV"}],"recommendation":"reject","confidential_remarks":"The manuscript is far below the standard for a refereed venue. The evaluation does not measure the claimed task, the dataset is not described, the results contradict the discussion, and there is no comparison to any existing scanner. A proper study would require a new evaluation using a real vulnerability-detection benchmark, baseline comparisons, and statistical analysis, which is beyond the scope of a revision. The topic is relevant but the current contribution is not sound; I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the note. I read the paper and I'm with the reader's verdict: reject, and I'd desk reject it. The core problem is in Section III-A: they say they apply the TruthfulQA benchmark to 'assess how well the selected LLMs can detect security issues,' but TruthfulQA is a multiple-choice benchmark about general knowledge and common misconceptions. It contains no source code, no vulnerabilities, no deprecated patterns. So Table I's TQA and Avg scores cannot be interpreted as vulnerability detection rates. The paper provides no adaptation, no prompts, no sample items, and no definition of 'Avg.' That's a load-bearing failure, not a quibble.\n\nThe dataset description in Section III-B is one paragraph: '500 vulnerabilities or deprecated anti-patterns' per language, no provenance, no labels, no examples, no release. The discussion makes specific claims about per-language performance (e.g., OpenLLaMA excelling in C and Objective-C, Mistral weak in Ruby and Perl) but no per-language results are shown anywhere. There's also an empty subsection (II-A) and a writing style that reads like a thesaurus experiment.\n\nTo be fair, the question is worth asking, and the conclusion is appropriately cautious—LLMs as augmentation, not replacement. The references are relevant, including Pearce et al., DiverseVul, and Ullah et al. So the authors know the literature. But the empirical work doesn't land. No comparison to any static or dynamic scanner, no statistical analysis, no error bars, no code/data. The findings replicate what prior work already showed; there's no new result here.\n\nThe limitations section mentions context length and prompt overreliance, but it never acknowledges the benchmark validity issue, which is the actual problem.\n\nWho is this for? Maybe someone wanting a survey of how not to evaluate LLMs, but not someone seeking reliable evidence. It doesn't deserve referee time because the central measurement is invalid; redoing the experiments would be needed from scratch.\n\nBottom line: desk reject. If the authors want to make a contribution, they should pick a real vulnerability detection benchmark, document the dataset, compare against existing tools, and release everything.","headline":"TruthfulQA is the wrong instrument for this task, so the paper's scores don't measure vulnerability detection; the rest is a plausible but unsupported argument for LLMs as a supplementary layer.","tokens_in":10010,"tokens_out":3224,"would_cite":false,"duration_ms":29644,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-weight LLMs can catch some code flaws, but the paper argues they should augment, not replace, rule-based security scanners in the software supply chain.","keywords":["large language models","software supply chain security","vulnerability detection","deprecated code","static and dynamic analysis","TruthfulQA benchmark","open-weight LLMs"],"falsifier":"Run the same five models on a corpus of real vulnerable and clean code with ground-truth labels, using a conventional static analyzer as a baseline and human review as ground truth; if LLM detection accuracy on novel vulnerabilities is at or below chance, or if the models add no correct findings beyond what the analyzer already finds, the paper's claim that LLMs can fortify supply chain security as an additional review layer would be undermined.","tokens_in":9090,"feed_emoji":"🛡️","tokens_out":8203,"duration_ms":67741,"temperature":0.7,"pith_summary":"This paper asks whether free, open-weight large language models can take over the job that rule-based static and dynamic security scanners do in the software supply chain. To find out, the authors ran five models—GPT-2, Gemma 7B, Mistral-7B, OpenLLaMA, and Phi2—on sets of source code containing vulnerabilities or deprecated patterns, scoring the models' answers with the TruthfulQA benchmark. The results are mixed: larger models generally detect more, but every model struggles with long inputs and with patterns outside its training data. The paper concludes that LLMs are not a drop-in replacement for deterministic scanners, and should instead be used as an additional review layer, kept fresh with security databases and continual updates.","feed_headline":"LLMs catch code flaws but not ready to replace scanners","feed_subtitle":"Five open models score code flaws but stay short of scanner precision; use them as an added review layer.","key_machinery":"The evaluation pipeline is the TruthfulQA benchmark, a question-answering test that checks whether models repeat common falsehoods, here repurposed to score model answers about vulnerable or deprecated code. Each prompt is run ten times, answers are checked against correct answers, and humans review truthfulness on complex questions. Model scores, reported as a TruthfulQA average and an average detection per task, are then interpreted in terms of architectural features—parameter count, training-data mixture, attention mechanisms, and context length—so the comparison across languages does the argument's work.","core_discovery":"The paper's central claim is that LLMs, applied proactively and paired with extensive security databases and regular updates, can fortify software supply chain processes against emerging threats, but only as augmentation—not as a replacement for conventional scanners. The authors' experiments yield per-model and per-language scores: OpenLLaMA leads overall, particularly in C and Objective-C; Gemma is the most consistent generalist, strongest in C++ and Python; GPT-2 is steady in Python and C++ but weaker in JavaScript and Ruby; Mistral-7B sits in the middle with limited language-specific retention; and Phi2 trails on most languages. These differences are attributed to model size, training-data mixture, attention mechanisms, and context length. The paper takes the observed failures—context-length limits, memory complexity, and difficulty with new or unfamiliar data patterns—as evidence that LLMs in their current form cannot fully replace deterministic scanners.","pith_inferences":["The paper reports no side-by-side baseline against a conventional static analyzer; a direct comparison on the same corpus would quantify how much of the reported detection is genuinely additive rather than redundant.","Because TruthfulQA measures general truthfulness rather than code-security competence, the reported scores may partly reflect general language ability; a vulnerability-labeled code benchmark could rank the same models differently.","The proposed remedy of continual retraining presumes that model weights can keep pace with novel attack patterns; a more testable extension would let the LLM query a live vulnerability database at inference time and measure whether that closes the gap."],"forward_implications":["A practical pipeline can pair LLMs with existing scanners so that the models act as a second reviewer for code errors and deprecated patterns, while deterministic tools keep their compliance-friendly, reproducible output.","Because larger models with broader training data scored higher overall, teams choosing an LLM for supply chain security should expect language-specific performance differences, not a single universal best choice.","Context-length limits mean large files must be split or pre-filtered before an LLM sees them, so integration is not a drop-in wrapper around a repository.","Since the tested models cannot retrieve up-to-date threat information, any deployed LLM layer would need regular fine-tuning or enrichment from a current security database to stay useful."],"supporting_citations":[{"why":"Supplies the TruthfulQA benchmark used to score every model's answers, making it the paper's central evaluation instrument.","marker":"[25]"},{"why":"Provides the vulnerability database that sources the 500 vulnerable or deprecated samples per programming language.","marker":"[26]"},{"why":"Defines OpenLLaMA, the model family whose top overall scores anchor the claim that training-data mixture and architecture drive detection.","marker":"[27]–[29]"},{"why":"Defines Gemma 7B, whose consistent performance across languages supports the generalist-comparison result.","marker":"[30], [31]"},{"why":"Defines Mistral-7B, whose mid-range results illustrate how smaller parameter counts limit language-specific knowledge.","marker":"[32]"},{"why":"Defines GPT-2, the smaller baseline whose steady Python and C++ scores are contrasted with weaker JavaScript and Ruby results.","marker":"[33]"},{"why":"Defines Phi2, whose low scores across most languages demonstrate the ceiling for small models on knowledge-heavy tasks.","marker":"[34]"},{"why":"Supplies the prior evidence that LLMs are not yet reliable at identifying and reasoning about security vulnerabilities, framing the paper's augmentation conclusion.","marker":"[20]"}],"fun_headline_variants":["Open LLMs bolster code review but can't oust scanners","LLM code checks: helpful add-on, not scanner killer","Five open models flag flaws, fall short of full scans","OpenLLaMA leads code security, but no scanner replacement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the TruthfulQA benchmark, a general-knowledge truthfulness test, actually measures a model's ability to detect code vulnerabilities and deprecated code; if that mapping is wrong, the reported scores do not support the paper's conclusions.","fun_headline_variants_meta":{"raw":{"variants":["Open LLMs bolster code review but can't oust scanners","LLM code checks: helpful add-on, not scanner killer","Five open models flag flaws, fall short of full scans","OpenLLaMA leads code security, but no scanner replacement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1203,"prompt_tokens":861,"completion_tokens":342,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":477,"completion_tokens_details":{"reasoning_tokens":272}},"tokens_in":477,"tokens_out":342,"duration_ms":141216,"temperature":1.0,"reasoning_tokens":272,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:57:14.724744+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same five models on a corpus of real vulnerable and clean code with ground-truth labels, using a conventional static analyzer as a baseline and human review as ground truth; if LLM detection accuracy on novel vulnerabilities is at or below chance, or if the models add no correct findings beyond what the analyzer already finds, the paper's claim that LLMs can fortify supply chain security as an additional review layer would be undermined.","supporting_citations":[{"cited_title":"National Vulnerability Database,","cited_arxiv_id":null,"evidence_quote":"Provides the vulnerability database that sources the 500 vulnerable or deprecated samples per programming language."}],"review_version":1}