{"id":"b62402aa-3b00-4b1c-a503-5942adcb1acc","arxiv_id":"2504.16429","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Injecting retrieved vulnerability root-cause and fix knowledge into RACG prompts improves generated-code security rates by 20% on average and by 22% to 32% under poisoning, while preserving functional correctness.","lead":"This paper introduces CodeGuarder, a framework that adds security knowledge retrieved from vulnerability databases to the prompts used by retrieval-augmented code generation systems. The approach raised the rate of secure code produced by four LLMs by 20% on average in standard settings and by 22% to 32% when the knowledge base was poisoned, without hurting functional correctness in the reported tests.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains are likely inflated: k′/k hyperparameters (§7.4) were tuned directly on the CyberSecEval test set, so the headline 20.12% improvement is not an unbiased estimate.","rationale":"The central claim is quantitative: CodeGuarder raises security rates by 20.12% on average in standard RACG and by 31.53% and 21.91% under poisoning. The strongest evidence for this claim is Table 4. But the method's two hyperparameters k′ and k were selected by measuring SR on the very same benchmark and scenario (Table 12, §7.4) before the main results were produced. This is selection-on-the-test-set: the reported numbers reflect a search over a small grid on the test data, not an independent evaluation. The paper reports no validation split, no cross-validation, and no confidence intervals, so the reader cannot assess how much of the 20.12% comes from the method versus the selection. The knowledge-base extraction concern is legitimate—no validation of the LLM-extracted triples is reported—but it is less directly tied to the headline numbers: even noisy extractions could produce the observed end-to-end gains, and the end-to-end results are the actual evidence for the claim. The test-set tuning, by contrast, directly undermines the validity of the measured improvement itself. Because the paper's contribution and all comparisons rest on these effect sizes, this is the most load-bearing spot. The verdict remains conditional: the framework is plausible and the improvements may well be real, but the quantitative claims need re-evaluation on a held-out split (and ideally with released code/data) before they can be accepted at face value.","tokens_in":24993,"tokens_out":9851,"duration_ms":96281,"concrete_test":"Re-run RQ1 with a strict split: randomly hold out 20% of CyberSecEval instances (stratified by language and CWE) as a test set, and use the remaining 80% only to select (k′, k) from the same grid as Table 12 for each LLM. Then evaluate CodeGuarder on the held-out test set with the selected hyperparameters and compare SR to the baseline. If the average improvement on the held-out set is materially smaller than the reported 20.12% (e.g., more than 5 percentage points lower), or if the selected (k′, k) differs from (2, 5), the headline effect size is inflated by test-set tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is not the unvalidated knowledge-base extraction (though that is real), but the selection of the injection hyperparameters k′ and k directly on the evaluation benchmark. Section 7.4 evaluates (k′, k) combinations using DS-V3 and CodeLlama-13B 'on the standard RACG scenario' and then sets the default to (k′ = 2, k = 5). This standard RACG scenario is exactly the evaluation reported in RQ1 (Table 4), and the same hyperparameters are used in all later scenarios and tables. No held-out validation split or cross-validation is mentioned anywhere in the paper. Because the headline average improvement of 20.12% in Table 4 is computed on the same data that was used to choose k′ and k, the reported effect size is an optimistically biased estimate of CodeGuarder's performance on unseen queries. The bias may be modest, since the authors chose a robust configuration rather than the single best cell in Table 12, but the evaluation is still not an unbiased estimate, and no confidence intervals or repeated-run statistics are provided to bound the selection bias. The knowledge-base extraction gap raised by the reader is secondary: even noisy extractions could yield the observed end-to-end gains, so the more direct threat to the central quantitative claim is the selection-on-the-test-set.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CodeGuarder, a framework for hardening retrieval-augmented code generation (RACG) against security vulnerabilities and knowledge-base poisoning. CodeGuarder builds a security knowledge base offline by prompting DeepSeek-V3 to extract functionality, root cause, and fixing-pattern triples from CVE descriptions, CWE labels, and function-level diffs from the ReposVul dataset. At query time, a retriever decomposes the user query into sub-tasks, retrieves security knowledge for each sub-task using dense embeddings, re-ranks sub-tasks by the estimated vulnerability risk of their retrieved knowledge (using empirically derived CWE frequencies from prior work), filters to the top-k sub-tasks, and injects the curated security knowledge into the generation prompt. The authors evaluate CodeGuarder on CyberSecEval across four LLMs (GPT-4o, DeepSeek-V3, CodeLlama-13B, DeepSeek-Coder-V2-16B) and four languages (C, C++, Java, Python) under a standard RACG scenario and two poisoning scenarios, reporting average relative security-rate improvements of 20.12%, 31.53%, and 21.91%, respectively, with no loss in functional similarity and with pass@1/pass@5 improvements on MBPP and HumanEval. The paper also reports generalization results in a non-retrieval setting and in a cross-language setting where the target language is absent from the security knowledge base.","tokens_in":25274,"tokens_out":5472,"duration_ms":52385,"significance":"If the central claims are robust, CodeGuarder would be a timely and useful contribution: it is the first framework explicitly designed to harden RACG systems against security vulnerabilities, including knowledge-base poisoning. The empirical study is broad in coverage (four LLMs, four languages, three scenarios, plus two generalization settings), uses a fixed, standard insecure-code detector for all comparisons, and includes an ablation study that isolates the contributions of query decomposition and knowledge re-ranking/filtering. The MBPP/HumanEval results are a credible check that the security intervention does not destroy functional correctness. The main caveats are that the headline improvements are computed after selecting the injection hyperparameters on the evaluation benchmark itself, and that the automatically constructed security knowledge base is not validated. These issues affect the unbiasedness and external validity of the headline numbers, but the underlying approach is defensible and the paper is worth revising rather than rejecting.","major_comments":[{"comment":"The default hyperparameters (k' = 2, k = 5) are selected by evaluating (k', k) combinations on the standard RACG scenario, which is exactly the scenario and benchmark used to report the headline 20.12% average improvement in Table 4 and subsequently reused in all other scenarios and tables. No held-out validation split, cross-validation, or independent tuning set is mentioned anywhere in the manuscript. As a result, the reported improvements are optimistically biased estimates of CodeGuarder's effect on unseen queries; the bias may be modest because the authors chose a robust configuration rather than the single best cell in Table 12, but its magnitude is not quantified. Please add a validation split or nested cross-validation for hyperparameter selection and report confidence intervals or repeated-run statistics for the main security-rate results to bound the selection bias.","section":"Section 7.4, Table 12; Section 6.2, Table 4"},{"comment":"The security knowledge base is constructed automatically by prompting DeepSeek-V3 to extract functionality, root-cause, and fixing-pattern triples from CVE descriptions, CWE labels, and function-level diffs, but no validation of these extractions is reported. If a substantial fraction of the extracted triples are inaccurate, irrelevant, or too generic, the retrieval and injection pipeline may not transfer to other benchmarks or languages even though the end-to-end CyberSecEval results are positive. Please provide a manual sample evaluation, agreement metrics against the CVE diff content, or an automatic cross-check of the extracted triples, and report how many entries were empty or unusable after extraction.","section":"Section 4.1.2, Prompt 1"},{"comment":"The non-retrieval comparison is not matched. CodeGuarder is given access to the security knowledge base, query decomposition, and injected security knowledge, while the baselines (SVEN, SafeCoder, CoSec) receive only the query and do not have access to any external knowledge. The conclusion that CodeGuarder 'outperforms' these approaches by 9.80% in security rate is therefore not a like-for-like comparison of security-hardening techniques under identical information. Please either restrict the baselines to methods with access to the same external knowledge, or clearly frame the result as a system-level comparison and temper the 'outperform' wording accordingly.","section":"Section 6.4.1, Table 7"}],"minor_comments":[{"comment":"The reported improvements (e.g., 20.12%, 31.53%, 21.91%) are relative percentage increases, not absolute percentage-point increases. Please state this explicitly in the abstract and table captions to avoid misinterpretation.","section":"Abstract and Table 4"},{"comment":"The decomposition output is written as Q_d = [q1, q1, ..., qn]; the second element should be q2.","section":"Section 4.2.1"},{"comment":"There is a typo: 'we built he poisoning retriever' should be 'we built the poisoning retriever'.","section":"Section 5.4"},{"comment":"The phrase 'when when there is no off-the-shelf knowledge base' contains a duplicated 'when'.","section":"Section 6.4"},{"comment":"'DS-VS' appears in the text and should be 'DS-V3'.","section":"Section 6.3.2"},{"comment":"The five-run DS-V3 variability check reports a maximum deviation of 0.38% in SR under the standard scenario; please clarify whether the same variability bound holds for the Sim metric and for the other LLMs.","section":"Section 7.5"},{"comment":"The legend for scenarios I and II appears only in the table footnote; making the scenario explicit in the table header would improve readability.","section":"Table 8"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is closely related to the authors' prior work [24], and the poisoning scenario design, poisoning retriever, and attack parameters (m = 5, p = 10) are reused from that study. This is not disqualifying, but the novelty of CodeGuarder relative to [24] should be stated more explicitly in the introduction and related work. No artifact or code link is provided, which limits reproducibility. The ACM template placeholders ('Conference'17') also need to be updated for the submission venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is the first RACG-specific security hardening framework I have seen, and it appears to do what it says: injecting filtered security knowledge into RACG prompts raises the insecure-code-detector pass rate across four LLMs and four languages, in clean and poisoned settings, without hurting functional scores. The direction is credible. The exact size of the headline effect is not.\n\nWhat is genuinely good: the knowledge base is built from the fixed side of ReposVul, not from CyberSecEval, which is a real design choice; the vulnerable code base for poisoning comes from CyberSecEval, so the defense and the attack are not drawn from the same pool. The ablation shows query decomposition does most of the work, with reranking/filtering adding a little. The DS-V3 five-run variance check (0.38% max deviation) is a nice touch, and MBPP/HumanEval results address the obvious functional-correctness worry. The cross-language section is useful, even if Rust gains are modest.\n\nSoft spots, in order of seriousness.\n\nOne: hyperparameters k' and k were chosen by running DS-V3 and CodeLlama on the standard RACG scenario in Table 12, and that same scenario is the one used for the headline 20.12% in Table 4. There is no held-out split or nested validation. They picked (2,5) rather than the best cell for either model, so the bias is probably modest, but the headline is still an in-sample estimate. Every table that uses those hyperparameters inherits the selection.\n\nTwo: no uncertainty on the main numbers. One variance check for one model is not enough. Temperature 0 does not guarantee determinism for Mixture-of-Experts models, and they say so themselves. I want confidence intervals or repeated-run summaries for at least DS-V3 and GPT-4o on the main tables.\n\nThree: the knowledge extraction from DeepSeek-V3 in Section 4.1.2 is never validated. Even if end-to-end results could be robust to noisy extractions, a sample audit would help.\n\nFour: no code or data release. For an empirical framework paper, that is a barrier.\n\nMinor: RQ3 compares against non-retrieval baselines; a RACG baseline with a generic security prompt would be a more natural control, and the improvement over SafeCoder is suggestive but not apples-to-apples. The citation pattern is fine; the prior threat-study self-citation is legitimate, and the CWE-frequency weights are sourced from an external large study.\n\nBottom line: this deserves a serious referee. The central mechanism—putting security knowledge into the retrieval context—is plausible and backed by consistent, if not unbiased, evidence. I would ask for held-out hyperparameter selection, uncertainty quantification, and data/code release, then accept conditionally.","headline":"CodeGuarder is a credible first RACG-specific security-hardening framework, but the headline gains are selected on the test set, so treat the effect sizes as in-sample estimates pending held-out tuning.","tokens_in":25803,"tokens_out":3361,"would_cite":true,"duration_ms":31994,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CodeGuarder claims that injecting retrieved security knowledge into prompts raises secure-code rates in retrieval-augmented code generation by 20.12% on average, and by 31.53% and 21.91% under two poisoning scenarios.","keywords":["retrieval-augmented code generation","software security","code generation","knowledge base poisoning","security knowledge injection","vulnerability mitigation","large language models"],"falsifier":"Have expert programmers independently judge a random sample of 100 extracted knowledge-base entries for correctness and specificity of the root cause and fixing pattern; if most are judged inaccurate or generic, the central mechanism loses its support. Alternatively, run CodeGuarder on a fresh query set whose CWEs are absent from the knowledge base and compare against a retrieval-free baseline; the claim predicts the improvement should shrink toward zero as relevant entries vanish.","tokens_in":24800,"feed_emoji":"🛡️","tokens_out":6755,"duration_ms":60250,"temperature":0.7,"pith_summary":"The paper argues that retrieval-augmented code generation (RACG) systems can be hardened by changing what they retrieve: instead of fetching only functional code examples, the retriever should also fetch security knowledge describing a vulnerability's root cause and its fix. It proposes CodeGuarder, which builds a security knowledge base from real-world CVEs, decomposes each code query into sub-tasks, retrieves relevant knowledge for each sub-task, and re-ranks that knowledge by how often the associated vulnerability type appears in LLM-generated code. The authors report relative improvements in the security rate of 20.12% in standard RACG, 31.53% in a targeted poisoning scenario, and 21.91% in an intent-agnostic poisoning scenario, while functional correctness, measured by CodeBLEU and by pass@1 and pass@5 on MBPP and HumanEval, stays flat or improves slightly. If correct, this would give RACG systems a prompt-level defense against both ordinary vulnerabilities and deliberately poisoned knowledge bases, without fine-tuning or post-generation scanning.","feed_headline":"Injected security knowledge lifts secure-code rates 20-32%","feed_subtitle":"CodeGuarder retrieves root causes and fixes from real vulnerabilities and feeds them to LLMs, with no loss of functionality.","key_machinery":"The load-bearing object is the security knowledge base entry: a triplet of functionality, root cause, and fixing pattern extracted from a CVE record and its vulnerable and fixed function-level diff. The functionality description is what makes retrieval work, because it closes the gap between the natural language of a CVE and the natural language of a code-generation query; the root cause and fixing pattern give the LLM both a warning and a concrete replacement. The second mechanism is query decomposition, which splits a user query into sub-tasks so that knowledge can be matched to the exact risky operation, such as memory allocation rather than the whole function. The third mechanism is a re-ranking and filtering step that uses a table of how often each vulnerability type appears in LLM-generated code to prioritize knowledge for the sub-tasks most likely to produce an insecure construct.","core_discovery":"The central claim is that the security of LLM-generated code in retrieval-augmented settings is determined less by the generative model than by the content of the retrieved context. CodeGuarder shifts the RACG retrieval target from 'code similar to the query' to 'security knowledge relevant to the query's sub-tasks,' operationalized as three extracted items per historical vulnerability: a functionality description, a root-cause explanation with vulnerable code, and a fixing pattern with secure code. Given a query, a backend LLM decomposes it into fine-grained sub-tasks; an embedding retriever fetches the top $k'$ knowledge entries per sub-task; and a re-ranking step weights those entries by the empirical prevalence of their vulnerability types in LLM-generated code, keeping only the top $k$ risky sub-tasks. The paper reports that this pipeline raises security rates across four LLMs and four languages in standard RACG, in two poisoning scenarios, and in non-retrieval generation where it outperforms the security-tuning baselines Sven, SafeCoder, and CoSec on average, with no loss of functional correctness.","pith_inferences":["One testable extension is to separate the effect of security knowledge from the effect of sub-task decomposition across a wider range of model sizes; the paper's ablation suggests decomposition carries most of the gain, but the two are entangled because disabling decomposition also disables re-ranking.","Because the re-ranking weights come from a static table of vulnerability prevalence in LLM-generated code, a natural refinement is to make those weights query-specific or model-specific, or to learn them from detector feedback, which could improve weak spots such as CWE-79 where the paper reports near-zero prevention.","Since the knowledge base is built from public CVE records, the same pipeline could be run continuously: newly disclosed vulnerabilities would yield new entries available to any RACG system without retraining, making the defense a service rather than a one-time artifact."],"forward_implications":["Verified, prompt-level knowledge injection becomes a defense that works even with closed models where fine-tuning is not an option.","RACG systems can keep their functional retrieval stream and add a security retrieval stream, with no weight changes and no separate post-generation analysis step.","Even when the knowledge base lacks target-language entries, cross-language security principles still lift security rates, so the approach degrades gracefully as knowledge coverage shrinks.","The two poisoning experiments suggest that injecting security knowledge can counteract poisoned retrieval examples rather than amplifying them, which is the core threat modeled in the paper.","The non-retrieval comparison indicates that the same injected security knowledge can compete with dedicated security fine-tuning and co-decoding baselines in ordinary code generation."],"supporting_citations":[{"why":"Prior study establishing the threat model: a single poisoned example can make 48% of generated code vulnerable, motivating the two poisoning scenarios.","marker":"[24]"},{"why":"CyberSecEval benchmark and its score definitions, used as the security evaluation testbed across 50 CWE types.","marker":"[43]"},{"why":"CyberSecEval's Insecure Code Detector, the static-analysis-based tool used to compute the security rate metric.","marker":"[3]"},{"why":"ReposVul dataset, the source of real-world vulnerable and fixed function-level pairs used to construct the security knowledge base.","marker":"[45]"},{"why":"Large-scale analysis of vulnerability-type distribution in LLM-generated code, supplying the weights used in re-ranking and filtering.","marker":"[41]"},{"why":"DeepSeek-V3 technical report; the model is used as the backend for automatic knowledge extraction and query decomposition.","marker":"[25]"},{"why":"SafeCoder instruction-tuning baseline, compared against in the non-retrieval generalization evaluation.","marker":"[14]"},{"why":"Sven prefix-tuning baseline, compared against in the non-retrieval generalization evaluation.","marker":"[13]"},{"why":"CoSec supervised co-decoding baseline, compared against in the non-retrieval generalization evaluation.","marker":"[21]"}],"fun_headline_variants":["Retrieved security fixes lift LLM code safety 20-32%","CodeGuarder: Security knowledge in retrieval bolsters code safety","Security-aware RAG cuts vulnerable code by up to 32%","Injecting vulnerability fixes into prompts improves code safety","From code examples to security lessons: Boosting LLM code safety"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The automatic knowledge-base construction assumes that DeepSeek-V3, given a CVE description, a CWE label, and a function-level diff, produces accurate functionality, root cause, and fixing-pattern triples; the paper reports no validation of these extractions, so if they are often wrong or too generic the retrieval pipeline could fail outside the tested benchmark.","fun_headline_variants_meta":{"raw":{"variants":["Retrieved security fixes lift LLM code safety 20-32%","CodeGuarder: Security knowledge in retrieval bolsters code safety","Security-aware RAG cuts vulnerable code by up to 32%","Injecting vulnerability fixes into prompts improves code safety","From code examples to security lessons: Boosting LLM code safety"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000331,"raw_usage":{"total_tokens":1894,"prompt_tokens":1050,"completion_tokens":844,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":757}},"tokens_in":666,"tokens_out":844,"duration_ms":7408,"temperature":1.0,"reasoning_tokens":757,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:03:22.059154+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have expert programmers independently judge a random sample of 100 extracted knowledge-base entries for correctness and specificity of the root cause and fixing pattern; if most are judged inaccurate or generic, the central mechanism loses its support. Alternatively, run CodeGuarder on a fresh query set whose CWEs are absent from the knowledge base and compare against a retrieval-free baseline; the claim predicts the improvement should shrink toward zero as relevant entries vanish.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ReposVul dataset, the source of real-world vulnerable and fixed function-level pairs used to construct the security knowledge base."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Large-scale analysis of vulnerability-type distribution in LLM-generated code, supplying the weights used in re-ranking and filtering."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CoSec supervised co-decoding baseline, compared against in the non-retrieval generalization evaluation."}],"review_version":1}