Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

Give LLMs a Security Course: Securing Retrieval-Augmented Code Generation via Knowledge Injection

T0 review · 3 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CodeGuarder claims that injecting retrieved security knowledge into prompts raises secure-code rates in retrieval-augmented code generation by 20.12% on average, and by 31.53% and 21.91% under two poisoning scenarios.

desk verdict CodeGuarder is a credible first RACG-specific security-hardening framework, but the headline gains are selected on the test set, so treat the effect sizes as in-sample estimates pending held-out tuning. read the letter →

arxiv 2504.16429 v1 pith:E6SYRR3T submitted 2025-04-23 cs.CR cs.SE

Bo Lin , Shangwen Wang , Yihao Qin , Liqian Chen , Xiaoguang Mao This is my paper · ORCID
classification cs.CRcs.SE
keywords retrieval-augmentedcodegenerationsoftwaresecurityknowledgebasepoisoninginjectionvulnerabilitymitigationlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that retrieval-augmented code generation (RACG) systems can be hardened by changing what they retrieve: instead of fetching only functional code examples, the retriever should also fetch security knowledge describing a vulnerability's root cause and its fix. It proposes CodeGuarder, which builds a security knowledge base from real-world CVEs, decomposes each code query into sub-tasks, retrieves relevant knowledge for each sub-task, and re-ranks that knowledge by how often the associated vulnerability type appears in LLM-generated code. The authors report relative improvements in the security rate of 20.12% in standard RACG, 31.53% in a targeted poisoning scenario, and 21.91% in an intent-agnostic poisoning scenario, while functional correctness, measured by CodeBLEU and by pass@1 and pass@5 on MBPP and HumanEval, stays flat or improves slightly. If correct, this would give RACG systems a prompt-level defense against both ordinary vulnerabilities and deliberately poisoned knowledge bases, without fine-tuning or post-generation scanning.

What carries the argument

The load-bearing object is the security knowledge base entry: a triplet of functionality, root cause, and fixing pattern extracted from a CVE record and its vulnerable and fixed function-level diff. The functionality description is what makes retrieval work, because it closes the gap between the natural language of a CVE and the natural language of a code-generation query; the root cause and fixing pattern give the LLM both a warning and a concrete replacement. The second mechanism is query decomposition, which splits a user query into sub-tasks so that knowledge can be matched to the exact risky operation, such as memory allocation rather than the whole function. The third mechanism is a re-ranking and filtering step that uses a table of how often each vulnerability type appears in LLM-generated code to prioritize knowledge for the sub-tasks most likely to produce an insecure construct.

What would settle it

Have expert programmers independently judge a random sample of 100 extracted knowledge-base entries for correctness and specificity of the root cause and fixing pattern; if most are judged inaccurate or generic, the central mechanism loses its support. Alternatively, run CodeGuarder on a fresh query set whose CWEs are absent from the knowledge base and compare against a retrieval-free baseline; the claim predicts the improvement should shrink toward zero as relevant entries vanish.

Watch

Extended reading notes

Core claim

The central claim is that the security of LLM-generated code in retrieval-augmented settings is determined less by the generative model than by the content of the retrieved context. CodeGuarder shifts the RACG retrieval target from 'code similar to the query' to 'security knowledge relevant to the query's sub-tasks,' operationalized as three extracted items per historical vulnerability: a functionality description, a root-cause explanation with vulnerable code, and a fixing pattern with secure code. Given a query, a backend LLM decomposes it into fine-grained sub-tasks; an embedding retriever fetches the top $k'$ knowledge entries per sub-task; and a re-ranking step weights those entries by the empirical prevalence of their vulnerability types in LLM-generated code, keeping only the top $k$ risky sub-tasks. The paper reports that this pipeline raises security rates across four LLMs and four languages in standard RACG, in two poisoning scenarios, and in non-retrieval generation where it outperforms the security-tuning baselines Sven, SafeCoder, and CoSec on average, with no loss of functional correctness.

Load-bearing premise

The automatic knowledge-base construction assumes that DeepSeek-V3, given a CVE description, a CWE label, and a function-level diff, produces accurate functionality, root cause, and fixing-pattern triples; the paper reports no validation of these extractions, so if they are often wrong or too generic the retrieval pipeline could fail outside the tested benchmark.

Editorial extensions

If this is right

  • Verified, prompt-level knowledge injection becomes a defense that works even with closed models where fine-tuning is not an option.
  • RACG systems can keep their functional retrieval stream and add a security retrieval stream, with no weight changes and no separate post-generation analysis step.
  • Even when the knowledge base lacks target-language entries, cross-language security principles still lift security rates, so the approach degrades gracefully as knowledge coverage shrinks.
  • The two poisoning experiments suggest that injecting security knowledge can counteract poisoned retrieval examples rather than amplifying them, which is the core threat modeled in the paper.
  • The non-retrieval comparison indicates that the same injected security knowledge can compete with dedicated security fine-tuning and co-decoding baselines in ordinary code generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension is to separate the effect of security knowledge from the effect of sub-task decomposition across a wider range of model sizes; the paper's ablation suggests decomposition carries most of the gain, but the two are entangled because disabling decomposition also disables re-ranking.
  • Because the re-ranking weights come from a static table of vulnerability prevalence in LLM-generated code, a natural refinement is to make those weights query-specific or model-specific, or to learn them from detector feedback, which could improve weak spots such as CWE-79 where the paper reports near-zero prevention.
  • Since the knowledge base is built from public CVE records, the same pipeline could be run continuously: newly disclosed vulnerabilities would yield new entries available to any RACG system without retraining, making the defense a service rather than a one-time artifact.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes CodeGuarder, a framework for hardening retrieval-augmented code generation (RACG) against security vulnerabilities and knowledge-base poisoning. CodeGuarder builds a security knowledge base offline by prompting DeepSeek-V3 to extract functionality, root cause, and fixing-pattern triples from CVE descriptions, CWE labels, and function-level diffs from the ReposVul dataset. At query time, a retriever decomposes the user query into sub-tasks, retrieves security knowledge for each sub-task using dense embeddings, re-ranks sub-tasks by the estimated vulnerability risk of their retrieved knowledge (using empirically derived CWE frequencies from prior work), filters to the top-k sub-tasks, and injects the curated security knowledge into the generation prompt. The authors evaluate CodeGuarder on CyberSecEval across four LLMs (GPT-4o, DeepSeek-V3, CodeLlama-13B, DeepSeek-Coder-V2-16B) and four languages (C, C++, Java, Python) under a standard RACG scenario and two poisoning scenarios, reporting average relative security-rate improvements of 20.12%, 31.53%, and 21.91%, respectively, with no loss in functional similarity and with pass@1/pass@5 improvements on MBPP and HumanEval. The paper also reports generalization results in a non-retrieval setting and in a cross-language setting where the target language is absent from the security knowledge base.

Significance. If the central claims are robust, CodeGuarder would be a timely and useful contribution: it is the first framework explicitly designed to harden RACG systems against security vulnerabilities, including knowledge-base poisoning. The empirical study is broad in coverage (four LLMs, four languages, three scenarios, plus two generalization settings), uses a fixed, standard insecure-code detector for all comparisons, and includes an ablation study that isolates the contributions of query decomposition and knowledge re-ranking/filtering. The MBPP/HumanEval results are a credible check that the security intervention does not destroy functional correctness. The main caveats are that the headline improvements are computed after selecting the injection hyperparameters on the evaluation benchmark itself, and that the automatically constructed security knowledge base is not validated. These issues affect the unbiasedness and external validity of the headline numbers, but the underlying approach is defensible and the paper is worth revising rather than rejecting.

major comments (3)
  1. [Section 7.4, Table 12; Section 6.2, Table 4] The default hyperparameters (k' = 2, k = 5) are selected by evaluating (k', k) combinations on the standard RACG scenario, which is exactly the scenario and benchmark used to report the headline 20.12% average improvement in Table 4 and subsequently reused in all other scenarios and tables. No held-out validation split, cross-validation, or independent tuning set is mentioned anywhere in the manuscript. As a result, the reported improvements are optimistically biased estimates of CodeGuarder's effect on unseen queries; the bias may be modest because the authors chose a robust configuration rather than the single best cell in Table 12, but its magnitude is not quantified. Please add a validation split or nested cross-validation for hyperparameter selection and report confidence intervals or repeated-run statistics for the main security-rate results to bound the selection bias.
  2. [Section 4.1.2, Prompt 1] The security knowledge base is constructed automatically by prompting DeepSeek-V3 to extract functionality, root-cause, and fixing-pattern triples from CVE descriptions, CWE labels, and function-level diffs, but no validation of these extractions is reported. If a substantial fraction of the extracted triples are inaccurate, irrelevant, or too generic, the retrieval and injection pipeline may not transfer to other benchmarks or languages even though the end-to-end CyberSecEval results are positive. Please provide a manual sample evaluation, agreement metrics against the CVE diff content, or an automatic cross-check of the extracted triples, and report how many entries were empty or unusable after extraction.
  3. [Section 6.4.1, Table 7] The non-retrieval comparison is not matched. CodeGuarder is given access to the security knowledge base, query decomposition, and injected security knowledge, while the baselines (SVEN, SafeCoder, CoSec) receive only the query and do not have access to any external knowledge. The conclusion that CodeGuarder 'outperforms' these approaches by 9.80% in security rate is therefore not a like-for-like comparison of security-hardening techniques under identical information. Please either restrict the baselines to methods with access to the same external knowledge, or clearly frame the result as a system-level comparison and temper the 'outperform' wording accordingly.
minor comments (7)
  1. [Abstract and Table 4] The reported improvements (e.g., 20.12%, 31.53%, 21.91%) are relative percentage increases, not absolute percentage-point increases. Please state this explicitly in the abstract and table captions to avoid misinterpretation.
  2. [Section 4.2.1] The decomposition output is written as Q_d = [q1, q1, ..., qn]; the second element should be q2.
  3. [Section 5.4] There is a typo: 'we built he poisoning retriever' should be 'we built the poisoning retriever'.
  4. [Section 6.4] The phrase 'when when there is no off-the-shelf knowledge base' contains a duplicated 'when'.
  5. [Section 6.3.2] 'DS-VS' appears in the text and should be 'DS-V3'.
  6. [Section 7.5] The five-run DS-V3 variability check reports a maximum deviation of 0.38% in SR under the standard scenario; please clarify whether the same variability bound holds for the Sim metric and for the other LLMs.
  7. [Table 8] The legend for scenarios I and II appears only in the table footnote; making the scenario explicit in the table header would improve readability.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity: the (k′,k) injection hyperparameters are selected on the same standard RACG scenario used to report the headline 20.12% improvement, making that headline an in-sample tuning result rather than an unbiased prediction.

  1. fitted input called prediction [§7.4, Table 12; default set in §4.2.3; headline result in §6.2, Table 4]
    "We empirically tuned these hyperparameters by evaluating various(𝑘′,𝑘) combinations using DS-V3 (a larger model) and CodeLlama-13B (a smaller model) on the standard RACG scenario. ... we selected 𝑘′ = 2 and𝑘 = 5 as the default hyperparameters for CodeGuarder in our main evaluations (§6)."

    RQ1/Table 4 reports the headline average improvement of 20.12% in the standard RACG scenario, and §7.4 selects the default (k′,k) by measuring SR in exactly that same scenario. For CodeLlama, the chosen cell (k′=2, k=5) in Table 12 is SR=69.74, which is the same value reported as CodeLlama's average SR with CodeGuarder in Table 4 (69.74). The reported effect is therefore not an out-of-sample estimate of CodeGuarder's benefit; it is the value of the tuning objective itself, or shares its evaluation instances. Choosing a robust configuration rather than the single best cell reduces but does not eliminate this dependence, and the same tuned configuration is then reused in all subsequent scenarios and tables.

full rationale

This is an empirical systems paper rather than a derivation chain, so there is no equation-level self-definitional circularity. The security knowledge base is built from ReposVul, the functional code base is built from ReposVul fixed code, the vulnerability-frequency weights come from an external prior study, and the poisoned code comes from CyberSecEval; none of these is defined in terms of the reported security rates. The one concrete circularity is selection-on-test: §7.4 tunes k′ and k on the standard RACG scenario, and §6.2 reports the improvement in that same scenario as the headline result, with the CodeLlama average in Table 4 being numerically identical to the selected cell in Table 12. This makes the central RQ1 effect size partially fitted rather than predicted. The self-citation to prior work [24] for the poisoning setup is not load-bearing because the paper independently measures poisoning damage in its own RQ2 experiments. Overall, the method has real independent content, but the headline quantitative claim is partially contaminated by in-sample hyperparameter selection, warranting a moderate circularity score of 4 rather than a higher score.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities; it reuses known LLMs, embeddings, datasets, and vulnerability taxonomies. The only hand-set quantities are retrieval hyperparameters and a default weight for unlisted CWEs.

free parameters (3)
  • k' (knowledge entries retrieved per sub-task) = 2
    Selected from Table 12 on the standard RACG scenario with DS-V3 and CodeLlama, then used for all main evaluations in Section 6.
  • k (number of top-ranked sub-tasks retained) = 5
    Selected from Table 12 on the standard scenario to balance prompt size and performance; used for all main evaluations.
  • default CWE weight for unlisted vulnerability types = 0.01
    Assigned in Section 4.2.3 to knowledge entries whose CWE is absent from the Table 1 vulnerability distribution, so they are not entirely filtered out.
assumptions (4)
  • domain assumption Security knowledge can be automatically extracted from CVE descriptions and fix diffs using DeepSeek-V3.
    The entire knowledge-base construction in Section 4.1.2 uses LLM extraction without validation of precision or recall.
  • domain assumption Cosine similarity over jina-embeddings-v3 representations of sub-task descriptions and functionality descriptions retrieves relevant security knowledge.
    Retrieval in Section 4.2.2 assumes embedding similarity aligns with security relevance; no retrieval quality evaluation is reported.
  • domain assumption CyberSecEval's Insecure Code Detector is a valid measurement of generated-code security.
    Section 7.5 acknowledges the detector covers only 50 CWE types and may not capture the full security posture of generated code.
  • domain assumption The vulnerability-type distribution from Tihanyi et al. generalizes to the four evaluated LLMs and CyberSecEval tasks.
    Re-ranking weights in Section 4.2.3 are drawn from a prior study of 13 LLMs; no sensitivity analysis for this distribution is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Give LLMs a Security Course: Securing Retrieval-Augmented Code Generation via Knowledge Injection." pith.science (2026). https://pith.science/paper/E6SYRR3T

@misc{pith2026250416429,
  author       = {Pith},
  title        = {Pith review of: Give LLMs a Security Course: Securing Retrieval-Augmented Code Generation via Knowledge Injection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E6SYRR3T}},
  note         = {Machine review of arXiv:2504.16429}
}
read the original abstract

Retrieval-Augmented Code Generation (RACG) leverages external knowledge to enhance Large Language Models (LLMs) in code synthesis, improving the functional correctness of the generated code. However, existing RACG systems largely overlook security, leading to substantial risks. Especially, the poisoning of malicious code into knowledge bases can mislead LLMs, resulting in the generation of insecure outputs, which poses a critical threat in modern software development. To address this, we propose a security-hardening framework for RACG systems, CodeGuarder, that shifts the paradigm from retrieving only functional code examples to incorporating both functional code and security knowledge. Our framework constructs a security knowledge base from real-world vulnerability databases, including secure code samples and root cause annotations. For each code generation query, a retriever decomposes the query into fine-grained sub-tasks and fetches relevant security knowledge. To prioritize critical security guidance, we introduce a re-ranking and filtering mechanism by leveraging the LLMs' susceptibility to different vulnerability types. This filtered security knowledge is seamlessly integrated into the generation prompt. Our evaluation shows CodeGuarder significantly improves code security rates across various LLMs, achieving average improvements of 20.12\% in standard RACG, and 31.53\% and 21.91\% under two distinct poisoning scenarios without compromising functional correctness. Furthermore, CodeGuarder demonstrates strong generalization, enhancing security even when the targeted language's security knowledge is lacking. This work presents CodeGuarder as a pivotal advancement towards building secure and trustworthy RACG systems.

Figures

Figures reproduced from arXiv: 2504.16429 by the authors.

Figure 1
Figure 1. An example of code generated by GPT-4o with and without [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Vulnerable code generated by GPT-4o along with its corresponding decomposed queries and retrieved knowledge. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. An example of security knowledge extracted from vulnera [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Knowdit: Agentic Smart Contract Vulnerability Detection with Auditing Knowledge Summarization

    cs.CR 2026-03 conditional novelty 6.0 of 10

    Knowdit links abstract DeFi semantics to vulnerability patterns in a knowledge graph and drives an agentic specify–harness–fuzz–reflect loop that finds all high-severity and most medium-severity bugs on held-out Code4...

Reference graph

Works this paper leans on

56 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    2024. Infer. https://fbinfer.com/

    work page 2024

  2. [2]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021)

    arXiv 2021

  3. [3]

    Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evti- mov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, et al. 2023. Purple llama cyberseceval: A secure coding benchmark for language models. arXiv preprint arXiv:2312.04724 (2023)

    arXiv 2023

  4. [4]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    2020

  5. [5]

    Kaiyan Chang, Songcheng Xu, Chenglong Wang, Yingfeng Luo, Xiaoqian Liu, Tong Xiao, and Jingbo Zhu. 2024. Efficient prompting methods for large language models: A survey. arXiv preprint arXiv:2404.01077 (2024)

    arXiv 2024

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    arXiv 2021

  7. [7]

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research 25, 70 (2024), 1–53

    2024

  8. [8]

    CWE Team. 2024. 2024 CWE Top 25 Most Dangerous Software Weaknesses. MITRE. https://cwe.mitre.org/top25/archive/2024/2024_cwe_top25.html

    work page 2024

Show all 56 references
  1. [9]

    Xinyu Gao, Yun Xiong, Deze Wang, Zhenhan Guan, Zejian Shi, Haofen Wang, and Shanshan Li. 2024. Preference-Guided Refactored Tuning for Retrieval Augmented Code Generation. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering . 65–77

    2024

  2. [10]

    GitHub. 2023. CodeQL. https://codeql.github.com

    2023

  3. [11]

    Suhas Hariharan, Zainab Ali Majid, Jaime Raldua Veuthey, and Jacob Haimes

  4. [12]

    Haitao He, Sheng Wang, Yanmin Wang, Ke Liu, and Lu Yu. 2025. VulTR: Software vulnerability detection model based on multi-layer key feature enhancement. Computers & Security 148 (2025), 104139

    2025

  5. [13]

    Jingxuan He and Martin Vechev. 2023. Large language models for code: Secu- rity hardening and adversarial testing. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security . 1865–1879

    2023

  6. [14]

    Jingxuan He, Mark Vero, Gabriela Krasnopolska, and Martin Vechev. 2024. In- struction tuning for secure code generation. arXiv preprint arXiv:2402.09497 (2024)

    2024 arXiv

  7. [15]

    Kaifeng Huang, Chenhao Lu, Yiheng Cao, Bihuan Chen, and Xin Peng. 2024. VMUD: Detecting Recurring Vulnerabilities with Multiple Fixing Functions via Function Selection and Semantic Equivalent Statement Matching. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer a...

    2024

  8. [16]

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436 (2019)

    2019 arXiv

  9. [17]

    Jiang, Arthur Sablayrolles, Alexandre Mensch, Chris Bamford, De- vendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gábor Lengyel, Guillaume Lample, Lucile Saulnier, et al

    Albert Q. Jiang, Arthur Sablayrolles, Alexandre Mensch, Chris Bamford, De- vendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gábor Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7B. CoRR abs/2310.06825 (2023). https://arxiv.org/abs/2310.06825

    2023 arXiv

  10. [18]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020)

    2020 arXiv

  11. [19]

    Arya Kavian, Mohammad Mehdi Pourhashem Kallehbasti, Sajjad Kazemi, Ehsan Firouzi, and Mohammad Ghafari. 2024. Llm security guard for code. InProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. 600–603

    2024

  12. [20]

    Jan H Klemmer, Stefan Albert Horstmann, Nikhil Patnaik, Cordelia Ludden, Cordell Burton Jr, Carson Powers, Fabio Massacci, Akond Rahman, Daniel Votipka, Heather Richter Lipford, et al. 2024. Using AI Assistants in Software Development: A Qualitative Study on Security Practices...

    2024

  13. [21]

    Dong Li, Meng Yan, Yaosheng Zhang, Zhongxin Liu, Chao Liu, Xiaohong Zhang, Ting Chen, and David Lo. 2024. CoSec: On-the-Fly Security Hardening of Code LLMs via Supervised Co-decoding. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analys...

    2024

  14. [22]

    Jia Li, Ge Li, Yongmin Li, and Zhi Jin. 2025. Structured chain-of-thought prompt- ing for code generation. ACM Transactions on Software Engineering and Method- ology 34, 2 (2025), 1–23

    2025

  15. [23]

    Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 (2021)

    2021 arXiv

  16. [24]

    Bo Lin, Shangwen Wang, Liqian Chen, and Xiaoguang Mao. 2025. Exploring the Security Threats of Knowledge Base Poisoning in Retrieval-Augmented Code Generation. arXiv preprint arXiv:2502.03233 (2025)

    2025 arXiv

  17. [25]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)

    2024 arXiv

  18. [26]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157–173

    2024

  19. [27]

    Microsoft. 2024. Retrieval-Augmented Generation (RAG) in Azure Machine Learning. https://learn.microsoft.com/en-us/azure/machine-learning/concept- retrieval-augmented-generation

    2024

  20. [28]

    Ollama. 2023. Ollama framework. https://ollama.com/

    2023

  21. [29]

    OpenAI. [n. d.]. GPT-4o. https://platform.openai.com/docs/models#gpt-4. Conference’17, July 2017, Washington, DC, USA Bo Lin, Shangwen Wang, Yihao Qin, Liqian Chen, and Xiaoguang Mao

    2017

  22. [30]

    OpenAI. 2024. ChatGPT Retrieval Plugin. https://github.com/openai/chatgpt- retrieval-plugin

    2024

  23. [31]

    OpenAI. 2024. OpenAI API Reference. https://platform.openai.com/docs/ apireference

    2024

  24. [32]

    OpenAI. 2025. OpenAI Models - Embeddings. https://platform.openai.com/docs/ models/gpt#embeddings

    2025

  25. [33]

    Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. 2024. An empirical study of the non-determinism of chatgpt in code generation. ACM Transactions on Software Engineering and Methodology (2024)

    2024

  26. [34]

    Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the keyboard? assessing the security of github copilot’s code contributions. In 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 754–768

    2022

  27. [35]

    Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby. 2023. From sparse to soft mixtures of experts. arXiv preprint arXiv:2308.00951 (2023)

    2023 arXiv

  28. [36]

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundare- san, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: a Method for Automatic Evaluation of Code Synthesis. arXiv:2009.10297 [cs.SE]

    2020 arXiv

  29. [37]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

  30. [38]

    Semgrep, Inc. 2025. Semgrep. https://semgrep.dev/

    2025

  31. [39]

    Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, et al. 2024. jina-embeddings-v3: Multilingual embeddings with task lora. arXiv preprint arXiv:2409.10173 (2024)

    2024 arXiv

  32. [40]

    Hongjin Su, Shuyang Jiang, Yuhang Lai, Haoyuan Wu, Boao Shi, Che Liu, Qian Liu, and Tao Yu. 2024. EvoR: Evolving Retrieval for Code Generation. In Findings of the Association for Computational Linguistics: EMNLP 2024 . 2538–2554

    2024

  33. [41]

    Norbert Tihanyi, Tamas Bisztray, Mohamed Amine Ferrag, Ridhi Jain, and Lucas C Cordeiro. 2025. How secure is AI-generated code: a large-scale comparison of large language models. Empirical Software Engineering 30, 2 (2025), 1–42

    2025

  34. [42]

    Catherine Tony, Markus Mutas, Nicolás E Díaz Ferreyra, and Riccardo Scandariato

  35. [43]

    Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman, Stephanie Ding, et al . 2024. Cyberseceval 3: Advancing the evaluation of cy- bersecurity risks and capabilities in large language mode...

    2024 arXiv

  36. [44]

    Jiexin Wang, Xitong Luo, Liuwen Cao, Hongkui He, Hailin Huang, Jiayuan Xie, Adam Jatowt, and Yi Cai. 2024. Is your ai-generated code really safe? evaluating large language models on secure code generation with codeseceval.arXiv preprint arXiv:2407.02395 (2024)

    2024 arXiv

  37. [45]

    Xinchen Wang, Ruida Hu, Cuiyun Gao, Xin-Cheng Wen, Yujia Chen, and Qing Liao. 2024. ReposVul: A Repository-Level High-Quality Vulnerability Dataset. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings. 472–483

    2024

  38. [46]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  39. [47]

    Felix Wilhelm, Fabian Freyer, Calle Svensson, Thomas Otto, Michal Melewski, William Woodruff, Disconnect3d, and Matthew Rinaldi. 2025. weggli. https: //github.com/weggli-rs/weggli

    2025

  40. [48]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis

  41. [49]

    Zezhou Yang, Sirong Chen, Cuiyun Gao, Zhenhao Li, Xing Hu, Kui Liu, and Xin Xia. 2025. An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunities. ACM Transactions on Software Engineering and Methodology (2025)

    2025

  42. [50]

    Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792 (2023)

    2023

  43. [51]

    arXiv preprint arXiv:2309.17453 (2023)

    Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453 (2023)

    2023 arXiv

  44. [52]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023)

    2023 arXiv

  45. [53]

    Functionality

    Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al . 2024. DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence.arXiv preprint arXiv:2406.11931 (2024). Give LLMs a Security Course...

    2024 arXiv

  46. [54]

    Xiangyu Zhang, Yu Zhou, Guang Yang, and Taolue Chen. 2023. Syntax-aware retrieval augmented code generation. In Findings of the Association for Computa- tional Linguistics: EMNLP 2023 . 1291–1302

    2023

  47. [2023]

    In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR)

    Llmseceval: A dataset of natural language prompts for security evaluations. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR). IEEE, 588–592

    2023

  48. [2024]

    arXiv preprint arXiv:2411.08813 (2024)

    Rethinking CyberSecEval: An LLM-Aided Approach to Evaluation Critique. arXiv preprint arXiv:2411.08813 (2024)

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.