Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Exploring the Security Threats of Knowledge Base Poisoning in Retrieval-Augmented Code Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Knowledge base poisoning can push nearly half of retrieval-augmented code generation into producing vulnerable code.

desk verdict Useful first mapping of a plausible RACG attack surface, but the headline VR numbers likely reflect an LLM-judge input confound more than actual code insecurity. read the letter →

arxiv 2502.03233 v1 pith:YZC5K3PL submitted 2025-02-05 cs.CR cs.SE

Bo Lin , Shangwen Wang , Liqian Chen , Xiaoguang Mao This is my paper · ORCID
classification cs.CRcs.SE
keywords retrieval-augmentedcodegenerationknowledgebasepoisoningLLMsecurityvulnerabilitypropagationdatapoisonedLLM-as-a-judge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish a concrete security risk in retrieval-augmented code generation (RACG): when the knowledge base used to fetch code examples contains a small amount of deliberately injected vulnerable code, the code the LLM generates becomes measurably less secure, even though its functional quality barely changes. Across four LLMs, two retrievers, and two attack scenarios, the authors show that one poisoned example can push the vulnerability rate of generated code to about 48% for CodeLlama with a dense retriever, and that a hidden-intent poisoning strategy with 20% of the knowledge base poisoned still yields roughly 36% vulnerable code for that model. The study matters because public code repositories are common sources for RACG knowledge bases, so poisoning does not require special access to the system; it only requires getting code into a repository that a retriever may later index.

What carries the argument

The load-bearing mechanism is the RACG pipeline itself: a retriever selects code snippets from a poisoned knowledge base, and the LLM treats those snippets as in-context examples when answering the query. The attack works because dense retrievers rank by semantic similarity, so vulnerable code that matches the query's meaning is likely to be retrieved, and the LLM then reproduces the vulnerable pattern it sees. The paper's experiment centers on the vulnerability rate (VR) metric, defined as the fraction of generated code snippets judged vulnerable, together with VRRC, the fraction of retrieved examples that are poisoned, which lets the authors connect retrieval success to generation risk.

What would settle it

Take a random sample of generated code from the BM25, low-similarity, and three-shot conditions, have human security experts label it for vulnerabilities, and compare those labels with the LLM judge's labels; if the judge's precision or recall in those settings differs materially from the 77-81% accuracy reported for the single validated setting, the poisoning effects reported there are not established.

Watch

Extended reading notes

Core claim

The paper's central claim is that the security of LLM-generated code in RACG systems depends on the contents of the retrieved knowledge base, not just on the model's own tendencies. It demonstrates this by measuring the vulnerability rate (VR) of generated code under clean and poisoned knowledge bases, with poisoning done either by matching vulnerable examples to the user's exposed query (Scenario I) or by selecting representative vulnerable examples through clustering when the query is hidden (Scenario II). The authors find that vulnerability propagates strongly with dense retrieval, that more few-shot examples increase the chance that vulnerabilities are copied, and that higher example-query similarity correlates sharply with higher vulnerability rates above the 60% similarity threshold. They also find that vulnerability type matters, with CWE-352 (Cross-Site Request Forgery) producing the highest vulnerability rates, and that code-specialized LLMs are more susceptible than general-purpose LLMs.

Load-bearing premise

The whole comparison rests on the LLM-as-a-judge that labels generated code as vulnerable or not; the judge has roughly 80% accuracy and was manually validated in only one of the sixteen sub-scenarios, so if its error rate differs in the other settings the reported vulnerability-rate gaps may be measurement artifacts rather than real poisoning effects.

Editorial extensions

If this is right

  • Dense semantic retrievers make RACG systems dramatically more poisonable than sparse lexical retrievers, so secure deployments should not assume retrieval quality alone protects generated code.
  • Small poisoning rates are enough to matter: in the exposed-intent scenario a single injected example produced a large VR spike, so knowledge base curation must treat even tiny contamination as a real threat.
  • Adding few-shot examples to improve code quality carries a security cost, increasing vulnerability rates by about 6.5% from one-shot to three-shot with the JINA retriever.
  • Code-specialized LLMs such as CodeLlama are more susceptible to copying vulnerable patterns than general LLMs, so defenses and evaluations should weight them more heavily.
  • Because vulnerability rates rise sharply when retrieved examples are more than 60% similar to the query, retrieval strategies that avoid the single highest-similarity example could reduce risk while preserving much of the performance benefit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's 48% figure likely underestimates the worst case, because real attackers can combine exposed-intent poisoning with targeted queries and larger knowledge bases; the authors only use a few poisoning quantities and one-shot context for that headline number.
  • A natural next experiment the authors leave implicit is to test retrieval defenses, such as sampling among top-k candidates or filtering retrieved code with a vulnerability judge before generation, to see whether the security gain costs acceptable similarity loss.
  • The strong CWE-352 result suggests a cheap defensive screen: scan knowledge bases for known high-risk vulnerability types rather than trying to detect all possible poisoned code.
  • Since the judge pipeline already extracts vulnerable and fixing patterns, the same mechanism could be deployed at generation time as a self-check, though it would need higher accuracy than the reported roughly 80% to be trusted as a filter.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper studies the security of retrieval-augmented code generation (RACG) under knowledge base poisoning. Using a dataset built from ReposVul, the authors construct two attack scenarios (exposed vs. hidden programming intent), poison a knowledge base of secure code with vulnerable examples, and measure the vulnerability rate (VR) of code generated by four LLMs (GPT-4o, Llama-3, CodeLlama, DS-Coder) with two retrievers (JINA and BM25). The central claim is that even a single poisoned example can make approximately 48% of generated code vulnerable (Finding 1, Section 5.1.1), that more shots increase vulnerability risk (Finding 2), and that programming language, example-query similarity, and CWE type are influential factors (Findings 3-5). Vulnerability is assessed by a two-step LLM-as-a-Judge pipeline described in Section 4.3 and Appendix E.

Significance. If the quantitative findings were fully supported, this would be a timely and useful first systematic study of knowledge base poisoning in RACG: the threat model is well motivated, the experimental matrix is broad (16 sub-scenarios, four LLMs, two retrievers, two scenarios), and the paper includes valuable design details such as temperature-0 generation, public model versions, and a detailed judge pipeline. The paper also makes a useful distinction from prior RAG poisoning work by focusing on code security rather than factual accuracy. However, the main quantitative contribution rests on a measurement pipeline whose input changes with the treatment, so the headline effect sizes and factor analyses need additional validation before the findings can be taken at face value.

major comments (3)
  1. [Section 4.3 / Appendix E / Prompt 3] The VR metric is computed by an LLM judge that receives vulnerability cause patterns extracted from the injected vulnerable examples. In the unpoisoned condition the judge receives only internal patterns from the query's own (v, s) pair, while in poisoned conditions it additionally receives external patterns from every retrieved vulnerable example. This means the judge's input is not held constant across conditions, so the reported VR increases may reflect the presence and number of supplied patterns rather than actual insecurity of the generated code. The manual validation in Table 10 covers only one sub-scenario (GPT-4o, JINA, Scenario I, one-shot, five poisoned examples) and the automated validation uses vulnerable/fixed pairs rather than generated code with varying external-pattern counts. I would like to see a control experiment that keeps the number and structure of supplied patterns fixed across poisoned and unpoisoned conditions, or an independent human/static validation on generated code from multiple conditions, before the 0.29-to-0.48 style increases are attributed to poisoning.
  2. [Table 4 / Finding 1] The headline '48% of generated code vulnerable' is an absolute VR, not the effect attributable to poisoning. For CodeLlama with JINA, the baseline VR is 0.29, so the marginal increase from a single poisoned sample is 19 percentage points, not 48 percentage points. The abstract and Finding 1 present the absolute number as if it were the poisoning effect, which conflates the LLM's baseline vulnerability with the attack's marginal impact. The paper should report baseline-subtracted effect sizes (e.g., +0.19 for CodeLlama/JINA) and should be careful in the abstract and conclusion to distinguish absolute VR from poisoning-induced increase.
  3. [Tables 4-6, 8, 9] All VR and similarity results are point estimates without confidence intervals, error bars, or significance tests. Many of the paper's conclusions rely on small differences, such as the 6.5% VR increase from one-shot to three-shot (0.46 to 0.49 in the 'All' column of Table 6) or the 0.01-0.03 changes reported in Tables 4 and 5. Given that the underlying judge itself has 77-84% accuracy, it is important to quantify the uncertainty in these proportions, for example via bootstrap confidence intervals or appropriate statistical tests, before asserting that the observed monotonic trends are meaningful. This is load-bearing for both RQ1 and RQ2.
minor comments (6)
  1. [Section 4.1] The text states the dataset contains 12,052 instances, but the rows in Table 2 sum to 12,053; please correct the inconsistency.
  2. [Section 6.2] "a 10 confidence interval" appears to be missing a percent sign; it should likely read "a 10% confidence interval" or should specify the intended interval width.
  3. [Section 4.5] The formal definition of VR is typeset incorrectly as 'V R= Nv Nt'; please use proper mathematical notation.
  4. [Section 3.2] In the paragraph after the optimization problem, the text says "all queries q share the same set of vulnerable examples Q"; the set should presumably be V, not Q.
  5. [Section 5.2.2] The text refers to "VRCC" in one place; this should be "VRRC" for consistency with the metric definition in Section 4.5.
  6. [Section 6.3 / Table 11] The definitions of MRR and SuccessRate@k are not fully operationalized: it is not specified what counts as "the relevant code snippet" for a query in the RACG setting. Please clarify the ground truth used for this evaluation.

Circularity Check

1 steps flagged · score 4.0 of 10

VR increases may reflect judge input rather than code insecurity: in poisoned conditions the LLM judge is given additional vulnerability patterns to match, and validation (Table 10) does not control for this confound.

  1. other [Appendix E.0.2 (Security Assessment), Prompt 3; see also §4.3]
    "For a given piece of generated code c derived from a query q, the code is deemed vulnerable if and only if c matches any vulnerability cause patterns from the following two sources: • External Sources: Vulnerable patterns extracted from examples of vulnerable code that were referenced (i.e., the vulnerable code that retrieved as examples) during the generation of c. • Internal Sources: Vulnerable patterns identified within the vulnerable version v and the corresponding secure version s associated with the query q."

    The judge's vulnerability label is defined as a match against patterns extracted from the retrieved vulnerable examples (external sources). The poisoning treatment directly controls how many such examples are retrieved: Table 4 shows VRRC rising from 0.38 (m=1) to 0.42 (m=9) for JINA, and from 0.05 to 0.07 for BM25. Because Prompt 3 asks the judge to output 'found' for any supplied pattern, adding poisoned examples adds candidate patterns to match against the same generated code, so the measured VR can rise even if the generated code is unchanged.

full rationale

The paper is an empirical measurement study, not a derivation, and its core causal direction (poisoned retrieved examples can lead LLMs to copy vulnerable patterns) is plausible and has some independent support: the manual inspection in §6.2 achieves 0.72–0.84 accuracy/F1 on a poisoned GPT-4o/JINA/Scenario-I condition, and the automated vulnerable/fixed-pair evaluation shows the judge can distinguish vulnerable from fixed code. However, the measurement pipeline is partially circular in the sense that the LLM judge's input pattern list is itself derived from the poisoned examples whose effect is being measured. Appendix E defines a code as vulnerable iff it matches any pattern from internal or external sources, and the external sources are exactly the retrieved vulnerable examples; the number of poisoned examples therefore directly increases the number of patterns supplied to the judge. Because Prompt 3 flags code if any supplied pattern matches, part of the VR increase in Tables 4–6 is a mechanical consequence of a larger pattern list rather than a demonstrated change in the generated code. The validation in Table 10 does not control for pattern-count effects: manual labels cover only one sub-scenario, and automated labels are on vulnerable/fixed pairs, not on generated code with varying external-pattern counts. The BM25 results, where VRRC is much lower (0.05–0.07) and VR rises less, are exactly what the confound predicts. I do not find load-bearing self-citation: the citations to the authors' prior work ([25] for query generation, [77] for retriever metrics) are auxiliary and not the basis of the central claim. The finding therefore retains independent content, but the headline VR numbers are not cleanly separable from the judge-input confound, so score 4 rather than 0 or 6.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the validity of the LLM judge, the correctness of the ReposVul labels, the threat model's assumption of injectable public code repositories, and the accuracy of LLM-generated queries. The number of clusters in Scenario II is chosen by an unreported elbow method, which is the main unreported design parameter. No new entities are introduced.

free parameters (1)
  • Number of clusters t in Scenario II = not reported (selected by elbow method)
    The clustering-based selection in Sec 4.2.2 depends on the number of clusters t, which is determined by an elbow method but not reported. Different t changes which vulnerable examples are injected and thus affects VRRC and VR.
assumptions (4)
  • domain assumption LLM-as-a-Judge provides accurate vulnerability labels for generated code across all settings.
    The VR metric in Sec 4.3 and 5 depends entirely on an LLM judge that is manually validated on only one sub-scenario (Sec 6.2). The assumption that accuracy extrapolates to all 16 sub-scenarios is load-bearing.
  • domain assumption ReposVul's vulnerable and secure version pairs are correctly labeled.
    The dataset is used as ground truth for poisoning and for extracting vulnerability patterns, and its quality is taken from the original dataset paper without independent verification.
  • domain assumption The attacker can inject code into the public repositories that form the RACG knowledge base.
    Stated in Sec 3.1 and supported by prior work [8], but it is a premise of the threat model.
  • domain assumption Generated queries (via DeepSeek-V2.5) accurately reflect function intent.
    Sec 4.1 uses LLM-generated queries, with manual review finding 86% accuracy. The remaining 14% may distort retrieval and generation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring the Security Threats of Knowledge Base Poisoning in Retrieval-Augmented Code Generation." pith.science (2026). https://pith.science/paper/YZC5K3PL

@misc{pith2026250203233,
  author       = {Pith},
  title        = {Pith review of: Exploring the Security Threats of Knowledge Base Poisoning in Retrieval-Augmented Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YZC5K3PL}},
  note         = {Machine review of arXiv:2502.03233}
}
read the original abstract

The integration of Large Language Models (LLMs) into software development has revolutionized the field, particularly through the use of Retrieval-Augmented Code Generation (RACG) systems that enhance code generation with information from external knowledge bases. However, the security implications of RACG systems, particularly the risks posed by vulnerable code examples in the knowledge base, remain largely unexplored. This risk is particularly concerning given that public code repositories, which often serve as the sources for knowledge base collection in RACG systems, are usually accessible to anyone in the community. Malicious attackers can exploit this accessibility to inject vulnerable code into the knowledge base, making it toxic. Once these poisoned samples are retrieved and incorporated into the generated code, they can propagate security vulnerabilities into the final product. This paper presents the first comprehensive study on the security risks associated with RACG systems, focusing on how vulnerable code in the knowledge base compromises the security of generated code. We investigate the LLM-generated code security across different settings through extensive experiments using four major LLMs, two retrievers, and two poisoning scenarios. Our findings highlight the significant threat of knowledge base poisoning, where even a single poisoned code example can compromise up to 48% of generated code. Our findings provide crucial insights into vulnerability introduction in RACG systems and offer practical mitigation recommendations, thereby helping improve the security of LLM-generated code in future works.

Figures

Figures reproduced from arXiv: 2502.03233 by the authors.

Figure 1
Figure 1. A typical workflow of the RACG system. with relevant information from the external knowledge base to improve the quality of code generation [23, 55, 82]. However, the adoption of RACG presents a double-edged sword: while it can improve code quality by providing relevant examples, it also introduces potential security vulnerabilities when the knowledge base contains vulnerable examples. This risk is particularly conc… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ImportSnare: Directed "Code Manual" Hijacking in Retrieval-Augmented Code Generation

    cs.CR 2025-09 conditional novelty 6.0 of 10

    Documentation poisoning with hidden ranking and suggestion sequences can make RAG-based code generators confidently recommend malicious dependencies, even at 0.01% poisoning ratios.

Reference graph

Works this paper leans on

93 extracted references · 47 canonical work pages · cited by 1 Pith paper

  1. [1]

    Cve- fixes: automated collection of vulnerabilities and their fixes from open-source software

    Guru Bhandari, Amara Naseer, and Leon Moonen. Cve- fixes: automated collection of vulnerabilities and their fixes from open-source software. In Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering, pages 30–39, 2021

    2021

  2. [2]

    Ebk-means: A clustering technique based on elbow method and k- means in wsn

    Purnima Bholowalia and Arvind Kumar. Ebk-means: A clustering technique based on elbow method and k- means in wsn. International Journal of Computer Ap- plications, 105(9), 2014

    2014

  3. [3]

    Poi- soning attacks against support vector machines

    Battista Biggio, Blaine Nelson, and Pavel Laskov. Poi- soning attacks against support vector machines. arXiv preprint arXiv:1206.6389, 2012

    arXiv 2012

  4. [4]

    Wild patterns: Ten years after the rise of adversarial machine learning

    Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pages 2154– 2156, 2018

    2018

  5. [5]

    A software assurance reference dataset: Thousands of programs with known bugs

    Paul E Black. A software assurance reference dataset: Thousands of programs with known bugs. Journal of research of the National Institute of Standards and Tech- nology, 123:1, 2018

    2018

  6. [6]

    Juliet 1

    Tim Boland and Paul E Black. Juliet 1. 1 c/c++ and java test suite. Computer, 45(10):88–90, 2012

    2012

  7. [7]

    Language models are few-shot learn- ers

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learn- ers. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  8. [8]

    Poisoning web-scale training datasets is practi- cal

    Nicholas Carlini, Matthew Jagielski, Christopher A Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning web-scale training datasets is practi- cal. In 2024 IEEE Symposium on Security and Privacy (SP), pages 407–425. IEEE, 2024

    2024

Show all 93 references
  1. [9]

    Deep learning based vulnerability detection: Are we there yet? IEEE Transactions on Software Engineering, 48(9):3280–3296, 2021

    Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. Deep learning based vulnerability detection: Are we there yet? IEEE Transactions on Software Engineering, 48(9):3280–3296, 2021

    2021

  2. [10]

    A survey on evaluation of large language models

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunx- iang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelli- gent Systems and Technology, 15(3):1–45, 2024

    2024

  3. [11]

    Rmcbench: Benchmarking large lan- guage models’ resistance to malicious code

    Jiachi Chen, Qingyuan Zhong, Yanlin Wang, Kaiwen Ning, Yongkun Liu, Zenan Xu, Zhe Zhao, Ting Chen, and Zibin Zheng. Rmcbench: Benchmarking large lan- guage models’ resistance to malicious code. In Proceed- ings of the 39th IEEE/ACM International Conference on Automated Softwar...

    2024

  4. [12]

    Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection

    Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David Wagner. Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection. In Proceedings of the 26th International Sym- posium on Research in Attacks, Intrusions and Defenses, pag...

    2023

  5. [13]

    Lift yourself up: Retrieval- augmented text generation with self-memory

    Xin Cheng, Di Luo, Xiuying Chen, Lemao Liu, Dongyan Zhao, and Rui Yan. Lift yourself up: Retrieval- augmented text generation with self-memory. Advances in Neural Information Processing Systems, 36, 2024

    2024

  6. [14]

    Introduction to the k-means cluster- ing algorithm based on the elbow method

    Mengyao Cui et al. Introduction to the k-means cluster- ing algorithm based on the elbow method. Accounting, Auditing and Finance, 1(1):5–8, 2020

    2020

  7. [15]

    Deepseek v2.5, 2024

    DeepSeek. Deepseek v2.5, 2024

    2024

  8. [16]

    Masterkey: Automated jailbreaking of large language model chatbots

    Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. Masterkey: Automated jailbreaking of large language model chatbots. In Proc. ISOC NDSS, 2024. 14

    2024

  9. [17]

    Vul-rag: Enhancing llm-based vulnerability detection via knowledge-level rag

    Xueying Du, Geng Zheng, Kaixin Wang, Jiayi Feng, Wentai Deng, Mingwei Liu, Bihuan Chen, Xin Peng, Tao Ma, and Yiling Lou. Vul-rag: Enhancing llm-based vulnerability detection via knowledge-level rag. arXiv preprint arXiv:2406.11147, 2024

    2024 arXiv

  10. [18]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  11. [19]

    Empirical review of automated analysis tools on 47,587 ethereum smart contracts

    Thomas Durieux, João F Ferreira, Rui Abreu, and Pedro Cruz. Empirical review of automated analysis tools on 47,587 ethereum smart contracts. In Proceedings of the ACM/IEEE 42nd International conference on software engineering, pages 530–541, 2020

    2020

  12. [20]

    Crystalbleu: pre- cisely and efficiently measuring the similarity of code

    Aryaz Eghbali and Michael Pradel. Crystalbleu: pre- cisely and efficiently measuring the similarity of code. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pages 1–12, 2022

    2022

  13. [21]

    Ac/c++ code vulnerability dataset with code changes and cve summaries

    Jiahao Fan, Yi Li, Shaohua Wang, and Tien N Nguyen. Ac/c++ code vulnerability dataset with code changes and cve summaries. In Proceedings of the 17th Inter- national Conference on Mining Software Repositories, pages 508–512, 2020

    2020

  14. [22]

    go-ethereum: Official go implementation of the ethereum protocol

    The Ethereum Foundation. go-ethereum: Official go implementation of the ethereum protocol. https:// github.com/ethereum/go-ethereum, 2024

    2024

  15. [23]

    Preference- guided refactored tuning for retrieval augmented code generation

    Xinyu Gao, Yun Xiong, Deze Wang, Zhenhan Guan, Zejian Shi, Haofen Wang, and Shanshan Li. Preference- guided refactored tuning for retrieval augmented code generation. In Proceedings of the 39th IEEE/ACM Inter- national Conference on Automated Software Engineer- ing, pages 65–77, 2024

    2024

  16. [24]

    Retrieval-augmented generation for large language mod- els: A survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jin- liu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmented generation for large language mod- els: A survey. arXiv preprint arXiv:2312.10997, 2023

    2023 arXiv

  17. [25]

    Large language models are few-shot summarizers: Multi-intent comment generation via in-context learn- ing

    Mingyang Geng, Shangwen Wang, Dezun Dong, Hao- tian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, and Xiangke Liao. Large language models are few-shot summarizers: Multi-intent comment generation via in-context learn- ing. In Proceedings of the 46th IEEE/ACM International Conference on...

    2024

  18. [26]

    How effec- tive are smart contract analysis tools? evaluating smart contract static analysis tools using bug injection

    Asem Ghaleb and Karthik Pattabiraman. How effec- tive are smart contract analysis tools? evaluating smart contract static analysis tools using bug injection. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis , pages 415–427, 2020

    2020

  19. [27]

    Re2g: Retrieve, rerank, generate

    Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Rajaram Naik, Pengshan Cai, and Alfio Gliozzo. Re2g: Retrieve, rerank, generate. arXiv preprint arXiv:2207.06300, 2022

    2022 arXiv

  20. [28]

    An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge models are task- specific classifiers

    Hui Huang, Yingqi Qu, Jing Liu, Muyun Yang, and Tiejun Zhao. An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge models are task- specific classifiers. arXiv preprint arXiv:2403.02839, 2024

    2024 arXiv

  21. [29]

    Codesearchnet challenge: Evaluating the state of semantic code search

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436, 2019

    1909 arXiv

  22. [30]

    Jigsaw: Large language models meet program synthesis

    Naman Jain, Skanda Vaidyanath, Arun Iyer, Nagarajan Natarajan, Suresh Parthasarathy, Sriram Rajamani, and Rahul Sharma. Jigsaw: Large language models meet program synthesis. In Proceedings of the 44th Inter- national Conference on Software Engineering , pages 1219–1231, 2022

    2022

  23. [31]

    How secure is code generated by chatgpt? In 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 2445–

    Raphaël Khoury, Anderson R Avila, Jacob Brunelle, and Baba Mamadou Camara. How secure is code generated by chatgpt? In 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 2445–

    2023

  24. [32]

    Using ai assistants in software development: A qualitative study on security practices and concerns

    Jan H Klemmer, Stefan Albert Horstmann, Nikhil Pat- naik, Cordelia Ludden, Cordell Burton Jr, Carson Pow- ers, Fabio Massacci, Akond Rahman, Daniel V otipka, Heather Richter Lipford, et al. Using ai assistants in software development: A qualitative study on security practices ...

    2024

  25. [33]

    Multi-step jail- breaking privacy attacks on chatgpt

    Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. Multi-step jail- breaking privacy attacks on chatgpt. arXiv preprint arXiv:2304.05197, 2023

    2023 arXiv

  26. [34]

    From classi- fication to generation: Insights into crosslingual retrieval augmented icl

    Xiaoqian Li, Ercong Nie, and Sheng Liang. From classi- fication to generation: Insights into crosslingual retrieval augmented icl. arXiv preprint arXiv:2311.06595, 2023

    2023 arXiv

  27. [35]

    Opportunities and challenges in code search tools

    Chao Liu, Xin Xia, David Lo, Cuiyun Gao, Xiaohu Yang, and John Grundy. Opportunities and challenges in code search tools. ACM Computing Surveys (CSUR), 54(9):1– 40, 2021

    2021

  28. [36]

    Determine the number of unknown targets in open world based on elbow method

    Fan Liu and Yong Deng. Determine the number of unknown targets in open world based on elbow method. IEEE Transactions on Fuzzy Systems, 29(5):986–995, 2020. 15

    2020

  29. [37]

    Trojan- ing attack on neural networks

    Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojan- ing attack on neural networks. In 25th Annual Network And Distributed System Security Symposium (NDSS 2018). Internet Soc, 2018

    2018

  30. [38]

    An empirical study on low code program- ming using traditional vs large language model support

    Yongkun Liu, Jiachi Chen, Tingting Bi, John Grundy, Yanlin Wang, Jianxing Yu, Ting Chen, Yutian Tang, and Zibin Zheng. An empirical study on low code program- ming using traditional vs large language model support. arXiv preprint arXiv:2402.01156, 2024

    2024

  31. [39]

    Prompt injection attacks and defenses in llm-integrated applications

    Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. Prompt injection attacks and defenses in llm-integrated applications. arXiv preprint arXiv:2310.12815, 2023

    2023

  32. [40]

    Formalizing and benchmarking prompt injection attacks and defenses

    Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. Formalizing and benchmarking prompt injection attacks and defenses. In 33rd USENIX Security Symposium (USENIX Security 24), pages 1831– 1847, 2024

    2024

  33. [41]

    Query rewriting for retrieval- augmented large language models

    Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. Query rewriting for retrieval- augmented large language models. arXiv preprint arXiv:2305.14283, 2023

    2023 arXiv

  34. [42]

    Some methods for classifica- tion and analysis of multivariate observations

    James MacQueen et al. Some methods for classifica- tion and analysis of multivariate observations. In Pro- ceedings of the fifth Berkeley symposium on mathemati- cal statistics and probability, volume 1, pages 281–297. Oakland, CA, USA, 1967

    1967

  35. [43]

    Github language statistics (githut), 2024

    madnight. Github language statistics (githut), 2024

    2024

  36. [44]

    Kragen: a knowledge graph- enhanced rag framework for biomedical problem solv- ing using large language models

    Nicholas Matsumoto, Jay Moran, Hyunjun Choi, Miguel E Hernandez, Mythreye Venkatesan, Paul Wang, and Jason H Moore. Kragen: a knowledge graph- enhanced rag framework for biomedical problem solv- ing using large language models. Bioinformatics, 40(6), 2024

    2024

  37. [45]

    Retrieval-augmented generation (rag) in azure machine learning, 2024

    Microsoft. Retrieval-augmented generation (rag) in azure machine learning, 2024

    2024

  38. [46]

    2024 cwe top 25 most danger- ous software weaknesses

    MITRE Corporation. 2024 cwe top 25 most danger- ous software weaknesses. https://cwe.mitre.org/ top25/archive/2024/2024_cwe_top25.html, 2024. Accessed: 2024-10-01

    2024

  39. [47]

    National vulnerability database (nvd)

    National Institute of Standards and Technology (NIST). National vulnerability database (nvd). https://nvd. nist.gov/, 2023. Accessed: 2023-10-05

    2023

  40. [48]

    Ollama framework

    Ollama. Ollama framework. https://ollama.com/,

  41. [49]

    OpenAI. Gpt-4o. https://platform.openai.com/ docs/models#gpt-4

  42. [50]

    Chatgpt retrieval plugin, 2024

    OpenAI. Chatgpt retrieval plugin, 2024

    2024

  43. [51]

    Openai api reference - chat create n, 2024

    OpenAI. Openai api reference - chat create n, 2024. Accessed: 2024-05-20

    2024

  44. [52]

    Openai models - embeddings

    OpenAI. Openai models - embeddings. https://platform.openai.com/docs/models/ gpt#embeddings, 2024. Accessed: 2024-10-01

    2024

  45. [53]

    An empirical study of the non-determinism of chatgpt in code generation

    Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. An empirical study of the non-determinism of chatgpt in code generation. ACM Transactions on Soft- ware Engineering and Methodology, 2024

    2024

  46. [54]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguis- tics, pages 311–318, 2002

    2002

  47. [55]

    Re- trieval augmented code generation and summarization

    Md Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Re- trieval augmented code generation and summarization. arXiv preprint arXiv:2108.11601, 2021

    2021 arXiv

  48. [56]

    Asleep at the key- board? assessing the security of github copilot’s code contributions

    Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Bren- dan Dolan-Gavitt, and Ramesh Karri. Asleep at the key- board? assessing the security of github copilot’s code contributions. In 2022 IEEE Symposium on Security and Privacy (SP), pages 754–768. IEEE, 2022

    2022

  49. [57]

    From prompt injections to sql injection attacks: How protected is your llm-integrated web application? arXiv preprint arXiv:2308.01990, 2023

    Rodrigo Pedro, Daniel Castro, Paulo Carreira, and Nuno Santos. From prompt injections to sql injection attacks: How protected is your llm-integrated web application? arXiv preprint arXiv:2308.01990, 2023

    2023 arXiv

  50. [58]

    Ignore previous prompt: Attack techniques for language models

    Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527, 2022

    2022 arXiv

  51. [59]

    Evaluating and optimiz- ing the effectiveness of neural machine translation in supporting code retrieval models: A study on the cat benchmark

    Hung Phan and Ali Jannesari. Evaluating and optimiz- ing the effectiveness of neural machine translation in supporting code retrieval models: A study on the cat benchmark. In Proceedings of the 32nd ACM Inter- national Conference on Information and Knowledge Management, pages ...

    2023

  52. [60]

    Visual adversarial examples jailbreak aligned large language models

    Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak aligned large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 21527–21536, 2024. 16

    2024

  53. [61]

    Rag-fusion: a new take on retrieval-augmented generation

    Zackary Rackauckas. Rag-fusion: a new take on retrieval-augmented generation. arXiv preprint arXiv:2402.03367, 2024

    2024 arXiv

  54. [62]

    The proba- bilistic relevance framework: Bm25 and beyond

    Stephen Robertson, Hugo Zaragoza, et al. The proba- bilistic relevance framework: Bm25 and beyond. Foun- dations and Trends® in Information Retrieval, 3(4):333– 389, 2009

    2009

  55. [63]

    Code llama: Open foundation models for code

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  56. [64]

    Llm safety leaderboard

    Secure Learning Lab. Llm safety leaderboard. https://huggingface.co/spaces/AI-Secure/ llm-trustworthy-leaderboard, 2024. Accessed: 2024-10-01

    2024

  57. [65]

    Poison frogs! targeted clean-label poisoning at- tacks on neural networks

    Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Gold- stein. Poison frogs! targeted clean-label poisoning at- tacks on neural networks. Advances in neural informa- tion processing systems, 31, 2018

    2018

  58. [66]

    A systematic literature review on automated soft- ware vulnerability detection using machine learning

    Nima Shiri Harzevili, Alvine Boaye Belle, Junjie Wang, Song Wang, Zhen Ming Jiang, and Nachiappan Nagap- pan. A systematic literature review on automated soft- ware vulnerability detection using machine learning. ACM Computing Surveys, 57(3):1–36, 2024

    2024

  59. [67]

    A comprehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities

    Yisheng Song, Ting Wang, Puyu Cai, Subrota K Mondal, and Jyoti Prakash Sahoo. A comprehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities. ACM Computing Surveys, 55(13s):1– 40, 2023

    2023

  60. [68]

    A statistical interpretation of term specificity and its application in retrieval

    Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11–21, 1972

    1972

  61. [69]

    Efficient avoidance of vulnerabilities in auto-completed smart contract code using vulnerability-constrained decoding

    André Storhaug, Jingyue Li, and Tianyuan Hu. Efficient avoidance of vulnerabilities in auto-completed smart contract code using vulnerability-constrained decoding. In 2023 IEEE 34th International Symposium on Soft- ware Reliability Engineering (ISSRE), pages 683–693. IEEE, 2023

    2023

  62. [70]

    jina-embeddings-v3: Multilingual embed- dings with task lora

    Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, et al. jina-embeddings-v3: Multilingual embed- dings with task lora. arXiv preprint arXiv:2409.10173, 2024

    2024 arXiv

  63. [71]

    Evor: Evolv- ing retrieval for code generation

    Hongjin Su, Shuyang Jiang, Yuhang Lai, Haoyuan Wu, Boao Shi, Che Liu, Qian Liu, and Tao Yu. Evor: Evolv- ing retrieval for code generation. In Findings of the As- sociation for Computational Linguistics: EMNLP 2024, pages 2538–2554, 2024

    2024

  64. [72]

    Integration k-means clustering method and elbow method for identification of the best customer profile cluster

    Muhammad Ali Syakur, B Khusnul Khotimah, EMS Rochman, and Budi Dwi Satoto. Integration k-means clustering method and elbow method for identification of the best customer profile cluster. In IOP conference series: materials science and engineering, volume 336, page 012017. IOP ...

    2018

  65. [73]

    Small models, big in- sights: Leveraging slim proxy models to decide when and what to retrieve for LLMs

    Jiejun Tan, Zhicheng Dou, Yutao Zhu, Peidong Guo, Kun Fang, and Ji-Rong Wen. Small models, big in- sights: Leveraging slim proxy models to decide when and what to retrieve for LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeti...

    2024

  66. [74]

    How secure is ai-generated code: a large-scale comparison of large lan- guage models

    Norbert Tihanyi, Tamas Bisztray, Mohamed Amine Fer- rag, Ridhi Jain, and Lucas C Cordeiro. How secure is ai-generated code: a large-scale comparison of large lan- guage models. Empirical Software Engineering, 30(2):1– 42, 2025

    2025

  67. [75]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  68. [76]

    Query2doc: Query expansion with large language models

    Liang Wang, Nan Yang, and Furu Wei. Query2doc: Query expansion with large language models. arXiv preprint arXiv:2303.07678, 2023

    2023 arXiv

  69. [77]

    Fusing code searchers

    Shangwen Wang, Mingyang Geng, Bo Lin, Zhensu Sun, Ming Wen, Yepang Liu, Li Li, Tegawendé F Bissyandé, and Xiaoguang Mao. Fusing code searchers. IEEE Transactions on Software Engineering, 2024

    2024

  70. [78]

    Benchmark self-evolving: A multi-agent framework for dynamic llm evaluation

    Siyuan Wang, Zhuohan Long, Zhihao Fan, Zhongyu Wei, and Xuanjing Huang. Benchmark self-evolving: A multi-agent framework for dynamic llm evaluation. arXiv preprint arXiv:2402.11443, 2024

    2024 arXiv

  71. [79]

    Rap-gen: Retrieval-augmented patch generation with codet5 for automatic program repair

    Weishi Wang, Yue Wang, Shafiq Joty, and Steven CH Hoi. Rap-gen: Retrieval-augmented patch generation with codet5 for automatic program repair. In Proceed- ings of the 31st ACM Joint European Software Engineer- ing Conference and Symposium on the Foundations of Software Enginee...

    2023

  72. [80]

    Reposvul: A repository- level high-quality vulnerability dataset

    Xinchen Wang, Ruida Hu, Cuiyun Gao, Xin-Cheng Wen, Yujia Chen, and Qing Liao. Reposvul: A repository- level high-quality vulnerability dataset. In Proceed- ings of the 2024 IEEE/ACM 46th International Confer- ence on Software Engineering: Companion Proceedings, pages 472–483, 2024

    2024

  73. [81]

    Generalizing from a few examples: A survey on few-shot learning

    Yaqing Wang, Quanming Yao, James T Kwok, and Li- onel M Ni. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1–34, 2020

    2020

  74. [82]

    Coderag-bench: Can retrieval augment code gen- eration? arXiv preprint arXiv:2406.14497, 2024

    Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F Xu, Yiqing Xie, Graham Neubig, and Daniel Fried. Coderag-bench: Can retrieval augment code gen- eration? arXiv preprint arXiv:2406.14497, 2024

    2024 arXiv

  75. [83]

    Mindmap: Knowledge graph prompting sparks graph of thoughts in large language models

    Yilin Wen, Zifeng Wang, and Jimeng Sun. Mindmap: Knowledge graph prompting sparks graph of thoughts in large language models. arXiv preprint arXiv:2308.09729, 2023

    2023 arXiv

  76. [84]

    Syntax-aware retrieval augmented code genera- tion

    Xiangyu Zhang, Yu Zhou, Guang Yang, and Taolue Chen. Syntax-aware retrieval augmented code genera- tion. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1291–1302, 2023

    2023

  77. [85]

    Hijack- rag: Hijacking attacks against retrieval-augmented large language models

    Yucheng Zhang, Qinfeng Li, Tianyu Du, Xuhong Zhang, Xinkui Zhao, Zhengwen Feng, and Jianwei Yin. Hijack- rag: Hijacking attacks against retrieval-augmented large language models. arXiv preprint arXiv:2410.22832 , 2024

    2024 arXiv

  78. [86]

    A survey of large lan- guage models

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xi- aolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large lan- guage models. arXiv preprint arXiv:2303.18223, 2023

    2023 arXiv

  79. [87]

    Judging llm-as- a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuo- han Li, Dacheng Li, Eric Xing, et al. Judging llm-as- a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595– 46623, 2023

    2023

  80. [88]

    D2a: A dataset built for ai-based vulnerability detection methods using differential analy- sis

    Yunhui Zheng, Saurabh Pujar, Burn Lewis, Luca Bu- ratti, Edward Epstein, Bo Yang, Jim Laredo, Alessandro Morari, and Zhong Su. D2a: A dataset built for ai-based vulnerability detection methods using differential analy- sis. In 2021 IEEE/ACM 43rd International Conference on Sof...

    2021

  81. [89]

    Deepseek-coder-v2: Breaking the bar- rier of closed-source models in code intelligence

    Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coder-v2: Breaking the bar- rier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931, 2024

    2024 arXiv

  82. [90]

    Open-source large language models are strong zero-shot query likelihood models for document ranking

    Shengyao Zhuang, Bing Liu, Bevan Koopman, and Guido Zuccon. Open-source large language models are strong zero-shot query likelihood models for document ranking. arXiv preprint arXiv:2310.13243, 2023

    2023 arXiv

  83. [91]

    Universal and trans- ferable adversarial attacks on aligned language models

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and trans- ferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023

    2023 arXiv

  84. [92]

    Poisonedrag: Knowledge poisoning attacks to retrieval-augmented generation of large language mod- els

    Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. Poisonedrag: Knowledge poisoning attacks to retrieval-augmented generation of large language mod- els. arXiv preprint arXiv:2402.07867, 2024. 18 A Prompt for Query Generation Prompt 1: Query Generation You are an expert {LA...

    2024 arXiv

  85. [2023]

    Accessed: 2024-10-01

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.