Pith. sign in

REVIEW 20 cited by

Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.11147 v3 pith:5LBZH3EQ submitted 2024-06-17 cs.SE cs.AI

Xueying Du , Geng Zheng , Kaixin Wang , Yi Zou , Yujia Wang , Wentai Deng , Jiayi Feng , Mingwei Liu
show 4 more authors
Bihuan Chen Xin Peng Tao Ma Yiling Lou
This is my paper · ORCID
classification cs.SEcs.AI
keywords vulnerabilitydetectionllmsaccuracyvul-ragcodeenhancingknowledge
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Although LLMs have shown promising potential in vulnerability detection, this study reveals their limitations in distinguishing between vulnerable and similar-but-benign patched code (only 0.06 - 0.14 accuracy). It shows that LLMs struggle to capture the root causes of vulnerabilities during vulnerability detection. To address this challenge, we propose enhancing LLMs with multi-dimensional vulnerability knowledge distilled from historical vulnerabilities and fixes. We design a novel knowledge-level Retrieval-Augmented Generation framework Vul-RAG, which improves LLMs with an accuracy increase of 16% - 24% in identifying vulnerable and patched code. Additionally, vulnerability knowledge generated by Vul-RAG can further (1) serve as high-quality explanations to improve manual detection accuracy (from 60% to 77%), and (2) detect 10 previously-unknown bugs in the recent Linux kernel release with 6 assigned CVEs.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 20 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DREA: Decoupled Reasoning and Exploration Agents for Repository-Level Vulnerability Detection

    cs.CR 2026-07 conditional novelty 6.0 of 10

    DREA improves repository-level vulnerability detection by coupling an LLM planner that forms hypotheses with a cheap local explorer that gathers cross-file evidence, lifting paired accuracy from 19-26% to 30-42% at mu...

  2. Knowdit: Agentic Smart Contract Vulnerability Detection with Auditing Knowledge Summarization

    cs.CR 2026-03 conditional novelty 6.0 of 10

    Knowdit links abstract DeFi semantics to vulnerability patterns in a knowledge graph and drives an agentic specify–harness–fuzz–reflect loop that finds all high-severity and most medium-severity bugs on held-out Code4...

  3. Multi-Modal Requirements Data-based Acceptance Criteria Generation using LLMs

    cs.SE 2025-08 conditional novelty 6.0 of 10

    Multi-modal RAG (text plus UI screenshots) with reward-based polishing generates acceptance criteria from user stories that three industry experts rated near 4/5 on relevance, correctness, and understandability.

  4. Improving LLM Reasoning for Vulnerability Detection via Group Relative Policy Optimization

    cs.CR 2025-07 conditional novelty 6.0 of 10

    GRPO fine-tuning with a dynamic formatting, reasoning, and correctness reward helps small LLMs detect vulnerabilities more accurately and reason more coherently than supervised fine-tuning.

  5. iPanda: An LLM-based Agent for Automated Conformance Testing of Communication Protocols

    cs.SE 2025-07 conditional novelty 6.0 of 10

    An LLM-based agent that parses protocol RFCs into test cases, synthesizes executable test programs with RAG and chain-of-thought, and iteratively debugs them, reports 4.7x to 10.8x Pass@1 improvements over a pure-LLM ...

  6. LLM Embedding-based Attribution (LEA): Quantifying Source Contributions to Generative Model's Response for Vulnerability Analysis

    cs.CR 2025-06 reject novelty 6.0 of 10

    LEA uses rank-based linear dependence of layer-0 hidden states to attribute each response token to query, retrieved context, or internal knowledge, and distinguishes valid from generic retrieval with over 95% accuracy.

  7. Minimizing False Positives in Static Bug Detection via LLM-Enhanced Path Feasibility Analysis

    cs.SE 2025-06 conditional novelty 6.0 of 10

    LLM4PFA combines iterative LLM-based constraint extraction with Z3 solving to reduce false positives in static analysis, achieving 72-96% false positive reduction and 93% recall on a new benchmark.

  8. CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution

    cs.CR 2025-05 conditional novelty 6.0 of 10

    CRAKEN, an LLM agent combining Self-RAG and Graph-RAG over a CTF writeup database, solves 22% of NYU CTF Bench challenges, three percentage points above the prior D-CIPHER baseline.

  9. Everything You Wanted to Know About LLM-based Vulnerability Detection But Were Afraid to Ask

    cs.CR 2025-04 reject novelty 6.0 of 10

    With context-rich prompts that include CWE hints and marked potential vulnerability sites, LLM detectors beat random baselines, but the evaluation may leak the answer.

  10. Exploring the Security Threats of Knowledge Base Poisoning in Retrieval-Augmented Code Generation

    cs.CR 2025-02 conditional novelty 6.0 of 10

    Poisoning the knowledge base of a retrieval-augmented code generator with vulnerable snippets raises the vulnerability rate of the model's output, with the size of the rise depending on the retriever, the model, and t...

  11. Code Change Intention, Development Artifact and History Vulnerability: Putting Them Together for Vulnerability Fix Detection by LLM

    cs.SE 2025-01 conditional novelty 6.0 of 10

    LLM4VFD combines LLM-generated summaries of commit intent, issue/PR context, and retrieved historical fixes to detect vulnerability-fixing commits, reporting F1 gains of 68.1% to 145.4% over prior PLM baselines.

  12. Enhancing Semantic Understanding in Pointer Analysis using Large Language Models

    cs.SE 2025-08 conditional novelty 5.0 of 10

    LMPA is a proposed, unevaluated framework for using LLMs to improve pointer analysis through API abstraction, parameter inference, and natural-language summaries.

  13. Identifying Helpful Context for LLM-based Vulnerability Repair: A Preliminary Study

    cs.SE 2025-06 conditional novelty 5.0 of 10

    Using CVE descriptions and manually selected code context in prompts, and combining the best prompts, GPT-4o fixed 26 of 42 Java vulnerabilities at least once, up from 19 with its baseline prompt.

  14. Detecting Malicious Source Code in PyPI Packages with LLMs: Does RAG Come in Handy?

    cs.SE 2025-04 reject novelty 5.0 of 10

    Retrieval-augmented generation with YARA rules, GitHub advisories, and malicious code snippets failed to improve LLM detection of malicious PyPI packages, while fine-tuned LLaMA-3.1-8B reached 97% accuracy.

  15. Large Language Models for Security Operations Centers: A Comprehensive Survey

    cs.CR 2025-09 conditional novelty 4.0 of 10

    A systematic review of 138 papers classifying LLM applications in SOC workflows by phase, model family, datasets, and maturity.

  16. WALL: A Web Application for Automated Quality Assurance using Large Language Models

    cs.SE 2025-09 reject novelty 4.0 of 10

    WALL combines SonarQube with GPT-3.5 Turbo and GPT-4o to automatically fix detected code issues, reporting 71.6% and 85.5% revision rates plus cost reductions from a hybrid workflow.

  17. VulBinLLM: LLM-powered Vulnerability Detection for Stripped Binaries

    cs.CR 2025-05 reject novelty 4.0 of 10

    An LLM pipeline that enriches decompiled code with vulnerability hints and then classifies CWEs reports high Juliet accuracy, but leaky hints and unmatched baselines weaken the claim.

  18. Forewarned is Forearmed: A Survey on Large Language Model-based Agents in Autonomous Cyberattacks

    cs.NI 2025-05 conditional novelty 4.0 of 10

    A review of LLM-based agents as autonomous cyberattackers, arguing that they lower attack costs, scale up threats, and outpace existing defenses.

  19. Context-Enhanced Vulnerability Detection Based on Large Language Model

    cs.SE 2025-04 reject novelty 4.0 of 10

    PacVD adds hand-built summaries of security-sensitive API calls to LLM prompts and reports F1 improvements of a few points, though the gains are not validated against a majority-class baseline.

  20. LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights

    cs.CR 2025-02 conditional novelty 4.0 of 10

    A survey of LLM-based vulnerability detection covering 58 papers, with a taxonomy, dataset overview, and gap analysis, but limited by non-transparent selection and unsupported quantitative claims.

Pith tools