REVIEW 21 cited by
LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoning
read the original abstract
Large language models (LLMs) have demonstrated significant potential in various tasks, including those requiring human-level intelligence, such as vulnerability detection. However, recent efforts to use LLMs for vulnerability detection remain preliminary, as they lack a deep understanding of whether a subject LLM's vulnerability reasoning capability stems from the model itself or from external aids such as knowledge retrieval and tooling support. In this paper, we aim to decouple LLMs' vulnerability reasoning from other capabilities, such as vulnerability knowledge adoption, context information retrieval, and advanced prompt schemes. We introduce LLM4Vuln, a unified evaluation framework that separates and assesses LLMs' vulnerability reasoning capabilities and examines improvements when combined with other enhancements. To support this evaluation, we construct UniVul, the first benchmark that provides retrievable knowledge and context-supplementable code across three representative programming languages: Solidity, Java, and C/C++. Using LLM4Vuln and UniVul, we test six representative LLMs (GPT-4.1, Phi-3, Llama-3, o4-mini, DeepSeek-R1, and QwQ-32B) for 147 ground-truth vulnerabilities and 147 non-vulnerable cases in 3,528 controlled scenarios. Our findings reveal the varying impacts of knowledge enhancement, context supplementation, and prompt schemes. We also identify 14 zero-day vulnerabilities in four pilot bug bounty programs, resulting in $3,576 in bounties.
Forward citations
Cited by 21 Pith papers
-
QASecClaw: A Multi-Agent LLM Approach for False Positive Reduction in Static Application Security Testing
A multi-agent LLM system cuts false positives in static application security testing by 88.6% on the OWASP Benchmark while dropping recall by only 3.1%.
-
Geometric quantification for nonlinear deformation in knitted fabrics
A geometric quantification framework reconstructs yarn centerlines and fabric surfaces from sparse data to quantify how nonlinear deformation distributes among stitch reorientation, loop bending, surface bending, and ...
-
An End-to-End Approach for Fixing Concurrency Bugs via SHB-Based Context Extractor
ConFixAgent repairs diverse concurrency bugs end-to-end by using Static Happens-Before graphs to extract relevant code context for LLMs, outperforming prior tools in benchmarks.
-
Malaika: Understanding Malware through Tri-Grounded Agentic Reasoning
A tri-grounded multi-agent harness improves precision and auditability of Android malware behavior reports over fixed LLM pipelines and frontier coding agents.
-
ShadowProbe: Language-Extensible Detection of Hidden Algorithmic Complexity Vulnerabilities
ShadowProbe detects language-extensible algorithmic complexity vulnerabilities from hidden library costs via static screening, context recovery, LLM input synthesis, and measured runtime growth.
-
Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detection
LLMs for code vulnerability detection show average susceptibility of 33.2% to framing, 23.5% to anchoring, and 18.4% to halo effects, with a black-box attack suppressing up to 97% of detections.
-
Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software
A curated temporal-split benchmark shows LLMs achieve at most 52.1% vulnerability detection accuracy after fine-tuning because backbone directional biases resist correction and contamination provides no benefit.
-
On the Shoulders of Giants: Empowering Automated Smart Contract Auditing via the GiAnt Corpus
GiANT uses divide-and-conquer and Chain-of-Thought prompting on 388 Code4rena reports to produce a 7,711-finding vulnerability corpus validated at 4.76/5 quality by manual review.
-
Quality-Assured Fuzz Harness Generation via the Four Principles Framework
QuartetFuzz introduces the Four Principles framework for harness correctness and deploys an autonomous LLM agent that produces verified harnesses, yielding 29 confirmed bugs across 23 projects and identifying violatio...
-
FuzzingBrain V2: A Multi-Agent LLM System for Automated Vulnerability Discovery and Reproduction
FuzzingBrain V2, a multi-agent LLM system with a novel Suspicious Point abstraction and dual-layer fuzzing, reports 90% detection on a C/C++ benchmark and 29 confirmed zero-day vulnerabilities in real open-source projects.
-
Adversarial SQL Injection Generation with LLM-Based Architectures
RADAGAS-GPT4o achieves a 22.73% bypass rate against 10 WAFs, succeeding more against AI/ML-based firewalls than rule-based ones.
-
Comprehensive AI governance requires addressing non-model gains
Non-model gains via inference, systems, and assets can drive AI capabilities independently of base models, requiring governance beyond model-level evaluation and mitigation.
-
SAGE: Signal-Amplified Guided Embeddings for LLM-based Vulnerability Detection
SAGE uses sparse autoencoders to boost vulnerability signals in LLMs, raising internal SNR 12.7x and delivering up to 318% MCC gains on vulnerability detection benchmarks.
-
Do Fine-Tuned LLMs Understand Vulnerabilities? An Investigation into the Semantic Trap
Fine-tuned decoder-only LLMs fall into a Semantic Trap on vulnerability detection, achieving high scores on unpaired normal code but failing on paired vulnerable-patched code, semantic perturbations, and gap analysis,...
-
QuiLL: An LLM-Based Vulnerability Assessment Framework for the Wild
QuiLL is a new evaluation pipeline that uses optimized LLM prompts, dynamic in-context learning from an NVD vector store, and a novel accuracy-plus-reasoning metric to benchmark vulnerability detection in real code.
-
Decoupled Smart Contract Audits: Lightweight LLM Framework via Distillation and Aggregation
A decoupled four-stage LLM pipeline with rsLoRA, distillation, and CoVe aggregation outperforms larger models on smart contract vulnerability detection and explanation using only 0.6B-4B parameter models.
-
Geometric quantification for nonlinear deformation in knitted fabrics
A geometric quantification framework reconstructs yarn centerlines and fabric surfaces from sparse knit data and partitions large deformation into stitch reorientation, loop bending, surface bending, and dilation.
-
From Theory to Practice: Code Generation Using LLMs for CAPEC and CWE Frameworks
LLMs generated 615 vulnerable code snippets aligned with CAPEC and CWE frameworks across three languages, with 0.98 cosine similarity between model outputs.
-
Security Document Classification with a Fine-Tuned Local Large Language Model: Benchmark Data and an Open-Source System
A fine-tuned local Qwen 3.5 27B model achieves 95% category-level accuracy on security document classification, outperforming commercial models on both internal and external test sets while keeping processing local.
-
Large Language Model-Based Agents for Software Engineering: A Survey
A literature survey that collects and categorizes 124 papers on LLM-based agents for software engineering from SE and agent perspectives.
-
Revisiting Vul-RAG: Reproducibility and Replicability of RAG-based Vulnerability Detection with Open-Weight Models
Reproducibility study of Vul-RAG confirms original findings in a fully local open-weights setting but identifies a persistent performance plateau at approximately 0.30 pairwise accuracy across diverse recent open-weight LLMs.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.