REVIEW 14 cited by
SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The code generation capabilities of large language models(LLMs) have emerged as a critical dimension in evaluating their overall performance. However, prior research has largely overlooked the security risks inherent in the generated code. In this work, we introduce SafeGenBench, a benchmark specifically designed to assess the security of LLM-generated code. The dataset encompasses a wide range of common software development scenarios and vulnerability types. Building upon this benchmark, we develop an automatic evaluation framework that leverages both static application security testing(SAST) and LLM-based judging to assess the presence of security vulnerabilities in model-generated code. Through the empirical evaluation of state-of-the-art LLMs on SafeGenBench, we reveal notable deficiencies in their ability to produce vulnerability-free code. Our findings highlight pressing challenges and offer actionable insights for future advancements in the secure code generation performance of LLMs. The data and code will be released soon.
Forward citations
Cited by 14 Pith papers
-
The Illusion of Safety: Multi-Tier Verification of AI vs. Human C++ Code
Multi-tier verification on VULBENCH-CPP shows AI-generated C++ code triggers confirmed runtime violations roughly twice as often as human code, while static analysis misleadingly indicates parity due to code length.
-
An Empirical Study of Security Calibration in Large Language Models for Code
Empirical evaluation of three LLMs finds prevalent overconfidence in insecure code generation, with security calibration outperforming functional calibration but both degrading in repository-level settings.
-
FinRED: An Expert-Guided Benchmark Generation and Evaluation Framework for Financial LLM Red-Teaming
FinRED creates an expert-validated benchmark and rubric for financial LLM safety that maps regulatory standards to specific threats and reduces critical false negatives in evaluation from 28 to 12.
-
ASSEMBLAGE-DEEPHISTORY: A Cross-Build Binary Dataset with Temporal Coverage
A new queryable binary dataset combining cross-build diversity, temporal history, and CVE labels with linked metadata for vulnerability research.
-
Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning
LLMVD.js uses LLM agents to confirm 84% of taint-style vulnerabilities on public benchmarks (vs. <22% for prior tools) and generates validated exploits for 36 of 260 new packages (vs. ≤2 for traditional tools).
-
The Patchwork Problem in LLM-Generated Code
Structural failures in LLM-generated code form a distinct class of defects that standard type checking, tests, and SAST miss, and models produce qualitatively different failure profiles.
-
Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios
In realistic ambiguous, underspecified, or conflict-laden prompts, all eight tested LLMs produced vulnerable code in over 56% of outputs, and security-aware prompting reduced vulnerability by up to 45%.
-
The Illusion of Safety: Multi-Tier Verification of AI vs. Human C++ Code
On 8,918 competitive-programming C++ programs, AI code roughly doubles the odds of sanitizer-confirmed runtime violations versus human code; static analysis alone masks the gap.
-
The Illusion of Safety: Multi-Tier Verification of AI vs. Human C++ Code
AI-generated C++ programs trigger confirmed runtime (sanitizer) violations at roughly twice the odds of human contest solutions, a gap static analysis does not reveal.
-
False Security Confidence in Benign LLM Code Generation
False Security Confidence (FSC) rate is defined as the prevalence of security failures inside the set of functionally correct LLM code outputs, with a three-ecosystem task view and FSC-hard category proposed for futur...
-
Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluation
A multi-agent adversarial evaluation platform with six progressive attack strategies shows that role-playing LLMs degrade under sustained pressure, with automated judging correlating with human ratings.
-
CoT-Guard: Small Models for Strong Monitoring
CoT-Guard is a 4B model using SFT and RL that achieves 75% G-mean^2 on hidden objective detection under prompt and code manipulation attacks, outperforming several larger models.
-
On Fixing Insecure AI-Generated Code through Model Fine-Tuning and Prompting Strategies
Fine-tuning and prompting reduce some CWEs in AI-generated code but frequently introduce new weaknesses, with no strategy working reliably across models or languages.
-
Extracting Recurring Vulnerabilities from Black-Box LLM-Generated Software
Frontend features of LLM-generated apps can predict hidden backend vulnerabilities that a given model tends to reproduce, enabling black-box attack triage.
Discussion (0). Sign in to comment.