Pith. sign in

REVIEW 5 cited by

SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2506.05692 v3 pith:WD2SZFTU submitted 2025-06-06 cs.CR cs.AI

Xinghang Li , Jingzhe Ding , Chao Peng , Bing Zhao , Xiang Gao , Hongwan Gao , Xinchen Gu This is my paper
classification cs.CRcs.AI
keywords codesecuritybenchmarkllmssafegenbenchassessevaluationframework
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The code generation capabilities of large language models(LLMs) have emerged as a critical dimension in evaluating their overall performance. However, prior research has largely overlooked the security risks inherent in the generated code. In this work, we introduce SafeGenBench, a benchmark specifically designed to assess the security of LLM-generated code. The dataset encompasses a wide range of common software development scenarios and vulnerability types. Building upon this benchmark, we develop an automatic evaluation framework that leverages both static application security testing(SAST) and LLM-based judging to assess the presence of security vulnerabilities in model-generated code. Through the empirical evaluation of state-of-the-art LLMs on SafeGenBench, we reveal notable deficiencies in their ability to produce vulnerability-free code. Our findings highlight pressing challenges and offer actionable insights for future advancements in the secure code generation performance of LLMs. The data and code will be released soon.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Illusion of Safety: Multi-Tier Verification of AI vs. Human C++ Code

    cs.SE 2026-06 unverdicted novelty 7.0 of 10

    On 8,918 competitive-programming C++ programs, AI code roughly doubles the odds of sanitizer-confirmed runtime violations versus human code; static analysis alone masks the gap.

  2. The Patchwork Problem in LLM-Generated Code

    cs.SE 2026-07 conditional novelty 6.5 of 10

    Structural failures in LLM-generated code form a distinct class of defects that standard type checking, tests, and SAST miss, and models produce qualitatively different failure profiles.

  3. Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios

    cs.CR 2026-07 conditional novelty 6.0 of 10

    In realistic ambiguous, underspecified, or conflict-laden prompts, all eight tested LLMs produced vulnerable code in over 56% of outputs, and security-aware prompting reduced vulnerability by up to 45%.

  4. Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluation

    cs.AI 2026-08 conditional novelty 5.0 of 10

    A multi-agent adversarial evaluation platform with six progressive attack strategies shows that role-playing LLMs degrade under sustained pressure, with automated judging correlating with human ratings.

  5. Extracting Recurring Vulnerabilities from Black-Box LLM-Generated Software

    cs.CR 2026-02 reject novelty 5.0 of 10

    Frontend features of LLM-generated apps can predict hidden backend vulnerabilities that a given model tends to reproduce, enabling black-box attack triage.

Pith tools