pith. machine review for the scientific record. sign in

arxiv: 2509.22097 · v4 · submitted 2025-09-26 · 💻 cs.SE · cs.AI· cs.CL· cs.CR

Recognition: unknown

SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios

Junkai Chen , Huihui Huang , Yunbo Lyu , Junwen An , Jieke Shi , Chengran Yang , Ting Zhang , Haoye Tian
show 5 more authors
Yikun Li Zhenhao Li Xin Zhou Xing Hu David Lo
Authors on Pith no claims yet
classification 💻 cs.SE cs.AIcs.CLcs.CR
keywords agentscodesecurevibebenchsecurecodingcorrectlargescenarios
0
0 comments X
read the original abstract

Large language model-powered code agents are rapidly transforming software engineering, yet the security risks of their generated code have become a critical concern. Existing benchmarks have provided valuable insights, but they fail to capture scenarios in which vulnerabilities are actually introduced by human developers, making fair comparisons between humans and agents infeasible. We therefore introduce SecureVibeBench, a benchmark of 105 C/C++ secure coding tasks sourced from 41 projects in OSS-Fuzz for code agents. SecureVibeBench has the following features: (i) realistic task settings that require multi-file edits in large repositories, (ii)~aligned contexts based on real-world open-source vulnerabilities with precisely identified vulnerability introduction points, and (iii) comprehensive evaluation that combines functionality testing and security checking with both static and dynamic oracles. We evaluate 5 popular code agents like OpenHands, supported by 5 LLMs (e.g., Claude sonnet 4.5) on SecureVibeBench. Results show that current agents struggle to produce both correct and secure code, as even the best-performing one, produces merely 23.8\% correct and secure solutions on SecureVibeBench. Our code and data are on https://github.com/iCSawyer/SecureVibeBench.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?

    cs.CR 2026-05 conditional novelty 7.0

    ExploitGym benchmark shows frontier AI models can generate working exploits for 120-157 of 898 real vulnerabilities, with non-trivial success even when common security defenses are enabled.

  2. AgentSZZ: Teaching the LLM Agent to Play Detective with Bug-Inducing Commits

    cs.SE 2026-04 conditional novelty 7.0

    AgentSZZ is an LLM-agent framework that identifies bug-inducing commits with up to 27.2% higher F1 scores than prior methods by enabling adaptive exploration and causal tracing, especially for cross-file and ghost commits.

  3. Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMs

    cs.SE 2026-04 unverdicted novelty 6.0

    A taxonomy and benchmark for logging security issues shows LLMs achieve 13-53% detection accuracy but struggle to produce correct repairs, with issue descriptions helping more than pattern explanations.