REVIEW 32 cited by
CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large language models (LLMs) introduce new security risks, but there are few comprehensive evaluation suites to measure and reduce these risks. We present BenchmarkName, a novel benchmark to quantify LLM security risks and capabilities. We introduce two new areas for testing: prompt injection and code interpreter abuse. We evaluated multiple state-of-the-art (SOTA) LLMs, including GPT-4, Mistral, Meta Llama 3 70B-Instruct, and Code Llama. Our results show that conditioning away risk of attack remains an unsolved problem; for example, all tested models showed between 26% and 41% successful prompt injection tests. We further introduce the safety-utility tradeoff: conditioning an LLM to reject unsafe prompts can cause the LLM to falsely reject answering benign prompts, which lowers utility. We propose quantifying this tradeoff using False Refusal Rate (FRR). As an illustration, we introduce a novel test set to quantify FRR for cyberattack helpfulness risk. We find many LLMs able to successfully comply with "borderline" benign requests while still rejecting most unsafe requests. Finally, we quantify the utility of LLMs for automating a core cybersecurity task, that of exploiting software vulnerabilities. This is important because the offensive capabilities of LLMs are of intense interest; we quantify this by creating novel test sets for four representative problems. We find that models with coding capabilities perform better than those without, but that further work is needed for LLMs to become proficient at exploit generation. Our code is open source and can be used to evaluate other LLMs.
Forward citations
Cited by 32 Pith papers
-
Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)
Training a 7B LLM planner with reinforcement learning on verifiable attack rewards lets it generate policies that reduce DRL defender scores by an average of 522% versus static red agents.
-
ACSE-Eval: Can LLMs threat model real-world cloud infrastructure?
ACSE-Eval benchmarks LLMs on threat-modeling 100 AWS architectures and finds that GPT-4.1 and Gemini 2.5 Pro lead threat identification, while all models score below 50% on exact CWE and ATT&CK classification.
-
Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interface
Attackers can use the loss signal from a remote LLM fine-tuning API to optimize adversarial prefix and suffix tokens, turning existing prompt injections into high-success attacks on closed-weight Gemini models.
-
SecDrift: Measuring Sector-Conditioned Security Drift in AI-Generated Code
Sector-conditioned prompt framing does not significantly change static-analysis vulnerability rates in AI-generated code; the apparent effect is a composition artifact of two CWE categories.
-
Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction
Tencent WorkBuddy Bench is a 260-task, four-domain coding-agent benchmark whose colloquial prompts are reverse-engineered from real commits, CVEs, and business scenarios, released openly with a dual-harness cross-mode...
-
Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents
Security-agent success changes differently with budget: offensive CTF tasks improve with more compute, while defensive SOC work depends more on tool discipline than spend.
-
Beyond Refusal: A Same-Lineage Study of Aligned and Abliterated LLMs for Vulnerability Analysis
Refusal-ablated LLMs outperform aligned models on code-grounded localization and early executable patch generation, while aligned models retain advantages on shallow diagnostic tasks under neutral wording.
-
Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges
Per-bias selection of a cross-family LLM auditor lifts biased-judgment accuracy from 0.805/0.824 baselines to 0.884.
-
AI Security Policy Should Assess Systems, Not Only Models
Coordinated swarms of small open LLMs achieve frontier-model jailbreaks and full vulnerability recovery at zero cost, demonstrating that system scaffolds enable capabilities previously thought to require restricted la...
-
Defending Against Prompt Injection With a Few DefensiveTokens
Five optimized token embeddings prepended to the prompt reduce prompt-injection attack success to near zero on standard benchmarks while preserving most model utility.
-
Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation
Across two Python security benchmarks, all eight LLMs produced vulnerable code at rates from 9.8% to 42.1%, and self-generated hints plus explained feedback reduced vulnerabilities mainly for the strongest models.
-
Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges
A benchmark and agent for CTF solving, but the agent's retrieval database appears to contain the answers to the test challenges, undermining the reported improvements.
-
SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis
Large language models perform poorly on a new C-code vulnerability benchmark, indicating they rely on pattern matching rather than genuine reasoning.
-
Evaluating AI cyber capabilities with crowdsourced elicitation
Crowdsourced AI teams solved nearly all challenges in a small CTF and beat 90% of registered human teams in a large one, pointing to open-market elicitation as a viable evaluation tool.
-
Sparse Activation Editing for Reliable Instruction Following in Narratives
An unsupervised SAE-based method that localizes and adjusts instruction-relevant neurons improves instruction adherence and reduces refusals on a new 1,212-example narrative benchmark.
-
CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution
CRAKEN, an LLM agent combining Self-RAG and Graph-RAG over a CTF writeup database, solves 22% of NYU CTF Bench challenges, three percentage points above the prior D-CIPHER baseline.
-
LLMSecConfig: An LLM-Based Approach for Fixing Software Container Misconfigurations
LLMSecConfig couples Checkov static analysis with retrieval-augmented LLM prompting to repair Kubernetes security misconfigurations, reaching a 94.3% pass rate with Mistral Large 2.
-
CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation
CWEval is a new benchmark that simultaneously checks functional correctness and security of AI-generated code with dynamic test oracles, exposing a large correct-but-insecure gap in current LLMs.
-
Rethinking CyberSecEval: An LLM-Aided Approach to Evaluation Critique
CyberSecEval's insecure-code tests partly measure refusal to follow bad instructions and susceptibility to leading comments, not natural insecure code generation.
-
Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense
OSB proposes frozen synthetic-enterprise snapshots with gold posture answers so AI agents can be benchmarked on security investigation via SQL or native vendor APIs.
-
Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Response
A structured review organizes cyber-capable-agent risks into five vulnerability classes and argues that evaluation environments must be treated as operational security systems rather than background.
-
Running in CIRCLE? A Simple Benchmark for LLM Code Interpreter Security
CIRCLE is a 1,260-prompt benchmark that measures how often commercial LLM code interpreters refuse, execute, or time out on resource-exhaustion tasks, revealing large and inconsistent safety gaps.
-
SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code
A 558-task benchmark with SAST and LLM judging shows state-of-the-art LLMs generate vulnerability-free code only about 37% of the time in zero-shot settings.
-
Detection Method for Prompt Injection by Integrating Pre-trained Model and Heuristic Feature Engineering
A dual-channel detector combining DeBERTa and heuristic rules is claimed to beat existing prompt injection detectors on three benchmarks and to reduce attack success on GLM-4, Llama 3, Qwen 2.5, and GPT-4o.
-
MEQA: A Meta-Evaluation Framework for Question & Answer LLM Benchmarks
MEQA scores eight cybersecurity QA benchmarks against a 44-sub-criteria rubric, finding strengths in reproducibility and comparability and weaknesses in prompt robustness and reliability.
-
Secure Code Generation at Scale with Reflexion
Three-round reflexion prompting raises LLM-generated code security from 70.74% to 79.43% as measured by the ICD static detector on the Instruct Prime benchmark.
-
Forewarned is Forearmed: A Survey on Large Language Model-based Agents in Autonomous Cyberattacks
A review of LLM-based agents as autonomous cyberattackers, arguing that they lower attack costs, scale up threats, and outpace existing defenses.
-
LLM Cyber Evaluations Don't Capture Real-World Risk
The paper argues and demonstrates with a 100-prompt case study that LLM cyber risk evaluations need to include threat actor adoption and impact, not just model capability.
-
What AI evaluations for preventing catastrophic risks can and cannot do
AI evaluations can establish lower bounds on capabilities but cannot establish upper bounds, forecast future capabilities robustly, or assess misalignment risk, so they should not be the primary basis for AI safety decisions.
-
Logic layer Prompt Control Injection (LPCI): A Novel Security Vulnerability Class in Agentic Systems
LPCI is presented as a new vulnerability class for agentic LLMs, but prior work already covers memory-based prompt injection and the paper's own numbers contain errors.
-
Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
A broad survey of LLM alignment that catalogs objectives, benchmarks, SFT/RLHF/DPO methods, and safety challenges, without contributing new experimental or theoretical results.
-
A Contemporary Survey of Large Language Model Assisted Program Analysis
A review that catalogs how large language models are used in static, dynamic, and hybrid program analysis, and outlines open challenges.
Discussion (0). Continue with ORCID to comment.