Introduces the first open multi-host cyber range benchmark AgentCyberRange with Cage toolchain and evaluates six frontier AI systems on web exploitation and post-exploitation tasks across 110 vulnerabilities.
hub
Cy- berGym: Evaluating AI agents’ real-world cybersecurity capabilities at scale
17 Pith papers cite this work. Polarity classification is still indexing.
17
Pith papers citing it
hub tools
citation-role summary
dataset 2
background 1
citation-polarity summary
years
2026 17representative citing papers
Cybersecurity AI agents invalidate the CRA's premises about vulnerability discovery, exploitation, and remediation speed, making static conformity certificates false and requiring continuous agent-operated defense.
CyberChainBench is a new benchmark evaluating LLM agents on vulnerability detection, exploit generation, and patch synthesis using 541 real-world DeFi incidents with on-chain historical evaluation, showing peak performance of 43.7% on exploitation.
AgentBeats implements agentified evaluation of diverse AI agents through standardized interfaces, validated at scale in a five-month competition with 298 judges and 467 subjects plus a coding case study.
CyberEvolver introduces a four-layer self-evolving agent architecture with trace-to-diagnosis and population beam search that raises seed agent success rates by 13.6% on CTF, exploitation, and penetration tasks across four LLMs.
The TAB benchmark reveals that frontier terminal agents achieve high task completion but low selective alignment with relevant environmental cues over distractors, and prompt-injection defenses block both.
CrackMeBench introduces 20 deterministic binary validation tasks and reports GPT-5.5 solving 11/12 generated ones at pass@3 while Claude and Kimi lag, especially on harder tasks.
SETC framework provides the first systematic comparison of CIM, OCSF, and ECS logging standards by running 50 RCE exploits and measuring how well each captures attack indicators.
AuditBench is a new benchmark of audit logs from 50+ malicious and benign scenarios that evaluates five LLMs on four security investigation tasks and analyzes their performance and error profiles.
CyberGym-E2E turns 920 real OSS-Fuzz vulnerabilities into end-to-end agent tasks; results show frontier agents patch known bugs well (65.9% S3) but find the exact ground-truth vulnerability less often (22.2% S4).
AnyPoC validates LLM bug reports by synthesizing, executing, and independently re-checking proof-of-concept tests, finding 121 new bugs with 108 confirmed across large systems.
PAGENT integrates static and dynamic program analysis guidance with an LLM agent to improve automated proof-of-concept generation success by 132% over prior agentic methods.
Bounded performance metrics always favor convergence of AI capabilities to meek models while unbounded metrics allow frontier models to maintain leads indefinitely, with policy implications for capability concentration.
GLM-5 is a foundation model that claims state-of-the-art results on coding benchmarks and superior performance on end-to-end software engineering tasks via new asynchronous RL methods and cost-saving DSA.
Kimi K2.5 combines joint text-vision training with an Agent Swarm parallel orchestration framework to reach claimed state-of-the-art results on coding, vision, reasoning, and agent tasks while cutting latency up to 4.5 times.
The paper proposes the Cybersecurity AI Scientist as a modular multi-agent architecture for automating cybersecurity research, distinguished by its focus on non-stationary threats and anchored in a four-zeros risk-trust-incident-energy frame.
Agentic AI lowers the cost and speed of cyber attacks, requiring immediate improvements in identity management, phishing-resistant authentication, patching, and agent governance for large enterprises and the Mittelstand.
citing papers explorer
-
AgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber Ranges
Introduces the first open multi-host cyber range benchmark AgentCyberRange with Cage toolchain and evaluates six frontier AI systems on web exploitation and post-exploitation tasks across 110 vulnerabilities.
-
Certifying Ghosts: How Cybersecurity AI Agents Break the EU Cyber Resilience Act
Cybersecurity AI agents invalidate the CRA's premises about vulnerability discovery, exploitation, and remediation speed, making static conformity certificates false and requiring continuous agent-operated defense.
-
CyberChainBench: Can AI Agents Secure Smart Contracts Against Real-World On-Chain Vulnerabilities?
CyberChainBench is a new benchmark evaluating LLM agents on vulnerability detection, exploit generation, and patch synthesis using 541 real-world DeFi incidents with on-chain historical evaluation, showing peak performance of 43.7% on exploitation.
-
AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility
AgentBeats implements agentified evaluation of diverse AI agents through standardized interfaces, validated at scale in a five-month competition with 298 judges and 467 subjects plus a coding case study.
-
CyberEvolver: Structured Self-Evolution for Cybersecurity Agents On the Fly
CyberEvolver introduces a four-layer self-evolving agent architecture with trace-to-diagnosis and population beam search that raises seed agent success rates by 13.6% on CTF, exploitation, and penetration tasks across four LLMs.
-
No More, No Less: Task Alignment in Terminal Agents
The TAB benchmark reveals that frontier terminal agents achieve high task completion but low selective alignment with relevant environmental cues over distractors, and prompt-injection defenses block both.
-
CrackMeBench: Binary Reverse Engineering for Agents
CrackMeBench introduces 20 deterministic binary validation tasks and reports GPT-5.5 solving 11/12 generated ones at pass@3 while Claude and Kimi lag, especially on harder tasks.
-
Beyond Collection: Measuring the Detection Efficacy of Modern Security Logging Standards
SETC framework provides the first systematic comparison of CIM, OCSF, and ECS logging standards by running 50 RCE exploits and measuring how well each captures attack indicators.
-
Benchmarking and Exploring the Capabilities of LLMs for Attack Investigations
AuditBench is a new benchmark of audit logs from 50+ malicious and benign scenarios that evaluates five LLMs on four security investigation tasks and analyzes their performance and error profiles.
-
CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities
CyberGym-E2E turns 920 real OSS-Fuzz vulnerabilities into end-to-end agent tasks; results show frontier agents patch known bugs well (65.9% S3) but find the exact ground-truth vulnerability less often (22.2% S4).
-
AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection
AnyPoC validates LLM bug reports by synthesizing, executing, and independently re-checking proof-of-concept tests, finding 121 new bugs with 108 confirmed across large systems.
-
Program Analysis Guided LLM Agent for Proof-of-Concept Generation
PAGENT integrates static and dynamic program analysis guidance with an LLM agent to improve automated proof-of-concept generation success by 132% over prior agentic methods.
-
Two AI Metrics Diverged: Will it Make All the Difference?
Bounded performance metrics always favor convergence of AI capabilities to meek models while unbounded metrics allow frontier models to maintain leads indefinitely, with policy implications for capability concentration.
-
GLM-5: from Vibe Coding to Agentic Engineering
GLM-5 is a foundation model that claims state-of-the-art results on coding benchmarks and superior performance on end-to-end software engineering tasks via new asynchronous RL methods and cost-saving DSA.
-
Kimi K2.5: Visual Agentic Intelligence
Kimi K2.5 combines joint text-vision training with an Agent Swarm parallel orchestration framework to reach claimed state-of-the-art results on coding, vision, reasoning, and agent tasks while cutting latency up to 4.5 times.
-
Hephaestus: Toward a Cybersecurity AI Scientist
The paper proposes the Cybersecurity AI Scientist as a modular multi-agent architecture for automating cybersecurity research, distinguished by its focus on non-stationary threats and anchored in a four-zeros risk-trust-incident-energy frame.
-
Agentic AI and the Industrialization of Cyber Offense: Forecast, Consequences, and Defensive Priorities for Enterprises and the Mittelstand
Agentic AI lowers the cost and speed of cyber attacks, requiring immediate improvements in identity management, phishing-resistant authentication, patching, and agent governance for large enterprises and the Mittelstand.