Cyber-Zero: Training Cybersecurity Agents without Runtime

Dingmin Wang; Hantian Ding; Terry Yue Zhuo; Varun Kumar; Zijian Wang

REVIEW 4 major objections 3 minor 2 cited by

Cyber-Zero: Training Cybersecurity Agents without Runtime

T0 review · 4 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Cyber-Zero claims that public CTF writeups, processed by persona-driven LLM simulation, can replace live runtime environments for training cybersecurity agents, yielding up to 13.1% absolute gains on three benchmarks.

desk verdict Useful idea, strong claim, but abstract-only leaves contamination as the open risk; worth sending to review with a demand for decontamination analysis. read the letter →

arxiv 2508.00910 v2 pith:GLDZBWVK submitted 2025-07-29 cs.CR cs.CLcs.LG

Terry Yue Zhuo , Dingmin Wang , Hantian Ding , Varun Kumar , Zijian Wang This is my paper

classification cs.CRcs.CLcs.LG

keywords cybersecurityLLMagentsCTFbenchmarkstrajectorysynthesisruntime-freetrainingpersona-drivensimulationopen-weightmodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that high-quality training data for cybersecurity AI agents can be manufactured without any live computing environment. It introduces Cyber-Zero, a pipeline that reads public CTF writeups and uses persona-driven LLM simulation to recreate the commands, outputs, and decisions an agent would make. Agents trained on these synthetic trajectories gain up to 13.1% absolute accuracy on InterCode-CTF, NYU CTF Bench, and Cybench, and the open-weight Cyber-Zero-32B matches closed proprietary models. The broader claim is that runtime-free synthesis can democratize state-of-the-art cybersecurity agents.

What carries the argument

The load-bearing mechanism is the persona-driven simulation step, which takes a public CTF writeup and prompts an LLM to act out the solving session from the perspective of the original solver, generating a full command-and-response trajectory as if it were recorded in a live environment. This converts passive documentation into active training data without ever running the challenge.

What would settle it

Take a sample of Cyber-Zero's synthesized trajectories and replay their commands against the real CTF environments they claim to simulate; if a large fraction of commands error out or produce different outputs than recorded, the simulation premise fails. A cleaner version: train two agents, one on synthesized logs and one on real execution logs from the same challenges, and compare performance on held-out tasks; if the synthesized-trained agent loses, the runtime-free premise is the bottleneck.

Watch

Extended reading notes

Core claim

Cyber-Zero is presented as the first runtime-free framework for synthesizing agent trajectories for cybersecurity LLMs. Its central discovery is that public CTF writeups, which describe how humans solved challenges, contain enough latent information about environment behavior that a persona-driven LLM can reverse-engineer realistic interaction logs. Training on these logs produces agents that outperform their base models on three CTF benchmarks, with the 32B open-weight model matching DeepSeek-V3-0324 and Claude-3.5-Sonnet. This establishes that executable environments are not strictly necessary to train competitive cybersecurity agents.

Load-bearing premise

The framework assumes that an LLM reading a public writeup can faithfully reconstruct what a real terminal session would have printed and accepted, without ever running the challenge; if the simulated commands or outputs drift from reality, the trained agents will learn from logs that do not match how actual systems behave.

Editorial extensions

If this is right

Cybersecurity agent training no longer requires access to ephemeral or restricted challenge environments.
Open-weight models can reach the level of proprietary systems purely from writeup-derived data at lower cost.
The same trajectory-synthesis approach can scale to any domain where written solutions exist but runtimes are unavailable.
The three CTF benchmarks can be used to measure gains even when true environment logs are absent.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the fidelity of simulation is high, the approach generalizes to penetration testing and incident response, where recording real sessions is often impossible for security reasons.
The reliance on public writeups means the data source is biased toward challenges that authors chose to document and solve in a particular style, which could narrow the behavior distribution of trained agents.
A natural extension is a hybrid loop: train on synthetic trajectories, then fine-tune on a small amount of real environment feedback when available, correcting any simulation drift.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague—you asked for a read on arXiv:2508.00910. I've only got the abstract (full text seems missing), so this is a judgment of the claim, not the paper.

The core idea is genuinely attractive: use public CTF writeups, run persona-driven LLM simulation to replay what an agent did in the challenge, and train a model on those synthetic interaction logs—no live environment needed. If it works, that's a real cost reduction for building security agents, and 'first runtime-free framework' for CTF agent training is a plausible novelty over prior LLM-simulation work. The authors picked three standard benchmarks and report up to 13.1% absolute gains plus open-weight parity with DeepSeek-V3-0324 and Claude-3.5-Sonnet. That is a strong, concrete claim.

The soft spots are real but not disqualifying. The big one is contamination. Training on public writeups and evaluating on public CTF benchmarks means overlapping challenges could leak into training. The abstract reports no decontamination procedure, no split audit, no ablation. If a chunk of the gains come from memorization, the generalization story collapses. That is load-bearing, and a referee should demand an explicit overlap analysis. Second, the abstract gives no protocol: no baselines beyond 'baseline models,' no error bars, no cost numbers, no hyperparameters. That is normal for an abstract, but it means the 13.1% number is currently a promise. Third, the reverse-engineering assumption—simulated sessions can hallucinate commands and outputs—needs evidence that trained agents solve unseen challenges rather than imitate template traces. That last concern is more about future work than a fatal flaw.

I would not write the paper off. The direction is useful, the claim is falsifiable, and the empirical setup is standard enough to evaluate. What it needs is a peer-review round that pushes for a decontamination audit and a comparison against simpler baselines like fine-tuning on writeup text alone. If those hold, this is a solid contribution; if they do not, the headline number is noise.

Recommendation: send to peer review, not desk reject. It is the kind of paper where a good referee run can separate a real technique from a benchmark artifact.

Referee Report

4 major / 3 minor

Summary. The manuscript (arXiv:2508.00910) introduces Cyber-Zero, a framework that synthesizes agent training trajectories from publicly available CTF writeups using persona-driven LLM simulation, without requiring executable runtime environments. The authors claim these synthetic trajectories train LLM-based agents that achieve up to 13.1% absolute performance gains over baseline models on InterCode-CTF, NYU CTF Bench, and Cybench. They further claim that Cyber-Zero-32B matches the capability of proprietary systems such as DeepSeek-V3-0324 and Claude-3.5-Sonnet while offering superior cost-effectiveness, thereby demonstrating that runtime-free trajectory synthesis can democratize cybersecurity agent development.

Significance. If the empirical claims hold, Cyber-Zero would be a meaningful contribution: it proposes a way to train capable cybersecurity agents without needing executable environments, which are often restricted or ephemeral. The idea of reverse-engineering runtime behavior from writeups via persona-driven LLM simulation is interesting and could reduce the resource barriers for building cybersecurity agents. The three-benchmark evaluation and the comparison with proprietary models, if substantiated, would strengthen the case for open-weight agents in this domain. However, the significance assessment is conditional on the missing experimental details and the decontamination status, because the training and evaluation sets both derive from public CTF sources.

major comments (4)

[Abstract] The reported 'up to 13.1% absolute performance gains' is not accompanied by any experimental protocol, including the number of evaluation runs, variance or error bars, baseline definitions, or the precise training data and hyperparameters; without these, the headline empirical claim cannot be independently assessed.
[Abstract] Because the training data are public CTF writeups and the evaluation benchmarks (InterCode-CTF, NYU CTF Bench, Cybench) are public CTF challenge collections, the absence of any decontamination or overlap audit raises the risk that the reported gains reflect memorization or retrieval of near-duplicate solutions rather than runtime-free generalization; this concern is load-bearing for the claim that 'runtime-free trajectory synthesis' teaches agents to handle real, unseen challenges.
[Abstract] The assertion that Cyber-Zero is 'the first runtime-free framework' requires a comparison with prior work on synthetic trajectory generation, imitation learning from static artifacts, or distillation from proprietary agents; no such comparison or citation is provided, so the novelty claim is not yet established.
[Abstract] The claim that Cyber-Zero-32B 'matches DeepSeek-V3-0324 and Claude-3.5-Sonnet' is ambiguous because it does not state the evaluation setting, the specific metrics, or whether the proprietary models are used under the same tool-use and observation conditions; a fair comparison would require identical interfaces, budgets, and starting contexts.

minor comments (3)

[Abstract] The phrase 'persona-driven LLM simulation' is used without a definition or example, making the core method unfalsifiable from the abstract alone.
[Abstract] The cost-effectiveness claim ('superior cost-effectiveness') is not supported by any cost model or measurement in the abstract.
[Abstract] The abstract does not state the model sizes or training compute for the baseline models, making the 'open-weight' comparison difficult to situate.

Circularity Check

0 steps flagged · score 0.0 of 10

No construction-level circularity: the abstract describes an empirical pipeline whose central claims are tested on external CTF benchmarks, not derived from its own training inputs.

full rationale

The paper's derivation chain, as visible in the abstract, is: collect public CTF writeups, use persona-driven LLM simulation to synthesize agent trajectories without runtime environments, train LLM-based agents on those trajectories, and evaluate the trained agents on InterCode-CTF, NYU CTF Bench, and Cybench. No step defines the target outcome in terms of the inputs. The reported 'up to 13.1% absolute performance gains' are empirical measurements on external benchmarks, not consequences of how the trajectories were constructed. There is no equation in the abstract that reduces a predicted quantity to a fitted parameter, and there is no self-citation invoked as a load-bearing premise. The concern that public writeup corpora may overlap with public benchmark challenges is a data contamination risk, not a circularity of derivation; contamination would undermine external validity but does not make the framework's reasoning self-referential by construction. Without access to the full text, no specific circular step can be quoted or exhibited, and the available abstract provides no basis for inferring one. Therefore the appropriate finding is no significant circularity, with score 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

From the abstract alone, the framework rests on three unstated assumptions: writeups contain enough information to reconstruct runtime traces, LLM simulation can proxy real environments, and evaluation benchmarks are not contaminated by public writeups. No free parameters or invented entities are visible in the abstract.

assumptions (3)

domain assumption CTF writeups contain sufficient information to reconstruct runtime behaviors.
The method grounds simulated trajectories in writeups; if writeups omit command outputs, environment states, or failure paths, the simulations will be inaccurate.
ad hoc to paper Persona-driven LLM simulation can faithfully reverse-engineer execution without a runtime.
This is the core premise of Cyber-Zero and is not validated in the abstract; simulated outputs may hallucinate or diverge from real executions.
domain assumption The evaluation benchmarks do not overlap with the public writeup training corpus.
Any overlap would make reported benchmark gains reflect memorization rather than generalization, so leakage-free evaluation is necessary.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Cyber-Zero: Training Cybersecurity Agents without Runtime." pith.science (2026). https://pith.science/paper/GLDZBWVK

@misc{pith2026250800910,
  author       = {Pith},
  title        = {Pith review of: Cyber-Zero: Training Cybersecurity Agents without Runtime},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GLDZBWVK}},
  note         = {Machine review of arXiv:2508.00910}
}

read the original abstract

Large Language Models (LLMs) have achieved remarkable success in software engineering tasks when trained with executable runtime environments, particularly in resolving GitHub issues. However, such runtime environments are often unavailable in other domains, especially cybersecurity, where challenge configurations and execution contexts are ephemeral or restricted. We present Cyber-Zero, the first runtime-free framework for synthesizing high-quality agent trajectories to train cybersecurity LLMs. Cyber-Zero leverages publicly available CTF writeups and employs persona-driven LLM simulation to reverse-engineer runtime behaviors and generate realistic, long-horizon interaction sequences without actual environments. Using trajectories synthesized by Cyber-Zero, we train LLM-based agents that achieve up to 13.1% absolute performance gains over baseline models on three prominent CTF benchmarks: InterCode-CTF, NYU CTF Bench, and Cybench. Our best model, Cyber-Zero-32B, establishes new state-of-the-art performance among open-weight models, matching the capabilities of proprietary systems like DeepSeek-V3-0324 and Claude-3.5-Sonnet while offering superior cost-effectiveness, and demonstrating that runtime-free trajectory synthesis can effectively democratize the development of state-of-the-art cybersecurity agents.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

AgentSnare: Learning to Delay, Divert, and Defuse Autonomous Penetration Agents
cs.CR 2026-07 conditional novelty 7.0 of 10

A trajectory-adaptive honeypot system, AgentSnare, achieves a 0/45 verified exploit rate against LLM-based penetration testers across 15 vulnerable web apps and three attacker models.
Uncovering Vulnerabilities of LLM-Assisted Cyber Threat Intelligence
cs.CR 2025-09 conditional novelty 6.0 of 10

LLMs assisting cyber threat intelligence fail mainly due to spurious correlations, contradictory knowledge, and constrained generalization that stem from the threat landscape itself.

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.