Pith. sign in

REVIEW 4 major objections 6 minor 67 references

Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a 7B-parameter language-model policy, trained with reinforcement learning on verifiable execution rewards, reduces the performance of DRL cyber defenders by an average of 522% compared to static red agents, and that…

desk verdict Valuable benchmark and pipeline, but the 522% headline is compared only to the default static baseline, so the central claim that static heuristics entirely fail is not yet supported. read the letter →

arxiv 2608.04317 v1 pith:5CPMXFNF submitted 2026-08-05 cs.CR cs.AIcs.LGcs.MA

Ryozo Masukawa , Ian Bryant , Armita Kazeminajafabadi , Sanggeon Yun , Hyunwoo Oh , SungHeon Jeong , Nathaniel D. Bastian , Mahdi Imani
show 1 more author
Mohsen Imani
This is my paper · ORCID
classification cs.CRcs.AIcs.LGcs.MA
keywords reinforcementlearningwithverifiablerewardsautonomouscyberdefenseLLMredteamingcode-as-policyagenticDRLrobustnessCybORGCAGE4Wheel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that current DRL-based cyber defenders are brittle against adaptive attackers, and that a comparably small 7B language-model policy trained with reinforcement learning on verifiable execution rewards can systematically destroy them. To show this, it introduces Trident, a benchmark and training infrastructure spanning two cyber-defense simulators, plus Trident Agentic, an architecture that writes attack policies as Python code. On average across five DRL defenders, the trained planner reduces the defenders' scores by 522% relative to the default static red agents, and it discovers behaviors the heuristics never exhibit, such as avoiding honeypots and prioritizing high-value host states. The paper's significance is in changing what an evaluation of a DRL defender must include: an adaptive, code-writing attacker rather than a fixed heuristic.

What carries the argument

The central object is Trident Agentic, a 'Code-as-Policy' contextual-bandit architecture. A frozen Log Summarizer compresses massive execution logs into a dense summary; a trainable Planner (a 7B instruction-tuned model) outputs a strategy JSON in one forward pass; a frozen Coder translates that JSON into executable Python that patches the red agent's core methods (state transition probabilities, host priority list, observation processing in CAGE 4; target-selection strategy in CyberWheel). The policy is optimized with GRPO using a hand-specified verifiable reward that counts killchain milestones, penalizes failed and wasteful actions, applies a decoy penalty in CyberWheel, and imposes system-level penalties for code that leaks simulator internals, errors, or times out. This reformulation turns the Dec-POMDP red-team problem into a contextual bandit: one code generation per episode, reward at the end.

What would settle it

Train a simple black-box optimizer (random search or Bayesian optimization) over the same strategy parameters the Planner controls—state-transition probabilities, host priority lists, and the decoy threshold—using the paper's own reward function, and compare the resulting blue-agent degradation with Trident Agentic's 522%. If the optimizer matches or exceeds Trident without any language model or RLVR training, the paper's central claim that RLVR-trained LLM reasoning is the source of the advantage is falsified.

Watch

Extended reading notes

Core claim

We show that a single trainable 7B planner, embedded in a Log Summarizer–Planner–Coder loop and trained by GRPO on verifiable rewards from sandboxed executions, learns to compromise state-of-the-art DRL blue agents far beyond any static red agent. Across CAGE 4 and CyberWheel, the resulting attack policies cut defender cumulative rewards by an average of 522.30% compared to the default finite-state or BFS heuristics, with the trained small model surpassing a frontier-scale GPT-4o planner. The training also yields reproducible emergent tactics—decoy avoidance in CyberWheel and adaptive host-state prioritization in CAGE 4—that the static baselines cannot represent.

Load-bearing premise

The load-bearing premise is that the paper's hand-designed verifiable reward function, with its chosen event weights and decoy penalty, faithfully measures genuine compromise of the DRL defender; if the reward rewards the wrong actions, the reported attack success and emergent behaviors partly reflect the reward design rather than real weaknesses in the defenders.

Editorial extensions

If this is right

  • Current DRL defenders, trained against static heuristics, are systematically under-evaluated; adaptive code-writing attackers of modest scale can expose order-of-magnitude performance drops.
  • RLVR is a viable training signal for cyber offense: verifiable execution logs from sandboxed simulators suffice to train a 7B model to outperform frontier prompt-only planners.
  • The decoy-avoidance finding implies that defenders' use of honeypots can be reverse-engineered and countered by attacker-learned heuristics, not just by chance.
  • The benchmark and dataset of over 13,000 red-blue interaction trajectories enable reproducible RLVR training against active defenders, inviting defense-side co-evolution.
  • The sandbox's RESTful architecture makes the framework simulator-agnostic, so future higher-fidelity environments can be plugged in without retraining the whole pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 522% reduction generalizes to higher-fidelity simulators (which the paper leaves as future work), the practical implication is that DRL defenders must be trained against adversarial code-generating red teams, not fixed finite-state machines, or else evasion of learned policies is likely in real deployment.
  • The paper's reward design includes a decoy penalty that the authors chose; without it, the 'emergent' decoy avoidance disappears (shown in the paper's own ablation). A sympathetic reader should ask whether other claimed emergent behaviors, such as adaptive state prioritization, are similarly reward-shaped—this is testable by ablating individual reward terms.
  • The contextual-bandit reformulation drops the sequential decision problem: an attacker that commits to a full policy before observing the defender's live reactions may be less adaptive than the framing suggests. A harder test would be to allow mid-episode replanning from updated logs.
  • The dataset mixes trajectories from many heuristic red agents; training on that mixture may partly explain the planner's breadth. An editor's hypothesis is that a planner trained only on default-heuristic trajectories would show lower zero-shot transfer across defenders, which the released data could test by subsampling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Trident, a red-teaming benchmark and dataset for training LLM-based attack agents against active DRL cyber defenders. Trident Agentic uses a frozen Log Summarizer, a trainable 7B Planner trained by GRPO under a Code-as-Policy contextual-bandit formulation, and a frozen Coder that translates strategic JSON into executable Python policies. The authors evaluate against three CAGE 4 defenders (GNN, HMARL3, HMARL4, MARL) and one CyberWheel defender (PPO NN), reporting an average 522% reduction in blue-agent reward relative to the default static red agent, plus qualitative 'emergent' behaviors such as decoy avoidance and adaptive state prioritization.

Significance. If the central claims hold, this is a valuable contribution: it is among the first demonstrations of RLVR training against active DRL defenders, and the released infrastructure (sandboxed RESTful interface, 13,000+ trajectories, code-as-policy architecture) could support future work on adaptive red agents and more robust blue agents. The paper provides a clean contextual-bandit formulation (Eq. 1) and a credible design for verifiable rewards (Eqs. 4-5). However, the headline empirical claim is currently supported only against the weakest baseline in the paper's own heuristic inventory, and the reward proxy's fidelity is asserted more strongly than the evidence shows.

major comments (4)
  1. [§4.1, Table 2, Appendix B] The headline 522.30% reduction is computed only against the single 'Default Agent' baseline (FiniteStateRedAgent for CAGE 4 and BFSServerDowntime for CyberWheel), even though §3.1 and Appendix B build and use a diverse suite of 12 CAGE 4 FSM agents and 8 CyberWheel strategies to construct the training dataset. None of the additional heuristics are evaluated in Table 2, so the §4.2 claim that 'static heuristics entirely fail to uncover' the brittleness is unsupported. Several Appendix B heuristics encode the same behaviors later attributed to RLVR (e.g., FSRedAgentTargetA with ×100 target-zone weights, FSRedAgentFailureTracker with failure-count-based host suppression). Please report blue-agent rewards for the full Appendix B inventory, or strictly limit the comparative claim to the default baseline.
  2. [§A.1.3, §A.2.3, Eqs. (4)-(5), Appendix C.3] The reward function is hand-authored, with weights chosen by the authors (qimpact=0.2, qroot=0.3, quser=0.1, qscan=0.05, r=2.0, sdecoy=0.5, etc.). Appendix C.3 shows that removing the decoy penalty eliminates the decoy-avoidance behavior, and Figure 9 shows that without this penalty red and blue rewards rise together. This demonstrates that the 'emergent' decoy avoidance is a direct consequence of the reward design rather than an independently discovered weakness in the PPO defender. The paper's claim that the verifiable reward is a 'high-fidelity proxy' (Appendix C.1) rests on correlations between the proxy and blue reward; for CyberWheel the Pearson correlation is only -0.42 (Figure 8). Please provide external validation of the proxy using defender-utility components not used in the red reward, and soften the emergence claim accordingly.
  3. [§4.1 Evaluation Metrics, Table 4] The baseline metrics for GPT-4o, Zero-shot, and CoT are computed exclusively over successful executions, while Trident's evaluation includes all episodes and execution failures. Table 4 shows baseline success rates of 26.3%, 51.7%, and 55.3%, respectively. Filtering failed episodes changes the composition of evaluated trajectories and can bias the comparison in either direction; the paper states this asymmetry favors baselines, but it does not report unfiltered numbers to verify the claim. Please report results over all episodes for every method, or provide a sensitivity analysis showing that filtering does not alter the conclusions.
  4. [Table 2 and Abstract] The 'Average Loss (∆)' of 522.30% is an unweighted mean of relative reductions across five defenders with very different reward scales, so a single row (GNN, 929.79%) dominates the average. The headline should also report an aggregate absolute reduction (e.g., pooled reward difference across defenders) or a normalized effect size, together with a confidence interval for the average. As written, the 522% figure overstates the consistency of the effect.
minor comments (6)
  1. [Abstract] The phrase 'a dataset comprises over 13,000' should read 'a dataset comprising over 13,000'; the abstract also says 'compared to static red agent baselines' but the comparison is to a single default baseline.
  2. [§5.2] The citation '[36] or and [24]' in the sentence about RL-related work is broken; please correct to a proper reference list.
  3. [Table 1] The header 'Active Defesne?' contains a typo and should read 'Active Defense?'.
  4. [§2.1 and §2.2] There is a notation collision: code is denoted c in §2.1 and context is denoted C in §2.2; Eq. (1) writes c ∼ πθ(·|C) and then uses πc for the trajectory distribution, which is ambiguous. Please rename one of the symbols.
  5. [Appendix C.1, Figure 8] The text states Pearson correlations range from -0.87 to -0.94 in CAGE 4 and 'a similar negative trend' in CyberWheel, but Figure 8 shows CyberWheel Pearson = -0.42 with Spearman = -0.73; the gap between Pearson and Spearman should be acknowledged and the text corrected.
  6. [Appendix A.2.2] The text contains an editorial leftover: 'here is the refined LaTeX snippet' followed by a self-referential sentence. This should be removed before submission.

Circularity Check

2 steps flagged · score 4.0 of 10

Main blue-reward reduction is externally measured and not circular, but the 'emergent' decoy avoidance and state-prioritization behaviors are engineered into the hand-authored RLVR reward, so those qualitative discoveries are partially circular.

  1. fitted input called prediction [Appendix A.2.3; Appendix C.3; Section 4.2 (Qualitative Analysis, Figure 6)]
    "Crucially, treating decoy hits as negative events addresses a misalignment in default simulation reward structures [39]. In CyberWheel's default deception scenario, a red agent receives a positive reward for impacting a decoy, while the blue agent is simultaneously rewarded for a successful misdirection. By classifying decoy interactions as strict negative penalties, we prevent a mutually beneficial local optimum where the red agent naively farms impact rewards on honeypots, ensuring the policy autonomously learns genuine decoy avoidance."

    The reward function (Eq. 4) contains an explicit hand-set penalty sdecoy=0.5 for decoy interactions. Section 4.2 then reports that Trident 'autonomously synthesizes a decoy-avoidance mechanism' as an emergent discovery. But decoy avoidance is the direct optimization target of the reward: GRPO maximizes R+ − R−, so a policy that avoids decoys receives higher reward by construction. The paper's own ablation (Appendix C.3, Figure 9) shows the behavior disappears when sdecoy is removed, confirming that the reported 'discovery' is a fitted consequence of the reward parameter, not an independent emergent property of the trained agent. Calling it 'autonomously discovered' reports the reward designer's choice back as a finding.

  2. fitted input called prediction [Appendix A.1.3; Section 4.2 (Qualitative Analysis, Figure 5)]
    "We assign weights of qimpact = 0.2, qroot = 0.3, quser = 0.1, and qscan = 0.05. ... In CAGE 4, rather than treating all network states equally, Trident Agentic learns to prioritize high-value states such as Root and User access over basic reconnaissance states."

    The reward weights already place the highest value on root compromise (qroot=0.3) and user access (quser=0.1) and the lowest value on scanning (qscan=0.05). Section 4.2/Figure 5 then presents as 'emergent' a learned priority ordering that favors Root and User states over reconnaissance. Since GRPO optimizes exactly this weighted sum, any near-optimal policy must favor root/user milestones; the learned host-state priority list reproduces the ordering already encoded in q. The behavior is therefore an expected consequence of the hand-authored reward, not an autonomously discovered weakness, and the paper's own Section 6 limitation that reward design is environment-dependent reinforces that the 'discovery' is an artifact of the reward scaffolding.

full rationale

Trident's central quantitative claim—the 522% average reduction in blue reward—is measured on the defenders' own cumulative rewards, which are external to the red RLVR training reward, so the headline performance result is not circular. The circularity is confined to the qualitative 'emergent behavior' claims. The hand-authored reward (Eqs. 4–5) explicitly contains per-event weights favoring root/user milestones and an explicit decoy penalty; the behaviors reported as autonomously discovered (decoy avoidance, adaptive state prioritization) are the direct optima of that reward. Appendix C.3's ablation demonstrates that removing sdecoy removes the reported decoy-avoidance behavior, confirming that this 'discovery' is a fitted consequence of the reward design rather than an independent emergent property. Likewise, the learned host-state priorities mirror the qroot > quser > qscan ordering built into the reward. No load-bearing self-citation chain was found: reference [24] includes present authors but is used only as a related-work contrast, not as evidence for Trident's claims. The comparison against only a single Default Agent baseline is a serious evaluation concern, but it is a baseline-selection issue, not circularity. Score 4 reflects partial circularity in supporting qualitative claims while the main performance result retains independent external content.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central result rests on two categories of uncharged inputs: hand-chosen reward and penalty parameters that directly shape the learned behaviors, and domain assumptions about simulator fidelity, defender representativeness, and the contextual-bandit reformulation. No new physical or natural entities are introduced.

free parameters (5)
  • Reward event weights = CAGE4: 0.2/0.3/0.1/0.05; CyberWheel: 0.2/0.1/0.02; root/user bonus 0.3/0.1
    Hand-chosen in Appendix A.1.3 and A.2.3 to shape the RLVR signal; they encode the priorities later reported as learned.
  • Efficiency multiplier r = 2.0 in both environments
    Chosen by hand; normalizes achievement count by episode length via Eq. 4.
  • Negative event penalties = CAGE4 invalid=0.05 exploit_fail=0.1; CyberWheel fail=0.05 repeat=0.02 decoy=0.5
    Hand-chosen; sdecoy directly rewards avoidance of honeypots, so the learned decoy avoidance is partly imposed.
  • Execution penalties p = -10*Nleak, -10 empty, -5 exec, -2 timeout, -1 format/http
    Hand-selected values in Eq. 6; affect training dynamics but not the headline comparison.
  • GRPO hyperparameters = G=8, lr=5e-7, beta=0.05 CyberWheel, beta=5e-4 CAGE4
    Chosen by the authors for stability; not fitted to holdout data.
assumptions (5)
  • domain assumption The simulator rewards from CAGE 4 and CyberWheel are deterministic, verifiable, and faithful to real cyber operations.
    Used throughout; the entire RLVR setup and sandbox design depend on this, and the paper relies on the simulators' documentation rather than real-world validation.
  • domain assumption Reformulating red teaming as a contextual bandit with one complete policy generation per episode loses no strategically important adaptivity.
    Section 2.2; the planner cannot react within an episode except through the conditional logic embedded in generated code.
  • domain assumption The selected DRL blue agents are state-of-the-art and their official pretrained weights are correctly configured.
    Section 4.1 victim models; the "fundamental brittleness" conclusion is only as strong as these checkpoints.
  • ad hoc to paper The hand-crafted reward function is a high-fidelity proxy for real defender degradation.
    Section 3.4 and Appendix C.1; the authors validate correlation post hoc, but the weights are ad hoc and the behaviors reported as emergent are encoded in the reward.
  • domain assumption The heuristic-generated trajectories in the dataset provide sufficient coverage to bootstrap RLVR training.
    Section 3.1; contexts are sampled from trajectories produced by static heuristics, not from an adaptive policy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)." pith.science (2026). https://pith.science/paper/5CPMXFNF

@misc{pith2026260804317,
  author       = {Pith},
  title        = {Pith review of: Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5CPMXFNF}},
  note         = {Machine review of arXiv:2608.04317}
}
read the original abstract

Autonomous cyber defense systems based on Deep Reinforcement Learning (DRL) have attracted significant research attention, yet remain evaluated almost exclusively against static, heuristic red agents, leaving their robustness against adaptive threats critically understudied. Meanwhile, recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) have improved LLM reasoning, but their integration into cybersecurity remains elusive due to the absence of suitable benchmark environments and interaction datasets. To bridge this gap, we introduce Trident, an agentic LLM red teaming framework comprising three components: a dynamic benchmark with isolated sandbox servers spanning CybORG CAGE 4 and CyberWheel, a dataset comprises over 13,000 high-fidelity red-blue interaction trajectories for RLVR, and a ``Code-as-Policy'' RLVR agentic architecture Trident Agentic). The latter reformulates red agent training as a contextual bandit via a tripartite Log Summarizer--Planner--Coder design, where a trainable Planner generates complete attack strategies from compressed execution logs, which a frozen Coder translates into executable Python policies deployed against live DRL defenders. Empirical evaluations reveal a fundamental brittleness in existing defenses: with a single trainable 7B planner, Trident reduces blue agent defensive performance by an average of 522% compared to static red agent baselines while autonomously discovering emergent behaviors such as decoy avoidance and adaptive state prioritization that static heuristics entirely fail to uncover.

Figures

Figures reproduced from arXiv: 2608.04317 by the authors.

Figure 1
Figure 1. Overview of the Trident agentic benchmark framework. A Log Summarizer observes and summarizes sanitized logs, enabling a trainable Planner (πθ) to formulate abstract tactics. A frozen Coder translates the tactic into an executable Python policy deployed against a live DRL defender. Execution logs provide a verifiable reward signalto update πθ via GRPO. The ideal red agent for rigorous evaluation should be adaptive: … view at source ↗
Figure 2
Figure 2. Tactical branching in CybORG and CyberWheel. Each row represents one of 50 sampled trajectories from a baseline heuristic against a fixed DRL defender. This seed variation ensures diverse data for RLVR training. Trajectory Rollout: We leverage a diverse suite of N red agent against a fixed DRL blue defender (Details in Appendix B), iteratively sampling a heuristic cj ∼ Uniform(1, N) and a randomized seed si ∼ S to g… view at source ↗
Figure 3
Figure 3. Example of a serialized trajectory raw log template from the CyberWheel environment. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Planner-Coder Pipeline. Furthermore, an iterative self-repair loop is employed to mitigate minor syntax errors that could disrupt the learn￾ing. A sandbox performs a fast-pass sanity check; if an execution error occurs, the traceback is provided to the Coder for correc…
Figure 5
Figure 5. Figure 5: CAGE4 policy generation example. Tri￾dent Agentic coder introduces a structured priority that shifts focus toward high-value targets. To evaluate the effectiveness and robustness of Trident Agentic, we conduct our experiments within multiple environments with different…
Figure 6
Figure 6. Figure 6: Example generated logic for decoy avoidance [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Learning dynamics of verifiable tactical rewards against defender performance. As the RLVR process progresses (x-axis: episodes), the red agent successfully maximizes its verifiable reward, consistently driving the blue agent’s accumulated reward into a steep decline. …
Figure 8
Figure 8. Figure 8: Correlation between verifiable tactical rewards and defender performance. Scatter plots illustrating the strong negative correlation between Trident’s episodic rewards and the blue agent’s rewards. The high correlation coefficients validate that our verifiable reward s…
Figure 9
Figure 9. Figure 9: Impact of decoy interaction penalties on optimization alignment in CyberWheel. (Left) Without a specific penalty, the agent reaches a misaligned local optimum by repeatedly interacting with decoys, mutually benefiting both agents. (Right) By incorporating a strict nega…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 38 canonical work pages

  1. [1]

    https://www.atomicredteam.io/

    Atomic Red Team™ — atomicredteam.io. https://www.atomicredteam.io/. [Accessed 20-04-2026]

    work page 2026

  2. [2]

    https://aicyberchallenge.com/ overview/

    Darpa’s artificial intelligence cyber challenge (aixcc). https://aicyberchallenge.com/ overview/. [Accessed 17-04-2026]

    work page 2026

  3. [3]

    https://attack.mitre.org/

    MITRE ATT&CK; — attack.mitre.org. https://attack.mitre.org/. [Accessed 20-04- 2026]

    work page 2026

  4. [4]

    EnIGMA: Interactive tools substantially assist LM agents in finding security vulnerabilities

    Talor Abramovich, Meet Udeshi, Minghao Shao, Kilian Lieret, Haoran Xi, Kimberly Milner, Sofija Jancheska, John Yang, Carlos E Jimenez, Farshad Khorrami, Prashanth Krishnamurthy, Brendan Dolan-Gavitt, Muhammad Shafique, Karthik R Narasimhan, Ramesh Karri, and Ofir Press. EnIGMA: Interactive tools substantially assist LM agents in finding security vulnerabi...

    work page 2025

  5. [5]

    Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational...

    2024

  6. [6]

    Ctibench: a benchmark for evaluating llms in cyber threat intelligence

    Md Tanvirul Alam, Le Nguyen, Dipkamal Bhusal, and Nidhi Rastogi. Ctibench: a benchmark for evaluating llms in cyber threat intelligence. InProceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY , USA, 2024. Curran Associates Inc

    work page 2024

  7. [7]

    Claude Mythos Preview red.anthropic.com — red.anthropic.com

    Anthropic. Claude Mythos Preview red.anthropic.com — red.anthropic.com. https://red. anthropic.com/2026/mythos-preview/. [Accessed 17-04-2026]

    work page 2026

  8. [8]

    Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models.arXiv preprint arXiv:2404.13161, 2024

    Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, et al. Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models.arXiv preprint arXiv:2404.13161, 2024

    arXiv 2024

Show all 67 references
  1. [9]

    Purple llama cyberseceval: A secure coding benchmark for language models.arXiv preprint arXiv:2312.04724, 2023

    Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evtimov, Do- minik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, et al. Purple llama cyberseceval: A secure coding benchmark for language models.arXiv preprint arXiv:2312.04724, 2023

    2023 arXiv

  2. [10]

    Large language models are autonomous cyber defenders

    Sebastián R Castro, Roberto Campbell, Nancy Lau, Octavio Villalobos, Jiaqi Duan, and Alvaro A Cardenas. Large language models are autonomous cyber defenders. In2025 IEEE Conference on Artificial Intelligence (CAI), pages 1125–1132. IEEE, 2025. 10

    2025

  3. [11]

    Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors

    Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agentverse: Facilitating multi-agent collaboration and exploring emergent behavio...

    2024

  4. [12]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  5. [13]

    PentestGPT: Evaluating and harnessing large language models for automated penetration testing

    Gelei Deng, Yi Liu, Víctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. PentestGPT: Evaluating and harnessing large language models for automated penetration testing. In33rd USENIX Security Symposium (USENIX Security...

    2024

  6. [14]

    Llm agents can autonomously exploit one-day vulnerabilities.arXiv preprint arXiv:2404.08144, 2024

    Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. Llm agents can autonomously exploit one-day vulnerabilities.arXiv preprint arXiv:2404.08144, 2024

    2024 arXiv

  7. [15]

    Llm agents can autonomously hack websites.arXiv preprint arXiv:2402.06664, 2024

    Richard Fang, Rohan Bindu, Akul Gupta, Qiusi Zhan, and Daniel Kang. Llm agents can autonomously hack websites.arXiv preprint arXiv:2402.06664, 2024

    2024 arXiv

  8. [16]

    Graphplanner: Graph- based agentic routing for LLMs

    Tao Feng, Haozhen Zhang, Peixuan Han, Zijie Lei, and Jiaxuan You. Graphplanner: Graph- based agentic routing for LLMs. InThe Fourteenth International Conference on Learning Representations, 2026

    2026

  9. [17]

    Redcode: Risky code execution and generation benchmark for code agents

    Chengquan Guo, Xun Liu, Chulin Xie, Andy Zhou, Yi Zeng, Zinan Lin, Dawn Song, and Bo Li. Redcode: Risky code execution and generation benchmark for code agents. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024

    2024

  10. [18]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  11. [19]

    Getting pwn’d by ai: Penetration testing with large language models

    Andreas Happe and Jürgen Cito. Getting pwn’d by ai: Penetration testing with large language models. InProceedings of the 31st ACM joint european software engineering conference and symposium on the foundations of software engineering, pages 2082–2086, 2023

    2023

  12. [20]

    Llms as hackers: Autonomous linux privilege escalation attacks.Empirical Software Engineering, 31(3):70, 2026

    Andreas Happe, Aaron Kaplan, and Juergen Cito. Llms as hackers: Autonomous linux privilege escalation attacks.Empirical Software Engineering, 31(3):70, 2026

    2026

  13. [21]

    Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning

    Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical datas...

    2026

  14. [22]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jia- jun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  15. [23]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  16. [24]

    Agentic ai for cyber defense: Llm-guided hierarchical multi-agent reinforcement learning

    Guangyu Jiang, Mahdi Imani, Nathaniel D Bastian, and Tian Lan. Agentic ai for cyber defense: Llm-guided hierarchical multi-agent reinforcement learning. InMILCOM 2025-2025 IEEE Military Communications Conference (MILCOM), pages 1518–1523. IEEE, 2025

    2025

  17. [25]

    Search-r1: Training LLMs to reason and leverage search engines with reinforcement learning

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan O Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training LLMs to reason and leverage search engines with reinforcement learning. InSecond Conference on Language Modeling, 2025. 11

    2025

  18. [26]

    Exploring the efficacy of multi-agent reinforcement learning for autonomous cyber defence: A cage challenge 4 perspective

    Mitchell Kiely, Metin Ahiskali, Etienne Borde, Benjamin Bowman, David Bowman, Dirk Van Bruggen, KC Cowan, Prithviraj Dasgupta, Erich Devendorf, Ben Edwards, et al. Exploring the efficacy of multi-agent reinforcement learning for autonomous cyber defence: A cage challenge 4 per...

    2025

  19. [27]

    Automated cyber defense with generalizable graph-based reinforcement learning agents.arXiv preprint arXiv:2509.16151, 2025

    Isaiah J King, Benjamin Bowman, and H Howie Huang. Automated cyber defense with generalizable graph-based reinforcement learning agents.arXiv preprint arXiv:2509.16151, 2025

    2025

  20. [28]

    Large language models are zero-shot reasoners.Advances in neural information processing systems, 35:22199–22213, 2022

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners.Advances in neural information processing systems, 35:22199–22213, 2022

    2022

  21. [29]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating System...

    2023

  22. [30]

    In-the-flow agentic system optimization for effective planning and tool use

    Zhuofeng Li, Haoxiang Zhang, Seungju Han, Sheng Liu, Jianwen Xie, Yu Zhang, Yejin Choi, James Zou, and Pan Lu. In-the-flow agentic system optimization for effective planning and tool use. InThe Fourteenth International Conference on Learning Representations, 2026

    2026

  23. [31]

    Code as policies: Language model programs for embodied control

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In2023 IEEE International conference on robotics and automation (ICRA), pages 9493–9500. IEEE, 2023

    2023

  24. [32]

    Et-bert: A contextualized datagram representation with pre-training transformers for encrypted traffic classification

    Xinjie Lin, Gang Xiong, Gaopeng Gou, Zhen Li, Junzheng Shi, and Jing Yu. Et-bert: A contextualized datagram representation with pre-training transformers for encrypted traffic classification. InProceedings of the ACM Web Conference 2022, pages 633–642, 2022

    2022

  25. [33]

    Cyberbench: A multi-task benchmark for evaluating large language models in cybersecurity

    Zefang Liu, Jialei Shi, and John F Buford. Cyberbench: A multi-task benchmark for evaluating large language models in cybersecurity. InAAAI 2024 Workshop on Artificial Intelligence for Cyber Security, 2024

    2024

  26. [34]

    Visual-rft: Visual reinforcement fine-tuning

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2034–2044, 2025

    2025

  27. [35]

    Contrasting centralized and decentralized critics in multi-agent reinforcement learning

    Xueguang Lyu, Yuchen Xiao, Brett Daley, and Christopher Amato. Contrasting centralized and decentralized critics in multi-agent reinforcement learning. InProceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’21, page 844–852, Rich...

    2021

  28. [36]

    Eureka: Human-level reward design via coding large language models

    Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  29. [37]

    Ray: A distributed framework for emerging {AI} applications

    Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, et al. Ray: A distributed framework for emerging {AI} applications. In13th USENIX symposium on operating systems design and ...

    2018

  30. [38]

    Experience with emerald to date

    Peter G Neumann and Phillip A Porras. Experience with emerald to date. InWorkshop on Intrusion Detection and Network Monitoring, pages 73–80, 1999

    1999

  31. [39]

    Towards a high fidelity training environment for autonomous cyber defense agents

    Sean Oesch, Amul Chaulagain, Brian Weber, Matthew Dixson, Amir Sadovnik, Benjamin Roberson, Cory Watson, and Phillipe Austria. Towards a high fidelity training environment for autonomous cyber defense agents. InProceedings of the 17th Cyber Security Experimentation and Test Wo...

    2024

  32. [40]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  33. [41]

    Nyu ctf bench: A scalable open-source benchmark dataset for evaluating llms in offensive security

    Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, haoran xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, and Muhammad Shafique. Nyu ctf bench: A scalable open-source benchmark dataset for eval...

    2024

  34. [42]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  35. [43]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, page 1279–1297, New Yor...

    2025

  36. [44]

    Hierarchical multi-agent reinforcement learning for cyber network defense.Reinforcement Learning Journal, 6:790–810, 2025

    Aditya Vikram Singh, Ethan Rathbun, Emma Graham, Lisa Oakley, Simona Boboila, Peter Chin, and Alina Oprea. Hierarchical multi-agent reinforcement learning for cyber network defense.Reinforcement Learning Journal, 6:790–810, 2025

    2025

  37. [45]

    A taxonomy of intrusion response systems

    Natalia Stakhanova, Samik Basu, and Johnny Wong. A taxonomy of intrusion response systems. Int. J. Inf. Comput. Secur., 1(1/2):169–184, January 2007

    2007

  38. [46]

    Redsage: A cybersecurity generalist LLM

    Naufal Suryanto, Muzammal Naseer, Pengfei Li, Syed Talal Wasim, Jinhui Yi, Juergen Gall, Paolo Ceravolo, and Ernesto Damiani. Redsage: A cybersecurity generalist LLM. InThe Fourteenth International Conference on Learning Representations, 2026

    2026

  39. [47]

    Cyberbattlesim, 2021

    Microsoft Defender Research Team. Cyberbattlesim, 2021. Created by Christian Seifert, Michael Betser, William Blum, James Bono, Kate Farris, Emily Goren, Justin Grana, Kris- tian Holsheimer, Brandon Marken, Joshua Neil, Nicole Nichols, Jugal Parikh, Haoran Wei

    2021

  40. [48]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024

    2024

  41. [49]

    Cyberseceval 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models

    Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman, Stephanie Ding, et al. Cyberseceval 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models. arXiv...

    2024 arXiv

  42. [50]

    Sppo: Sequence-level ppo for long-horizon reasoning tasks.arXiv preprint arXiv:2604.08865, 2026

    Tianyi Wang, Yixia Li, Long Li, Yibiao Chen, Shaohan Huang, Yun Chen, Peng Li, Yang Liu, and Guanhua Chen. Sppo: Sequence-level ppo for long-horizon reasoning tasks.arXiv preprint arXiv:2604.08865, 2026

    2026 arXiv

  43. [51]

    SymRTLO: Enhancing RTL code optimization with LLMs and neuron-inspired symbolic reasoning

    Yiting Wang, Wanghao Ye, Ping Guo, Yexiao He, Ziyao Wang, Bowei Tian, Shwai He, Guoheng Sun, Zheyu Shen, Sihan Chen, Ankur Srivastava, Qingfu Zhang, Gang Qu, and Ang Li. SymRTLO: Enhancing RTL code optimization with LLMs and neuron-inspired symbolic reasoning. InThe Thirty-nin...

    2026

  44. [52]

    Cyber- gym: Evaluating AI agents’ real-world cybersecurity capabilities at scale

    Zhun Wang, Tianneng Shi, Jingxuan He, Matthew Cai, Jialin Zhang, and Dawn Song. Cyber- gym: Evaluating AI agents’ real-world cybersecurity capabilities at scale. InThe Fourteenth International Conference on Learning Representations, 2026

    2026

  45. [53]

    SWE-RL: Advancing LLM reasoning via reinforcement learning on open software evolution

    Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, LINGMING ZHANG, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida Wang. SWE-RL: Advancing LLM reasoning via reinforcement learning on open software evolution. InThe Thirty-ninth Annual Conference on Neural I...

    2025

  46. [54]

    Autogen: Enabling next-gen LLM applications via multi-agent conversations

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi-agent conversations. InFirst Confer...

    2024

  47. [55]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  48. [56]

    Intercode: Standardizing and benchmarking interactive coding with execution feedback.Advances in Neural Information Processing Systems, 36:23826–23854, 2023

    John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback.Advances in Neural Information Processing Systems, 36:23826–23854, 2023

    2023

  49. [57]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InThe eleventh international conference on learning representations, 2022

    2022

  50. [58]

    Primus: A pioneering collection of open-source datasets for cybersecurity LLM training

    Yao-Ching Yu, Tsun-Han Chiang, Cheng-Wei Tsai, Chien-Ming Huang, and Wen-Kwang Tsao. Primus: A pioneering collection of open-source datasets for cybersecurity LLM training. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors,Pro- ceedings...

    2025

  51. [59]

    ACECODER: Acing coder RL via automated test-case synthesis

    Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. ACECODER: Acing coder RL via automated test-case synthesis. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Proceedings of the 63rd Annual Meeting of the Ass...

    2025

  52. [60]

    Ho, and Percy Liang

    Andy K Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Julian Jasper, Pura Peetathawatchai, Ari Glenn, Vikram Sivashankar, Daniel Zamoshchin, Leo Glikbarg, Derek Askaryar, Haoxiang Yang, Aolin Zhang, R...

    2025

  53. [61]

    Abdi, William Blum, and Muhammad Abdul-Mageed

    Chiyu Zhang, Marc-Alexandre Côté, Michael Albada, Anush Sankaran, Jack W Stokes, Tong Wang, Amir H. Abdi, William Blum, and Muhammad Abdul-Mageed. Defenderbench: A toolkit for evaluating language agents in cybersecurity environments. InWorkshop on Scaling Environments for Agents, 2025

    2025

  54. [62]

    Yet another traffic classifier: A masked autoencoder based traffic transformer with multi- level flow representation

    Ruijie Zhao, Mingwei Zhan, Xianwen Deng, Yanhao Wang, Yijun Wang, Guan Gui, and Zhi Xue. Yet another traffic classifier: A masked autoencoder based traffic transformer with multi- level flow representation. InProceedings of the AAAI Conference on Artificial Intelligence, volum...

    2023

  55. [63]

    Curran Associates Inc., Red Hook, NY , USA, 2019

    Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu.Devign: effective vulnerability identification by learning comprehensive program semantics via graph neural networks. Curran Associates Inc., Red Hook, NY , USA, 2019

    2019

  56. [64]

    More than just functional: LLM-as-a-critique for efficient code generation

    Derui Zhu, Dingfan Chen, jinfu chen, Jens Grossklags, Walter Pretschner, and Weiyi Shang. More than just functional: LLM-as-a-critique for efficient code generation. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026

    2026

  57. [65]

    CVE-bench: A benchmark for AI agents’ ability to exploit real-world web application vulnerabilities

    Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang, Conner Jensen, Eric Ihli, Jason Benn, Jet Geronimo, Avi Dhir, Sudhit Rao, Kaicheng Yu, Twm Stone, and Daniel Kang. CVE-bench: A benchmark for AI agents’ ability to exploit real-worl...

    2025

  58. [66]

    Teams of LLM agents can exploit zero-day vulnerabilities

    Yuxuan Zhu, Antony Kellermann, Akul Gupta, Philip Li, Richard Fang, Rohan Bindu, and Daniel Kang. Teams of LLM agents can exploit zero-day vulnerabilities. In Vera Demberg, Kentaro Inui, and Lluís Marquez, editors,Proceedings of the 19th Conference of the European Chapter of t...

    2026

  59. [67]

    Cyber-zero: Training cybersecurity agents without runtime

    Terry Yue Zhuo, Dingmin Wang, Hantian Ding, Varun Kumar, and Zijian Wang. Cyber-zero: Training cybersecurity agents without runtime. InThe Fourteenth International Conference on Learning Representations, 2026. A Environment Setup & Reward Design This section provides a compreh...

    2026

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.