pith. sign in

arxiv: 2605.17075 · v1 · pith:YCNZFSXZnew · submitted 2026-05-16 · 💻 cs.CR

A Red Teaming Framework for Evaluating Robustness of AI-enabled Security Orchestration, Automation, and Response Systems

Ayan Javeed Shaikh , Nathaniel D. Bastian , Ankit Shah This is my paper

Pith reviewed 2026-05-20 15:19 UTC · model grok-4.3

classification 💻 cs.CR
keywords red teamingSOAR systemslarge language modelsreinforcement learningcybersecurityautonomous agentsattack simulationAI robustness
0
0 comments X

The pith

A hybrid framework pairing large language models for strategy with reinforcement learning for tactics generates sustained multi-stage attacks against AI security defenders.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces an autonomous red teaming framework that combines large language models for high-level planning with reinforcement learning for low-level execution to probe the robustness of AI-enabled security orchestration, automation, and response systems. The design uses a hierarchical structure and reward signals tied to cyber kill-chain stages to produce adaptive attack sequences in enterprise network simulations. Results indicate that this hybrid method achieves higher compromise levels than either standalone language models, which struggle to maintain multi-stage campaigns, or domain-specific cybersecurity models, which reach only limited success. If the evaluation holds, the work implies that effective red teaming of autonomous defenders requires integrated LLM-RL techniques rather than single-technology approaches.

Core claim

The paper establishes that a hierarchical red teaming framework integrating an LLM-based planner for strategic intent with an RL controller for tactical execution, supported by kill-chain-aligned reward shaping, produces adaptive multi-stage attack campaigns that succeed against autonomous defenders in high-fidelity enterprise simulations, whereas standalone LLM agents fail to sustain such campaigns and domain-specific models achieve only limited compromise.

What carries the argument

The hierarchical LLM-RL architecture, with an LLM planner handling strategic intent and an RL controller managing tactical actions via reward shaping aligned to kill-chain progression.

If this is right

  • Standalone LLM agents cannot sustain multi-stage attack campaigns in the tested environment.
  • Domain-specific cybersecurity models reach only limited levels of compromise against the simulated defenders.
  • Hybrid LLM-RL methods are required to achieve effective autonomous red teaming of AI security systems.
  • Reward shaping based on kill-chain progression enables the RL component to support longer attack sequences.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If simulation fidelity can be increased, the framework might transfer to testing real deployed SOAR systems without major redesign.
  • The observed gaps in standalone agents could guide the creation of hybrid defender architectures that combine language-based reasoning with learned policies.
  • The kill-chain reward approach might extend to red teaming other autonomous AI systems outside cybersecurity, such as robotic or logistics agents.

Load-bearing premise

The high-fidelity enterprise simulation accurately models real-world enterprise networks, adaptive adversaries, and defender behaviors so that performance differences generalize beyond the simulated setting.

What would settle it

Running the same hybrid framework and baseline agents against a live, non-simulated enterprise network equipped with actual AI-enabled SOAR systems and measuring whether the hybrid still produces measurably higher compromise rates than the baselines.

Figures

Figures reproduced from arXiv: 2605.17075 by Ankit Shah, Ayan Javeed Shaikh, Nathaniel D. Bastian.

Figure 1
Figure 1. Figure 1: Hierarchical LLM-RL red teaming framework architecture implemented in the CybORG CAGE 4 [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: MITRE ATT&CK kill chain progression in CAGE 4. Each stage requires completion of the [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Non-monotonic scaling in the Qwen3 model family. Left axis: episodes achieving compromise (out [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
read the original abstract

AI-enabled Security Orchestration, Automation, and Response (SOAR) systems increasingly employ autonomous agents for cyber defense, yet their resilience to adaptive adversaries is underexplored. We introduce an autonomous red teaming framework that integrates large language models (LLMs) with reinforcement learning (RL) to generate adaptive, multi-stage attack campaigns against autonomous defenders in enterprise networks. A hierarchical design combines an LLM-based planner for strategic intent with an RL controller for tactical execution, supported by reward shaping aligned with kill-chain progression. Evaluation in a high-fidelity enterprise simulation demonstrates the effectiveness of the proposed approach, while also showing that standalone LLM agents fail to sustain multi-stage attack campaigns and that domain-specific cybersecurity models achieve only limited levels of compromise, highlighting the necessity for hybrid LLM-RL approaches to red teaming.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The manuscript introduces an autonomous red teaming framework integrating large language models (LLMs) with reinforcement learning (RL) for generating adaptive, multi-stage attack campaigns against AI-enabled SOAR systems in enterprise networks. A hierarchical design pairs an LLM-based planner for strategic intent with an RL controller for tactical execution, using reward shaping aligned with kill-chain progression. Evaluation in a high-fidelity enterprise simulation is claimed to demonstrate the hybrid approach's effectiveness, while showing that standalone LLM agents fail to sustain multi-stage campaigns and domain-specific cybersecurity models achieve only limited compromise, thereby highlighting the necessity of hybrid LLM-RL methods.

Significance. If the simulation results are shown to be robust, the work could meaningfully advance evaluation of robustness in autonomous cyber-defense systems by providing a concrete method for testing against adaptive adversaries and identifying failure modes in pure LLM or domain-specific approaches. The emphasis on hybrid architectures and kill-chain-aligned rewards offers a structured way to probe SOAR resilience.

major comments (1)
  1. [Evaluation] Evaluation section (and associated results): The central claim that the hybrid LLM-RL approach is necessary rests on performance gaps observed in the high-fidelity enterprise simulation (standalone LLMs fail to sustain campaigns; domain models achieve limited compromise). However, the manuscript supplies no description of how the simulation was constructed, calibrated against real telemetry or network traces, or validated (e.g., via expert review or sensitivity analysis to defender heuristics). Without such grounding, the reported gaps could be artifacts of the chosen network topology, reward shaping, or defender policies rather than intrinsic limitations, undermining generalization to real-world SOAR systems.
minor comments (2)
  1. Clarify the precise simulation platform, observation model, and state-space representation used for the enterprise network; this would aid reproducibility and allow readers to assess fidelity.
  2. The abstract and introduction should explicitly state the number of independent runs, statistical significance tests, and any error bars or variance measures accompanying the reported compromise levels and campaign success rates.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their constructive feedback, which helps us improve the clarity and rigor of our work. We address the major comment on the evaluation section below.

read point-by-point responses

  1. Referee: [Evaluation] Evaluation section (and associated results): The central claim that the hybrid LLM-RL approach is necessary rests on performance gaps observed in the high-fidelity enterprise simulation (standalone LLMs fail to sustain campaigns; domain models achieve limited compromise). However, the manuscript supplies no description of how the simulation was constructed, calibrated against real telemetry or network traces, or validated (e.g., via expert review or sensitivity analysis to defender heuristics). Without such grounding, the reported gaps could be artifacts of the chosen network topology, reward shaping, or defender policies rather than intrinsic limitations, undermining generalization to real-world SOAR systems.

    Authors: We agree that additional explicit details on simulation construction, calibration, and validation are needed to fully support generalization claims and rule out artifacts. In the revised manuscript we will add a dedicated subsection in the Evaluation section describing: the enterprise network topology (modeled on standard reference architectures with concrete host/service counts and vulnerability distributions); the data generation and calibration process (using synthetic telemetry aligned with publicly documented network traces and benchmarks); validation steps (including internal consistency checks and consultation with domain experts); and sensitivity analysis over defender policy variations. These additions will directly address the concern while preserving the reported performance gaps as evidence for the hybrid approach. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical simulation evaluation is self-contained

full rationale

The paper proposes a hierarchical LLM-RL red teaming framework and reports performance differences observed inside a high-fidelity enterprise simulation. No equations, derivations, fitted parameters, or self-referential definitions appear in the provided text. Central claims rest on empirical outcomes rather than any reduction of a prediction to its own inputs by construction, self-citation load-bearing, or imported uniqueness theorems. The evaluation is presented as direct experimental evidence, satisfying the criteria for a non-circular, self-contained result.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only the abstract is available, so no explicit free parameters, axioms, or invented entities can be extracted or audited from the full text.

pith-pipeline@v0.9.0 · 5676 in / 1074 out tokens · 42666 ms · 2026-05-20T15:19:36.691590+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

24 extracted references · 24 canonical work pages · 2 internal anchors

  1. [1]

    Cyber security report 2026, 2026

    Check Point Software Technologies Ltd. Cyber security report 2026, 2026. URLhttps://www. checkpoint.com/security-report/. Accessed: 2026-03-11

    work page 2026

  2. [2]

    A survey on agentic security: Applications, threats and defenses, 2025

    Asif Shahriar, Md Nafiu Rahman, Sadif Ahmed, Farig Sadeque, and Md Rizwan Parvez. A survey on agentic security: Applications, threats and defenses.arXiv preprint arXiv:2510.06445, 2025

    work page arXiv 2025

  3. [3]

    Large language models for cyber security: A systematic literature review.ACM Transactions on Software Engineering and Methodology

    HanXiang Xu, ShenAo Wang, Ningke Li, Kailong Wang, Yanjie Zhao, Kai Chen, Ting Yu, Yang Liu, and HaoYu Wang. Large language models for cyber security: A systematic literature review.ACM Transactions on Software Engineering and Methodology

    work page

  4. [4]

    MITRE ATT&CK: Adversarial tactics, techniques, and common knowledge

    The MITRE Corporation. MITRE ATT&CK: Adversarial tactics, techniques, and common knowledge. https://attack.mitre.org/, 2024

    work page 2024

  5. [5]

    Longjing Yang, Ayong Ye, Yuanhuang Liu, Wenting Lu, and Chuang Huang. Llm-aptds: A high- precision advanced persistent threat detection system for imbalanced data based on large language models with strong interpretabilit.Future Generation Computer Systems, page 108315, 2025

    work page 2025

  6. [6]

    Llama-3.1-foundationai-securityllm- base-8b technical report.arXiv preprint arXiv:2504.21039, 2025

    Paul Kassianik, Baturay Saglam, Alexander Chen, Blaine Nelson, Anu Vellore, Massimo Aufiero, Fraser Burch, Dhruv Kedia, Avi Zohary, Sajana Weerawardhena, et al. Llama-3.1-foundationai-securityllm- base-8b technical report.arXiv preprint arXiv:2504.21039, 2025

    work page arXiv 2025

  7. [7]

    Generative artificial intelligence tools for red teams

    Cameron Thomas Stark. Generative artificial intelligence tools for red teams. Technical report, Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States), 2024. 13 A Red Teaming Framework for Evaluating Robustness of AI-enabled Security Orchestration, Automation, and Response Systems A Preprint

    work page 2024

  8. [8]

    Deep packgen: A deep reinforcement learning framework for adversarial network packet generation

    Soumyadeep Hore, Jalal Ghadermazi, Diwas Paudel, Ankit Shah, Tapas Das, and Nathaniel Bastian. Deep packgen: A deep reinforcement learning framework for adversarial network packet generation. ACM Transactions on Privacy and Security, 28(2):1–33, 2025

    work page 2025

  9. [9]

    Large language models are autonomous cyber defenders

    Sebasti´ an R Castro, Roberto Campbell, Nancy Lau, Octavio Villalobos, Jiaqi Duan, and Alvaro A Cardenas. Large language models are autonomous cyber defenders. In2025 IEEE Conference on Artificial Intelligence (CAI), pages 1125–1132. IEEE, 2025

    work page 2025

  10. [10]

    Reflexion: Language agents with verbal reinforcement learning.Advances in neural information processing systems, 36:8634–8652, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in neural information processing systems, 36:8634–8652, 2023

    work page 2023

  11. [11]

    Cyborg: An autonomous cyber operations research gym.arXiv preprint arXiv:2002.10667, 2020

    Callum Baillie, Maxwell Standen, Jonathon Schwartz, Michael Docking, David Bowman, and Junae Kim. Cyborg: An autonomous cyber operations research gym.arXiv preprint arXiv:2002.10667, 2020

    work page arXiv 2002

  12. [12]

    Exploring the efficacy of multi-agent reinforcement learning for autonomous cyber defence: A cage challenge 4 perspective

    Mitchell Kiely, Metin Ahiskali, Etienne Borde, Benjamin Bowman, David Bowman, Dirk Van Bruggen, KC Cowan, Prithviraj Dasgupta, Erich Devendorf, Ben Edwards, et al. Exploring the efficacy of multi-agent reinforcement learning for autonomous cyber defence: A cage challenge 4 perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volu...

    work page 2025

  13. [13]

    Hierarchical multi-agent reinforcement learning for cyber network defense

    Aditya Vikram Singh, Ethan Rathbun, Emma Graham, Lisa Oakley, Simona Boboila, Peter Chin, and Alina Oprea. Hierarchical multi-agent reinforcement learning for cyber network defense. InProceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, pages 2747–2749, 2025

    work page 2025

  14. [14]

    CyberBattleSim: An experimentation and research platform for autonomous cyber agents.https://github.com/microsoft/CyberBattleSim, 2021

    Microsoft Defender Research Team. CyberBattleSim: An experimentation and research platform for autonomous cyber agents.https://github.com/microsoft/CyberBattleSim, 2021

    work page 2021

  15. [15]

    Network environment design for autonomous cyberdefense.arXiv preprint arXiv:2103.07583, 2021

    Andres Molina-Markham, Cory Miniter, Becky Powell, and Ahmad Ridley. Network environment design for autonomous cyberdefense.arXiv preprint arXiv:2103.07583, 2021

    work page arXiv 2021

  16. [16]

    In33rd USENIX Security Symposium (USENIX Security 24), pages 847–864, 2024

    Gelei Deng, Yi Liu, V´ ıctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass.{PentestGPT}: Evaluating and harnessing large language models for automated penetration testing. In33rd USENIX Security Symposium (USENIX Security 24), pages 847–864, 2024

    work page 2024

  17. [17]

    Muzsai, D

    Lajos Muzsai, David Imolai, and Andr´ as Luk´ acs. Hacksynth: Llm agent and evaluation framework for autonomous penetration testing.arXiv preprint arXiv:2412.01778, 2024

    work page arXiv 2024

  18. [18]

    Out of the cage: How stochastic parrots win in cyber security environments.arXiv preprint arXiv:2308.12086, 2023

    Maria Rigaki, Ondˇ rej Luk´ aˇ s, Carlos A Catania, and Sebastian Garcia. Out of the cage: How stochastic parrots win in cyber security environments.arXiv preprint arXiv:2308.12086, 2023

    work page arXiv 2023

  19. [19]

    Toward cybersecurity-expert small language models.arXiv preprint arXiv:2510.14113, 2025

    Matan Levi, Daniel Ohayon, Ariel Blobstein, Ravid Sagi, Ian Molloy, and Yair Allouche. Toward cybersecurity-expert small language models.arXiv preprint arXiv:2510.14113, 2025. IBM Research

    work page arXiv 2025

  20. [20]

    Automated cyber defense with generalizable graph-based reinforcement learning agents.arXiv preprint arXiv:2509.16151, 2025

    Isaiah J King, Benjamin Bowman, and H Howie Huang. Automated cyber defense with generalizable graph-based reinforcement learning agents.arXiv preprint arXiv:2509.16151, 2025

    work page arXiv 2025

  21. [21]

    Large language model integration with reinforcement learning to augment decision-making in autonomous cyber operations.arXiv preprint arXiv:2509.05311, 2025

    Konur Tholl, Fran¸ cois Rivest, Mariam El Mezouar, Adrian Taylor, and Ranwa Al Mallah. Large language model integration with reinforcement learning to augment decision-making in autonomous cyber operations.arXiv preprint arXiv:2509.05311, 2025

    work page arXiv 2025

  22. [22]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  23. [23]

    High-Dimensional Continuous Control Using Generalized Advantage Estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. InProc. Int. Conf. Learning Representations (ICLR), 2016. arXiv:1506.02438. 14 A Red Teaming Framework for Evaluating Robustness of AI-enabled Security Orchestration, Automation, and Response Systems ...

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  24. [24]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gon- zalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th symposium on operating systems principles, pages 611–626, 2023. 15

    work page 2023