REVIEW 4 major objections 7 minor 2 cited by
Large Language Models are Autonomous Cyber Defenders
T0 review · 4 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read This paper claims that pre-trained large language models, without reinforcement learning or security-specific fine-tuning, can act as blue-team defenders in a multi-agent cyber defense environment and coordinate with…
desk verdict First real integration of LLM defenders into multi-agent CybORG, but the communication-protocol claim is unmeasured for want of an ablation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the LLM adapter pipeline that converts CybORG's numeric observation vectors into natural-language observations, feeds them with a role-based prompt to an LLM, and parses the model's JSON response into a defender action. On top of that sits the proposed 8-bit communication protocol: bits 0-4 flag malicious activity detected from each of the other agents' networks, bits 5-6 encode the compromise level in the sender's own subnet (no compromise, scan or remote exploit, user-level compromise, admin-level compromise), and bit 7 marks whether the sender is busy. This vector is what the paper claims lets heterogeneous defenders coordinate, and it is the object whose contribution to team performance is asserted rather than isolated.
What would settle it
Run the same all-LLM, all-RL, and hybrid teams with the communication vector disabled (for example, always broadcasting zeros) and compare rewards and action patterns; if removing the protocol leaves team performance and coordination unchanged, the claim that LLM and RL agents communicate effectively through it is unsupported.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that a pre-trained LLM can be placed in a multi-agent autonomous cyber defense environment and behave as a coherent blue-team operator: it parses natural-language observations, chooses among defender actions, and responds to messages from other defenders with security-relevant actions such as removing malicious processes or blocking traffic. The authors argue that when all five blue agents are LLM-driven, the team exhibits a recognizable defensive posture—preventive decoy deployment, analysis of repeated low-severity connections, and quarantine responses to remote alerts—even though it accumulates more availability penalties than the reinforcement-learning baseline. They further report that a hybrid team with one LLM and four RL agents is feasible and that the LLM's decisions can be clustered into four recurring reasoning patterns. The paper concludes by asserting that LLM ACD agents can communicate effectively when they share a communication protocol and that together they can protect a network with a security reasoning similar to a team of security operators.
Load-bearing premise
The paper's central coordination claim rests on the assumption that its 8-bit communication vector actually carries useful information between defenders, yet every reported team uses the protocol, so its contribution to the observed behavior is never measured.
Editorial extensions
If this is right
- LLM defenders can be added to an existing RL-based autonomous cyber defense system without retraining the LLM; only the prompt and observation formatting need to change.
- A mixed team of one LLM and four RL agents is operational in the CAGE 4 scenario, demonstrating that a shared communication protocol can bridge heterogeneous agent types.
- Because LLM reasoning traces are text, defender strategy can be inspected and modified through prompt design, which the paper suggests could eventually reproduce or replace trained RL policies.
- The paper's reward comparisons quantify the current gap: the all-RL team averages around -493 reward, the all-LLM team around -2547, and the LLM's action selection is about 104 times slower, so real-time deployment is not yet practical.
Reading between the lines
- If the protocol's coordination value were confirmed by ablation, a natural next step would be to let the communication vector be learned or negotiated by the agents rather than fixed by hand, since the fixed encoding is currently a design assumption.
- The paper's observed misreads of the vector (an agent attributing a message to the wrong sender) suggest that future work should test whether simpler encodings—or natural-language messages instead of bitfields—reduce coordination errors.
- The higher reward achieved by the slower reasoning model hints that the LLM's security reasoning quality, not raw speed, is the lever that matters; a faster reasoning model or distilled expert could close the runtime gap.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper claims to be the first study of LLMs in multi-agent autonomous cyber defense. It adapts CybORG CAGE 4 so that one or more blue-team agents are driven by an LLM through a natural-language observation formatter and a JSON action parser, and it introduces an 8-bit communication vector by which defenders share compromise levels, remote detections, and availability. Experiments compare no-defender, all-LLM, all-RL (Cybermonic KEEP), and mixed 1LLM+4RL teams against several finite-state red agents, with two 500-step episodes per condition, and use clustering of LLM reasoning logs to characterize defensive strategies. The paper concludes that LLM agents can communicate effectively with the protocol and exhibit security reasoning similar to human operators.
Significance. If the findings survive the added controls, the paper contributes a useful open-source integration of LLM agents into CybORG CAGE 4, a multi-agent ACD testbed, and provides a concrete 8-bit coordination protocol plus qualitative reasoning traces. Its strengths include a clearly described adapter architecture, use of an established external RL baseline (Cybermonic KEEP), and a transparent discussion of hallucinations and prompt influence. The qualitative action-clustering analysis is a reasonable first attempt to inspect LLM security reasoning. However, the protocol's contribution is not measured, and the quantitative evaluation is too under-powered to support the central claims; with those addressed, the paper would be a valuable resource for the ACD community.
major comments (4)
- [IV, Scenarios; Fig. 5] The central quantitative comparison between the all-LLM and all-RL teams (Fig. 5; 'For all our experiments, we run 2 episodes') rests on two episodes per configuration and no significance test. Since temperature is set to 1, episode-level variance is expected to be large, so the reported means and standard deviations are not sufficient to support the conclusion that RL agents perform better against diverse red adversaries. Report repeated runs with more seeds and include confidence intervals or a statistical test.
- [III-C; VI] The 8-bit communication protocol is never ablated: every reported team configuration uses it, and the LLM prompt instructs agents to read the Communication Vectors. The conclusion that 'LLM ACD agents can communicate effectively when they share a communication protocol' is therefore not directly tested, because the coordinated behavior could be driven by local observations, the prompt's action descriptions, or the protocol. Add a no-protocol control condition and a quantitative measure of protocol usage/parse accuracy, especially since Section IV documents the LLM misreading the vector (confusing agent 3 and agent 4, and misreading compromise levels).
- [IV, Figs. 4-5] The all-LLM versus all-RL comparison uses GPT-4o-mini, which Fig. 4 identifies as the lowest-reward OpenAI model in the 1LLM+4RL configuration; o3-mini achieves the highest reward. The headline claim comparing 'LLM agents' with RL agents is thus not representative of the best LLM configuration tested. Either include the best-performing LLM in the comparison or explicitly restrict the claim to GPT-4o-mini.
- [III-D, Table III] The final prompt is selected by comparing average rewards on the default CAGE 4 red agent, and the same reward signal is then the main evaluation metric in Section IV. This creates a selection effect that should be acknowledged; reporting results for the alternative prompt variants, or at least stating that the choice was made on a small tuning set, would let readers judge the sensitivity.
minor comments (7)
- [I] The sentence 'are not all publicly accessible, actively maintained, or are limited to to particular adversarial scenarios' contains a duplicated 'to'.
- [IV, Scenarios] The sentence 'we protect 3 subnets with 1 LLM agent (i.e., blue agent 4) and 4 subnets with 4 RL (KEEP) agents' is confusing given that the architecture has five blue agents; clarify how the five agents map to subnets in this scenario.
- [III-D, Table III] The table header lists 'Instruct' twice; label the three prompt conditions (Instructional, Few-Shot, Role) explicitly.
- [IV, Environment and models] The phrase 'and the the latest DeepSeek model' has a duplicated 'the'.
- [IV, Performance] The phrase 'GPT-4o-mini had the quickest execution time and but low reward' should read 'and the lowest reward'.
- [IV, Reasoning for Action Selection] The reference [31] to Weber and Morris's sorption kinetics paper is not a standard PCA citation; please cite a canonical PCA reference instead.
- [VI] The statement that the team can 'protect a network with a security reasoning similar to a team of security operators' is not operationalized; consider removing it or specifying which comparison supports it.
Circularity Check
No circular derivation: the empirical claims are benchmarked against external KEEP and no-defender baselines; the unablated communication protocol is a methodological gap, not a circular reduction.
full rationale
This paper is an empirical evaluation rather than an analytical derivation, and its main comparisons are against external references: Cybermonic's KEEP RL baseline [16], the CAGE 4 environment [13], and adversary policies from Singh et al. [14]. No equation is shown to be equivalent to its own input. The prompt design is tuned on CAGE 4 reward in Table III, which is a mild form of fitting, but the chosen prompt is then evaluated against independent baselines and diverse red agents, so the reported rewards and action analyses are not forced by that tuning. The claim that LLM ACD agents 'can communicate effectively' (Section VI) is supported by logged action-reason clusters (Table IV), not by construction; the communication protocol in Section III-C is a proposed mechanism whose behavior is measured, not assumed. The paper openly reports limitations in Section V, including hallucinated communication-vector readings and prompt-induced avoidance of Restore, which weaken the strength of the conclusions but do not make them circular. The only self-citation ([2]) supports a background statement about rising attack sophistication and is not load-bearing. The absence of a no-protocol ablation is a completeness gap, not a circularity: the protocol's contribution is unmeasured, but no result reduces to its own input by definition. Overall circularity score is therefore minimal.
Assumptions & free parameters
free parameters (1)
- Number of reason clusters K =
4
assumptions (4)
- domain assumption CAGE 4 is a valid and representative ACD testbed
- domain assumption LLM outputs can be converted to valid environment actions; invalid outputs are logged and replaced by Sleep
- domain assumption The 8-bit communication vector is transmitted without loss between agents
- domain assumption General-purpose LLMs can ground natural-language observations into security-relevant action selection
invented entities (1)
-
8-bit Communication Vector protocol
Cite this review
Pith. "Pith review of Large Language Models are Autonomous Cyber Defenders." pith.science (2026). https://pith.science/paper/CZBRSMDZ
@misc{pith2026250504843,
author = {Pith},
title = {Pith review of: Large Language Models are Autonomous Cyber Defenders},
year = {2026},
howpublished = {\url{https://pith.science/paper/CZBRSMDZ}},
note = {Machine review of arXiv:2505.04843}
}
read the original abstract
Fast and effective incident response is essential to prevent adversarial cyberattacks. Autonomous Cyber Defense (ACD) aims to automate incident response through Artificial Intelligence (AI) agents that plan and execute actions. Most ACD approaches focus on single-agent scenarios and leverage Reinforcement Learning (RL). However, ACD RL-trained agents depend on costly training, and their reasoning is not always explainable or transferable. Large Language Models (LLMs) can address these concerns by providing explainable actions in general security contexts. Researchers have explored LLM agents for ACD but have not evaluated them on multi-agent scenarios or interacting with other ACD agents. In this paper, we show the first study on how LLMs perform in multi-agent ACD environments by proposing a new integration to the CybORG CAGE 4 environment. We examine how ACD teams of LLM and RL agents can interact by proposing a novel communication protocol. Our results highlight the strengths and weaknesses of LLMs and RL and help us identify promising research directions to create, train, and deploy future teams of ACD agents.
Figures
Figures from the paper (5 more)
Forward citations
Cited by 2 Pith papers
-
Online Incident Response Planning under Model Misspecification through Bayesian Learning and Belief Quantization
MOBAL learns a model of an ongoing cyberattack with Bayesian updates and computes incident responses with a quantized version of that model, giving robustness to model misspecification on CAGE-2.
-
Architectural Backdoors in Deep Learning: A Survey of Vulnerabilities, Detection, and Defense
Architectural backdoors are a persistent class of neural-network backdoors that survive clean retraining, and current detection tools and benchmarks are not ready for them.
Reference graph
Works this paper leans on
-
[1]
A Closer Look at Q3 2024: 75% Surge in Cyber Attacks Worldwide,
CheckPoint, “A Closer Look at Q3 2024: 75% Surge in Cyber Attacks Worldwide,”
work page 2024
-
[2]
Ghost in the SAM: Stealthy, robust, and privileged persistence through invisible accounts,
S. R. Castro and A. A. C ´ardenas, “Ghost in the SAM: Stealthy, robust, and privileged persistence through invisible accounts,” in Proceedings of the 2024 Workshop on Research on Offensive and Defensive Techniques in the Context of Man At The End (MATE) Attacks, ser. CheckMATE ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 59–72. [...
arXiv 2024
-
[3]
Beyond cage: Investigating generalization of learned autonomous network defense policies,
M. Wolk, A. Applebaum, C. Dennler, P. Dwyer, M. Moskowitz, H. Nguyen, N. Nichols, N. Park, P. Rachwalski, F. Rau, and A. Webster, “Beyond cage: Investigating generalization of learned autonomous network defense policies,” 2022. [Online]. Available: https://arxiv.org/abs/2211.15557
arXiv 2022
-
[4]
CybORG: A Gym for the Development of Autonomous Cyber Agents,
M. Standen, M. Lucas, D. Bowman, T. J. Richer, J. Kim, and D. Marriott, “CybORG: A Gym for the Development of Autonomous Cyber Agents,” Aug. 2021
work page 2021
-
[5]
Autonomous Intelligent Cyber-defense Agent: Introduction and Overview
A. Kott, “Autonomous intelligent cyber-defense agent: Introduction and overview,” 2023. [Online]. Available: https://arxiv.org/abs/2304.12408
work page Pith review arXiv 2023
-
[6]
Alternative architectural approaches,
P. Theron, “Alternative architectural approaches,” in Autonomous In- telligent Cyber Defense Agent (AICA): A Comprehensive Guide , ser. Advances in Information Security, A. Kott, Ed. Springer International Publishing, 2023, vol. 87, pp. 17–46
work page 2023
-
[7]
Design and evaluation of an autonomous cyber defence agent using drl and an augmented llm,
J. Loevenich, E. Adler, T. Huerten, and R. Rigolin Ferreira Lopes, “Design and evaluation of an autonomous cyber defence agent using drl and an augmented llm,” 2024, available at SSRN: https: //ssrn.com/abstract=5076836 or http://dx.doi.org/10.2139/ssrn.5076836. [Online]. Available: https://ssrn.com/abstract=5076836
-
[8]
Deep reinforcement learning for autonomous cyber defence: A survey,
G. Palmer, C. Parry, D. J. B. Harrold, and C. Willis, “Deep reinforcement learning for autonomous cyber defence: A survey,” 2024. [Online]. Available: https://arxiv.org/abs/2310.07745
arXiv 2024
Show all 34 references
-
[9]
Autonomous cyber defence: a roadmap from lab to ops,
A. Lohn, A. Knack, A. Burke, and K. Jackson, “Autonomous cyber defence: a roadmap from lab to ops,” Center for Emerging Technology and Security, 2023
2023
-
[10]
Cyberbattlesim,
C. Seifert, M. Betser, W. Blum, J. Bono, K. Farris, E. Goren, J. Grana, K. Holsheimer, B. Marken, J. Neil, N. Nichols, J. Parikh, and H. Wei, “Cyberbattlesim,” https://github.com/microsoft/cyberbattlesim, 2021
2021
-
[11]
Nasimemu: Network attack simulator & emulator for training agents generalizing to novel scenarios,
J. Janisch, T. Pevn ´y, and V . Lis´y, “Nasimemu: Network attack simulator & emulator for training agents generalizing to novel scenarios,” in Computer Security. ESORICS 2023 International Workshops: CPS4CIP , ADIoT, SecAssure, WASP , TAURIN, PriST-AI, and SECAI, The Hague, Th...
2023 doi
-
[12]
Network environment design for autonomous cyberdefense,
A. Molina-Markham, C. Miniter, B. Powell, and A. Ridley, “Network environment design for autonomous cyberdefense,” https://arxiv.org/abs/ 2103.07583, 2021
2021 arXiv
-
[13]
Ttcp cage challenge 4,
TTCP Working Group, “Ttcp cage challenge 4,” https://github.com/ cage-challenge/cage-challenge-4, 2023
2023
-
[14]
Hierarchical Multi-agent Reinforcement Learning for Cyber Network Defense,
A. V . Singh, E. Rathbun, E. Graham, L. Oakley, S. Boboila, A. Oprea, and P. Chin, “Hierarchical Multi-agent Reinforcement Learning for Cyber Network Defense,” https://arxiv.org/abs/2410.17351, Oct. 2024
2024 arXiv
-
[16]
KEEP: A GNN-based PPO Model for MARL,
Cybermonic, “KEEP: A GNN-based PPO Model for MARL,” 2025, available: https://github.com/cybermonic/cage-4-submission. [Online]. Available: https://github.com/cybermonic/cage-4-submission
2025
-
[17]
Out of the cage: How stochastic parrots win in cyber security environments,
M. Rigaki, O. Luk ´aˇs, C. Catania, and S. Garcia, “Out of the cage: How stochastic parrots win in cyber security environments,” in Proceedings of the 16th International Conference on Agents and Artificial Intelligence . SCITEPRESS - Science and Technology Publications, 2024, ...
2024 doi
-
[18]
Depending on yourself when you should: Mentoring llm with rl agents to become the master in cybersecurity games,
Y . Yan, Y . Zhang, and K. Huang, “Depending on yourself when you should: Mentoring llm with rl agents to become the master in cybersecurity games,” 2024. [Online]. Available: https://arxiv.org/abs/2403.17674
2024 arXiv
-
[19]
OpenAI Models and Research,
OpenAI, “OpenAI Models and Research,” 2024, available: https://openai.com. [Online]. Available: https://openai.com
2024
-
[20]
Language Models are Few-Shot Learners,
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. C...
2020
-
[21]
Better Zero-Shot Reasoning with Role-Play Prompting,
A. Kong, S. Zhao, H. Chen, Q. Li, Y . Qin, R. Sun, X. Zhou, E. Wang, and X. Dong, “Better Zero-Shot Reasoning with Role-Play Prompting,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- n...
2024
-
[22]
DeepSeek: Advanced AI Language Models and Enterprise Solutions,
DeepSeek, “DeepSeek: Advanced AI Language Models and Enterprise Solutions,” 2025, available: https://deepseek.ai. [Online]. Available: https://deepseek.ai
2025
-
[23]
LLaMA: Open and Efficient Foundation Language Models,
Meta AI, “LLaMA: Open and Efficient Foundation Language Models,” 2023, available: https://ai.meta.com/llama. [Online]. Available: https://ai.meta.com/llama
2023
-
[24]
OpenRouter: A Unified Interface for Large Language Models,
OpenRouter, Inc., “OpenRouter: A Unified Interface for Large Language Models,” 2025, available: https://openrouter.ai. [Online]. Available: https://openrouter.ai
2025
-
[25]
Hugging Face: Democratizing Machine Learning,
Hugging Face, “Hugging Face: Democratizing Machine Learning,” 2024, available: https://huggingface.co. [Online]. Available: https://huggingface.co
2024
-
[26]
Weights & Biases: Machine Learning Experiment Tracking,
Weights & Biases, “Weights & Biases: Machine Learning Experiment Tracking,” 2025, available: https://wandb.ai/site/. [Online]. Available: https://wandb.ai/site/
2025
-
[27]
K-means clustering,
P. Berkhin, “K-means clustering,” in Encyclopedia of Machine Learning, C. Sammut and G. I. Webb, Eds. Springer, 2011, pp. 563–
2011
-
[28]
A density-based al- gorithm for discovering clusters in large spatial databases with noise,
M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, “A density-based al- gorithm for discovering clusters in large spatial databases with noise,” in Proceedings of the Second International Conference on Knowledge Discovery and Data Mining , ser. KDD’96. AAAI Press, 1996, p. 226–231
1996
-
[29]
Determining the appropiate cluster number using elbow method for k-means algorithm,
H. Humaira and R. Rasyidah, “Determining the appropiate cluster number using elbow method for k-means algorithm,” in Proceedings of the 2nd Workshop on Multidisciplinary and Applications (WMA) , 2020, pp. 1–8
2020
-
[30]
Silhouettes: A graphical aid to the interpretation and validation of cluster analysis,
P. J. Rousseeuw, “Silhouettes: A graphical aid to the interpretation and validation of cluster analysis,” Journal of Computational and Applied Mathematics , vol. 20, pp. 53–65, 1987. [Online]. Available: https://www.sciencedirect.com/science/article/pii/0377042787901257
1987
-
[31]
Kinetic models of sorption processes,
W. J. Weber and J. C. Morris, “Kinetic models of sorption processes,” Journal of the Sanitary Engineering Division , vol. 89, no. 2, pp. 31– 60, 1963. [Online]. Available: https://www.sciencedirect.com/science/ article/abs/pii/009830049390090R
1963
-
[32]
Large Language Models as Analogical Reasoners,
M. Yasunaga, X. Chen, Y . Li, P. Pasupat, J. Leskovec, P. Liang, E. H. Chi, and D. Zhou, “Large Language Models as Analogical Reasoners,” https://arxiv.org/abs/2310.01714, Mar. 2024
2024 arXiv
-
[564]
Available: https://link.springer.com/referenceworkentry/ 10.1007/978-0-387-30164-8 425
[Online]. Available: https://link.springer.com/referenceworkentry/ 10.1007/978-0-387-30164-8 425
-
[2017]
Available: http://arxiv.org/abs/1707.06347
[Online]. Available: http://arxiv.org/abs/1707.06347
-
[2024]
Available: https://blog.checkpoint.com/research/ a-closer-look-at-q3-2024-75-surge-in-cyber-attacks-worldwide/
[Online]. Available: https://blog.checkpoint.com/research/ a-closer-look-at-q3-2024-75-surge-in-cyber-attacks-worldwide/
2024
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.