{"id":"637ddb52-0b72-4a13-8ed1-7d47a3e8ba39","arxiv_id":"2505.04843","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LLM agents can be integrated as blue-team defenders in the CybORG CAGE 4 multi-agent environment, but they are about 100x slower and earn lower rewards than a GNN-based RL team.","lead":"This paper connects large language models to a multi-agent cyber defense simulator and lets them team up with reinforcement learning defenders. It finds the LLM defenders are far slower and score worse than RL agents, but they can reason about security in ways RL cannot easily explain.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The protocol's contribution is never ablated: all reported configurations use the 8-bit vector, so the central 'can communicate effectively' conclusion lacks a direct test; a no-protocol baseline and parse-accuracy check are needed.","rationale":"The reader's weakest_assumption identified the protocol ablation; I agree. The reason this is the single most load-bearing issue is that it directly targets Contribution 2 and the conclusion's causal claim: the word 'effectively' cannot be evaluated without a control. The paper's quantitative results are also underpowered (two episodes, no significance tests), but those flaws weaken comparisons to RL; they do not, by themselves, invalidate the qualitative claim that LLMs can act and reason as defenders. The protocol issue is more central because the headline claim names communication as the mechanism of team performance. I give credit for the open-source framework, the honest limitations section, and the qualitative cluster analysis, which shows structured defensive reasoning; these are real contributions. However, none of them independently verifies the communication protocol. A single ablation plus parse-accuracy measurement would settle the issue. If the check shows the protocol matters, the current conditional verdict can move to accept; if not, the communication claim should be dropped or reframed. Thus UNCHANGED relative to the reader's conditional verdict is appropriate.","tokens_in":11846,"tokens_out":5182,"duration_ms":53601,"concrete_test":"Run the all-LLM and 1LLM+4RL scenarios for at least 10 episodes (5 seeds x 2 episodes) under two conditions: the current protocol and a control with the Communication Vectors field removed or zeroed in both prompt and RL observations. Compare cumulative reward, per-action distributions, and the conditional probability of Remove/BlockTrafficZone after a remote-alert bit. Also score the LLM's parsed interpretation of each incoming vector (origin, compromise level, availability) against ground truth. If reward/action patterns are unchanged, the protocol is not contributing; if parse accuracy is near chance, 'communicate effectively' is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central conclusion ('LLM ACD agents can communicate effectively when they share a communication protocol', Section VI) depends on the 8-bit protocol of Section III-C being both used and useful. No condition without the protocol is run: the all-LLM, all-RL, and 1LLM+4RL scenarios all include the communication vectors, and the LLM prompt instructs the agent to read them. Consequently, the reported coordination behavior (e.g., the LLM choosing Remove or BlockTrafficZone 'when other agents notified a possible compromise') could be caused by the protocol, by local observations, or by the prompt's action descriptions. The paper even documents systematic misreading of the vector—agent identity and compromise-level errors—so the reliability of the channel is itself uncertain. Because the protocol is Contribution 2 and the conclusion asserts effective communication, the missing ablation is load-bearing: without it the contribution is asserted rather than measured, and the 'security reasoning similar to a team of security operators' claim has no direct support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims to be the first study of LLMs in multi-agent autonomous cyber defense. It adapts CybORG CAGE 4 so that one or more blue-team agents are driven by an LLM through a natural-language observation formatter and a JSON action parser, and it introduces an 8-bit communication vector by which defenders share compromise levels, remote detections, and availability. Experiments compare no-defender, all-LLM, all-RL (Cybermonic KEEP), and mixed 1LLM+4RL teams against several finite-state red agents, with two 500-step episodes per condition, and use clustering of LLM reasoning logs to characterize defensive strategies. The paper concludes that LLM agents can communicate effectively with the protocol and exhibit security reasoning similar to human operators.","tokens_in":12020,"tokens_out":6272,"duration_ms":58406,"significance":"If the findings survive the added controls, the paper contributes a useful open-source integration of LLM agents into CybORG CAGE 4, a multi-agent ACD testbed, and provides a concrete 8-bit coordination protocol plus qualitative reasoning traces. Its strengths include a clearly described adapter architecture, use of an established external RL baseline (Cybermonic KEEP), and a transparent discussion of hallucinations and prompt influence. The qualitative action-clustering analysis is a reasonable first attempt to inspect LLM security reasoning. However, the protocol's contribution is not measured, and the quantitative evaluation is too under-powered to support the central claims; with those addressed, the paper would be a valuable resource for the ACD community.","major_comments":[{"comment":"The central quantitative comparison between the all-LLM and all-RL teams (Fig. 5; 'For all our experiments, we run 2 episodes') rests on two episodes per configuration and no significance test. Since temperature is set to 1, episode-level variance is expected to be large, so the reported means and standard deviations are not sufficient to support the conclusion that RL agents perform better against diverse red adversaries. Report repeated runs with more seeds and include confidence intervals or a statistical test.","section":"IV, Scenarios; Fig. 5"},{"comment":"The 8-bit communication protocol is never ablated: every reported team configuration uses it, and the LLM prompt instructs agents to read the Communication Vectors. The conclusion that 'LLM ACD agents can communicate effectively when they share a communication protocol' is therefore not directly tested, because the coordinated behavior could be driven by local observations, the prompt's action descriptions, or the protocol. Add a no-protocol control condition and a quantitative measure of protocol usage/parse accuracy, especially since Section IV documents the LLM misreading the vector (confusing agent 3 and agent 4, and misreading compromise levels).","section":"III-C; VI"},{"comment":"The all-LLM versus all-RL comparison uses GPT-4o-mini, which Fig. 4 identifies as the lowest-reward OpenAI model in the 1LLM+4RL configuration; o3-mini achieves the highest reward. The headline claim comparing 'LLM agents' with RL agents is thus not representative of the best LLM configuration tested. Either include the best-performing LLM in the comparison or explicitly restrict the claim to GPT-4o-mini.","section":"IV, Figs. 4-5"},{"comment":"The final prompt is selected by comparing average rewards on the default CAGE 4 red agent, and the same reward signal is then the main evaluation metric in Section IV. This creates a selection effect that should be acknowledged; reporting results for the alternative prompt variants, or at least stating that the choice was made on a small tuning set, would let readers judge the sensitivity.","section":"III-D, Table III"}],"minor_comments":[{"comment":"The sentence 'are not all publicly accessible, actively maintained, or are limited to to particular adversarial scenarios' contains a duplicated 'to'.","section":"I"},{"comment":"The sentence 'we protect 3 subnets with 1 LLM agent (i.e., blue agent 4) and 4 subnets with 4 RL (KEEP) agents' is confusing given that the architecture has five blue agents; clarify how the five agents map to subnets in this scenario.","section":"IV, Scenarios"},{"comment":"The table header lists 'Instruct' twice; label the three prompt conditions (Instructional, Few-Shot, Role) explicitly.","section":"III-D, Table III"},{"comment":"The phrase 'and the the latest DeepSeek model' has a duplicated 'the'.","section":"IV, Environment and models"},{"comment":"The phrase 'GPT-4o-mini had the quickest execution time and but low reward' should read 'and the lowest reward'.","section":"IV, Performance"},{"comment":"The reference [31] to Weber and Morris's sorption kinetics paper is not a standard PCA citation; please cite a canonical PCA reference instead.","section":"IV, Reasoning for Action Selection"},{"comment":"The statement that the team can 'protect a network with a security reasoning similar to a team of security operators' is not operationalized; consider removing it or specifying which comparison supports it.","section":"VI"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central claims are broader than the evidence provided; please also verify the novelty claim against workshop/arXiv literature, since hybrid LLM+RL agents for CAGE have appeared in single-agent settings and the 'first multi-agent' assertion would benefit from a more thorough related-work search."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick read of arXiv:2505.04843. The paper does something genuinely new: it integrates LLM-based blue-team agents into the multi-agent CybORG CAGE 4 environment, alongside RL agents, and it ships an open-source adapter. That is the first multi-agent LLM-for-ACD integration I know of, and the reasoning-cluster analysis is a thoughtful way to inspect what LLM defenders are doing. They also report honestly that the RL baseline (KEEP) beats the LLM team on reward, and they discuss speed and hallucination issues rather than overselling LLMs.\n\nThe soft spot is the one the stress-test flags: the 8-bit communication protocol is never ablated. Every configuration, including the all-RL baseline, runs with the communication vectors in place. So the headline conclusion that 'LLM ACD agents can communicate effectively when they share a communication protocol' is asserted, not measured. To support it you'd need a no-communication baseline, or at least a direct measure of how often an agent's action actually responds to a correctly parsed vector. The paper itself documents the LLM misreading vector fields, so the channel's reliability is in question.\n\nTwo episodes per scenario and no significance tests also leave the quantitative claims shaky. The prompt tuning in Table III is a form of fitting to the reward, even though the central comparison against KEEP is independent. And the headline all-LLM-vs-all-RL comparison uses only GPT-4o-mini, the weakest OpenAI model they tested; with better models the gap might shrink, but that's still not tested. These are addressable weaknesses rather than fatal ones. The integration is real, the code is promised, and the discussion of limitations is unusually candid.\n\nAnyone working on LLM/RL hybrid agents for network defense will get value from this, especially the adapter design and the catalog of failure modes. I'd send it to peer review but with clear expectations: add a no-protocol ablation, more episodes, and a parse-accuracy check for the communication vector. It's workshop-level now, but with that extra work it could become a solid citable paper. If they fix the ablation, I'd cite it.","headline":"First real integration of LLM defenders into multi-agent CybORG, but the communication-protocol claim is unmeasured for want of an ablation.","tokens_in":12583,"tokens_out":2831,"would_cite":true,"duration_ms":28113,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that pre-trained large language models, without reinforcement learning or security-specific fine-tuning, can act as blue-team defenders in a multi-agent cyber defense environment and coordinate with…","keywords":["autonomous cyber defense","large language models","multi-agent systems","reinforcement learning","incident response","communication protocol","cyber defense agents"],"falsifier":"Run the same all-LLM, all-RL, and hybrid teams with the communication vector disabled (for example, always broadcasting zeros) and compare rewards and action patterns; if removing the protocol leaves team performance and coordination unchanged, the claim that LLM and RL agents communicate effectively through it is unsupported.","tokens_in":11658,"feed_emoji":"🛡️","tokens_out":6037,"duration_ms":56536,"temperature":0.7,"pith_summary":"This paper tries to establish that general-purpose large language models, with no reinforcement learning or security-specific fine-tuning, can serve as defensive agents in a multi-agent cyber incident-response setting. It presents the first integration of LLM agents into the CybORG CAGE 4 challenge, where teams of defenders protect an enterprise network against scripted attackers, and proposes an 8-bit communication protocol that lets LLM and RL defenders share alerts, compromise levels, and availability. The authors report that an all-LLM team reasons in structured, human-readable ways—deploying decoys, analyzing suspicious connections, and responding to teammates' alerts—but earns markedly worse availability rewards than an all-RL team and acts about 100 times slower. The central claim is that coordination through the protocol, not reward maximization, makes LLM agents viable team members whose reasoning is transparent and whose strategy can be changed by editing the prompt.","feed_headline":"Untrained LLMs coordinate with RL agents to defend a simulated network","feed_subtitle":"A new 8-bit protocol lets language models share alerts and compromise levels with reinforcement-learning teammates, no retraining.","key_machinery":"The load-bearing mechanism is the LLM adapter pipeline that converts CybORG's numeric observation vectors into natural-language observations, feeds them with a role-based prompt to an LLM, and parses the model's JSON response into a defender action. On top of that sits the proposed 8-bit communication protocol: bits 0-4 flag malicious activity detected from each of the other agents' networks, bits 5-6 encode the compromise level in the sender's own subnet (no compromise, scan or remote exploit, user-level compromise, admin-level compromise), and bit 7 marks whether the sender is busy. This vector is what the paper claims lets heterogeneous defenders coordinate, and it is the object whose contribution to team performance is asserted rather than isolated.","core_discovery":"On the paper's own terms, the discovery is that a pre-trained LLM can be placed in a multi-agent autonomous cyber defense environment and behave as a coherent blue-team operator: it parses natural-language observations, chooses among defender actions, and responds to messages from other defenders with security-relevant actions such as removing malicious processes or blocking traffic. The authors argue that when all five blue agents are LLM-driven, the team exhibits a recognizable defensive posture—preventive decoy deployment, analysis of repeated low-severity connections, and quarantine responses to remote alerts—even though it accumulates more availability penalties than the reinforcement-learning baseline. They further report that a hybrid team with one LLM and four RL agents is feasible and that the LLM's decisions can be clustered into four recurring reasoning patterns. The paper concludes by asserting that LLM ACD agents can communicate effectively when they share a communication protocol and that together they can protect a network with a security reasoning similar to a team of security operators.","pith_inferences":["If the protocol's coordination value were confirmed by ablation, a natural next step would be to let the communication vector be learned or negotiated by the agents rather than fixed by hand, since the fixed encoding is currently a design assumption.","The paper's observed misreads of the vector (an agent attributing a message to the wrong sender) suggest that future work should test whether simpler encodings—or natural-language messages instead of bitfields—reduce coordination errors.","The higher reward achieved by the slower reasoning model hints that the LLM's security reasoning quality, not raw speed, is the lever that matters; a faster reasoning model or distilled expert could close the runtime gap."],"forward_implications":["LLM defenders can be added to an existing RL-based autonomous cyber defense system without retraining the LLM; only the prompt and observation formatting need to change.","A mixed team of one LLM and four RL agents is operational in the CAGE 4 scenario, demonstrating that a shared communication protocol can bridge heterogeneous agent types.","Because LLM reasoning traces are text, defender strategy can be inspected and modified through prompt design, which the paper suggests could eventually reproduce or replace trained RL policies.","The paper's reward comparisons quantify the current gap: the all-RL team averages around -493 reward, the all-LLM team around -2547, and the LLM's action selection is about 104 times slower, so real-time deployment is not yet practical."],"supporting_citations":[{"why":"Defines the CybORG simulation environment that all experiments run in and whose CAGE 4 scenario supplies the network, agents, and actions.","marker":"[4]"},{"why":"Provides the CAGE 4 challenge repository with the scenario specification, reward function, and default red and green agent policies used in evaluation.","marker":"[13]"},{"why":"Supplies the GNN-based PPO multi-agent reinforcement learning defender used as the RL baseline and as the RL teammates in hybrid teams.","marker":"[16]"},{"why":"Contributes the red-agent strategy implementations (aggressive, stealthy, impact, and degrade-service variants) used to diversify adversary evaluation.","marker":"[14]"},{"why":"Represents prior single-agent LLM autonomous cyber defense work that this paper extends to the multi-agent setting.","marker":"[17]"},{"why":"Represents prior LLM-and-RL hybrid work in a single-agent CAGE 1 setting, the baseline this paper moves beyond.","marker":"[18]"}],"fun_headline_variants":["Untrained LLMs and RL agents defend networks together","LLM cyber defenders work with RL teammates without retraining","Multi-agent cyber defense: LLMs join RL with no extra training","First study: LLM and RL agents collaborate in cyber defense"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central coordination claim rests on the assumption that its 8-bit communication vector actually carries useful information between defenders, yet every reported team uses the protocol, so its contribution to the observed behavior is never measured.","fun_headline_variants_meta":{"raw":{"variants":["Untrained LLMs and RL agents defend networks together","LLM cyber defenders work with RL teammates without retraining","Multi-agent cyber defense: LLMs join RL with no extra training","First study: LLM and RL agents collaborate in cyber defense"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00032,"raw_usage":{"total_tokens":1792,"prompt_tokens":924,"completion_tokens":868,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":799}},"tokens_in":540,"tokens_out":868,"duration_ms":8941,"temperature":1.0,"reasoning_tokens":799,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:19:19.805859+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same all-LLM, all-RL, and hybrid teams with the communication vector disabled (for example, always broadcasting zeros) and compare rewards and action patterns; if removing the protocol leaves team performance and coordination unchanged, the claim that LLM and RL agents communicate effectively through it is unsupported.","supporting_citations":[{"cited_title":"CybORG: A Gym for the Development of Autonomous Cyber Agents,","cited_arxiv_id":null,"evidence_quote":"Defines the CybORG simulation environment that all experiments run in and whose CAGE 4 scenario supplies the network, agents, and actions."},{"cited_title":"Ttcp cage challenge 4,","cited_arxiv_id":null,"evidence_quote":"Provides the CAGE 4 challenge repository with the scenario specification, reward function, and default red and green agent policies used in evaluation."},{"cited_title":"KEEP: A GNN-based PPO Model for MARL,","cited_arxiv_id":null,"evidence_quote":"Supplies the GNN-based PPO multi-agent reinforcement learning defender used as the RL baseline and as the RL teammates in hybrid teams."}],"review_version":1}