Pith. sign in

REVIEW 4 major objections 5 minor 47 references

An Empirical Game-Theoretic Analysis of Autonomous Cyber-Defence Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a multi-oracle extension of the double-oracle algorithm, accelerated by value-function potential-based reward shaping, yields cyber-defence agents that hold their ground as adaptive attackers keep learning.

desk verdict A worthwhile framework paper for adversarial evaluation of ACD agents, but the robustness claims run ahead of the evidence—single MRO runs and 100-episode payoffs are doing too much work. read the letter →

arxiv 2501.19206 v1 pith:27VNV4VT submitted 2025-01-31 cs.AI cs.CRcs.GT

Gregory Palmer , Luke Swaby , Daniel J.B. Harrold , Matthew Stewart , Alex Hiles , Chris Willis , Ian Miles , Sara Farmer This is my paper · ORCID
classification cs.AIcs.CRcs.GT
keywords autonomouscyberdefenceempiricalgametheorydoubleoraclealgorithmmultipleresponseoraclespotential-basedrewardshapingdeepreinforcementlearningNashequilibriumCAGEChallenge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that adversarial learning via an empirical game-theoretic loop is the right organising principle for training and assuring autonomous cyber-defence agents. To make that practical, it introduces a multiple-response-oracles (MRO) extension of the double-oracle algorithm, where each side can draw approximate best responses from several independent learning approaches rather than one, and proves the convergence guarantees are unchanged. It also proposes value-function potential-based reward shaping (VF-PBRS), which reuses value functions from earlier response iterations as a shaping potential, making each response cheaper to compute and, for the defender, significantly stronger than vanilla training. In two standard cyber-defence benchmark environments (CAGE Challenges 2 and 4), defenders produced by the loop keep their ground as learning attackers try new tactics, and mixture play substantially reduces worst-case exposure compared to deploying any single open-source defender.

What carries the argument

The load-bearing object is the empirical matrix game, a finite two-player zero-sum normal-form game in which each action is a policy trained for one side, and each payoff entry is the mean episodic reward of one policy pair estimated from 100 evaluation episodes. The MRO algorithm replaces the single approximate-best-response oracle of the standard double-oracle loop with a response function that returns a set of candidate policies from several learning approaches, plus a selection step that keeps the best one; because the selected policy is still an approximate best response, the proof that exploitability $\le 0$ implies a resource-bounded Nash equilibrium goes through unchanged. VF-PBRS shapes rewards with $F(s,s') = \gamma\Phi(s') - \Phi(s)$, where the potential $\Phi$ is the mixture-weighted, Z-score-normalised sum of value functions from earlier iterations. Since this is a potential-based shaping function, the optimal policy is invariant, so oracles using VF-PBRS still return approximate best responses. Pre-trained model sampling with $\epsilon$-greedy decay balances reuse of past policies against exploration of new strategies.

What would settle it

Run the MRO loop repeatedly from different random seeds and fresh initialisations in CAGE Challenge 2, and then train a fresh attacker with a larger budget against the final Blue mixture; if the final game value varies across seeds by more than the reported standard errors, or if that fresh attacker improves on the Nash payoff by more than evaluation noise, the paper's robustness claim would be contradicted.

Watch

Extended reading notes

Core claim

The central claim is that the approximate double-oracle procedure, extended to multiple response oracles and accelerated by VF-PBRS and pre-trained model sampling, converges to a resource-bounded Nash equilibrium of the empirical game. The argument is that potential-based reward shaping preserves the optimal policy, so a VF-PBRS oracle still returns an approximate best response, and that the best policy within a response set computed by MRO is itself an approximate best response. In the experiments, the value of the CC2 game stabilises around $-29.63$ from Blue's perspective after roughly 23 iterations, and a Red response trained from scratch initially jumps to $93.05$ but is neutralised to $29.75$ once Blue's mixture adjusts. The CC4 game value changes little after the first iteration, settling near $-411.91$, with the estimated value of the game essentially constant. The paper concludes that defenders obtained this way are robust to learning attackers, which struggle to find new successful tactics, techniques and procedures.

Load-bearing premise

The single load-bearing premise is that the empirical payoff matrix, estimated from 100 evaluation episodes per policy pair, is accurate enough for the Nash solver to identify a meaningful equilibrium, and that the one MRO run per environment is representative of the stochastic training process.

Editorial extensions

If this is right

  • A defender mixture produced by the CC2 MRO run held the Nash payoff to about $-29.63$; when a freshly initialised Red oracle found a response that briefly raised the payoff to $93.05$, the Blue mixture adjusted and pushed it back to $29.75$, and no later attacker response significantly moved the game value.
  • Under pre-trained initialisation in CC2, VF-PBRS with value-function ensembling significantly outperformed vanilla training for Blue responses (mean $-29.8 \pm 1.2$ versus $-38.3 \pm 2.7$), while for Red the shaping benefit was small and not statistically significant.
  • Because the selected response in MRO is still an approximate best response, the standard double-oracle termination condition applies: if the combined exploitability is at most zero, the algorithm has found a resource-bounded Nash equilibrium.
  • The final CC2 Blue mixture consisted of graph-based PPO policies, and a single Red policy that scored $-117.82$ against the original defender parameterisation scored only $-21.96$ against the mixture, illustrating that mixture play reduces worst-case exposure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the MRO loop is a general evaluation harness for any domain where several candidate training recipes compete against an adaptive adversary; red-teaming of language models or fraud-detection systems could reuse the same structure, provided per-pair payoffs can be estimated reliably.
  • Inference: VF-PBRS helping the defender much more than the attacker suggests the shaping signal matters most when the reward is sparse and delayed, as in long attack chains; a targeted comparison of short versus long attack sequences would test this directly.
  • Inference: the reported game values come from a single MRO run per environment, so the variance of the equilibrium itself is not measured; re-running the loop across random seeds would show whether the robustness result is a property of the method or of one training trajectory.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a multiple response oracles (MRO) extension of the approximate double oracle (ADO) algorithm for evaluating and combining deep reinforcement learning agents for autonomous cyber defence. It introduces value-function potential-based reward shaping (VF-PBRS) with mixture-weighted ensembles, together with pre-trained model sampling, and provides an empirical game-theoretic analysis in CybORG CAGE Challenges 2 and 4. The authors claim that MRO preserves ADO's convergence guarantees, that adversarial training produces Blue mixtures robust to learned Red agents, and that VF-PBRS yields significantly stronger policies than vanilla training in at least one setting.

Significance. If the MRO convergence result and the empirical findings hold, the paper offers a useful framework for comparing and ensembling heterogeneous ACD-DRL approaches, an area with little prior work. The paper's strengths include a natural and practically motivated extension of DO, a mostly correct convergence argument for MRO in Appendix A.2.1, a careful VF-PBRS design rooted in Ng et al.'s policy invariance theorem, and detailed appendices with hyperparameters, network diagrams, and action-level analyses. The VF-PBRS comparison for Blue GPPO in CC2 is supported by ten runs per setting with significance tests. However, the central empirical claims rest on a single MRO run per environment and on payoff matrices estimated from 100 evaluation episodes, and the claimed theoretical guarantee that MRO strictly improves on ADO is not proven. These issues currently limit the certainty with which the conclusions can be accepted.

major comments (4)
  1. [Section 4.2 and Appendix A.2.2 (Theorem 5, Definition A.3, Eqs. 24-26)] The proof that MRO is guaranteed to produce stronger mixture agents than ADO is circular. Definition A.3 defines 'minimal viable support' but does not establish that the policy πi,n_p1 from Assumption A.1 belongs to such a support; Eq. (24) is essentially the assertion to be proved. Moreover, Eq. (26), 'µ′_p1 ⊂ µp1', need not hold: ADO and MRO are different algorithms that may generate different policy sets over the same iterations, so an ADO mixture can include policies not present in the MRO mixture. The claim in Section 4.2 that MRO will produce stronger mixtures is therefore unsupported as stated and should be removed or replaced with a correct theorem.
  2. [Sections 7.1, 7.2 and Appendix C] The central empirical claims—the final game values GBlue(µBlue, µRed) = −29.63 in CC2 and −411.91 in CC4, the statement that Red 'struggles to find new successful TTPs', and the convergence observations around iteration 20—are based on a single MRO run per environment, with payoff entries estimated from 100 evaluation episodes and no repeated-run or bootstrap intervals propagated into the Nash solver or the GE termination check. Table 1 reports standard errors of 0.4–3.7 for several responses; these are not reflected in the reported game values or in the Nash payoff line of Figure 1. Because the gaps supporting 'no significant improvement' are often small (e.g., Red VF-PBRS vs vanilla means 24.72/24.85 vs 24.22), the equilibrium mixture and the GE ≤ 0 conclusion could be artifacts of estimation noise. Please add repeated-seed MRO runs or, at minimum, bootstrap confidence intervals for the payoff matrix and game values, and qualify the convergence and robustness claims accordingly.
  3. [Abstract and Section 7.1 (Table 1, Figure 2)] The abstract's claim that VF-PBRS oracles 'converge upon significantly stronger ACD policies compared to vanilla approaches' is overgeneralized. The significant improvement is demonstrated only for Blue GPPO responses in CC2; for Red in CC2 the differences (24.72/24.85 vs 24.22) are explicitly not significant, and in CC4 the paper reports no significant VF-PBRS advantage. Please restate the contribution as an empirical finding for the specific oracle and environment in which it is supported.
  4. [Section 4.2, Eq. (9), and Algorithm 2] The existence of an ABR policy in the response set Ri(µj) is assumed, not guaranteed. If none of the response oracles yields a policy that improves on the current mixture, GE ≤ 0 certifies only a resource-bounded equilibrium relative to the specific response functions, not that no approximate best response exists in the broader policy space. The paper should state this qualification explicitly when claiming the extension preserves ADO's theoretical properties, and Section 7.1's conclusion that Red 'struggles to find new successful TTPs' should be framed as a property of the tested response oracles.
minor comments (5)
  1. [Appendix C] The paper switches between γ_φ and γ when describing the potential discount factor; please use one symbol consistently, preferably γ_φ for the potential discount in Eq. (5).
  2. [Figures 1 and 14] The Nash payoff lines would be substantially more informative with pointwise confidence bands, especially where the text refers to 'significant' improvements or their absence.
  3. [Section 8] The discussion of strictly and weakly dominated policies treats the estimated payoff matrix as exact; a sentence acknowledging that dominance relations inherit sampling noise from the 100-episode evaluations would make the pruning discussion more cautious.
  4. [References and Figure 17] There are minor typographical errors, including 'V on Neumann' in the bibliography and 'Fale' in Figure 17; these should be corrected.
  5. [Algorithm 2, line 6] The notation π∗_Blue is used before being defined in the algorithm; please state explicitly that it denotes the best policy within R_Blue(µ_Red) according to Eq. (9).

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: VF-PBRS is grounded in the external Ng et al. invariance theorem and the empirical game analysis is self-contained; only minor non-load-bearing self-citations are present.

full rationale

The paper's central derivation chain is not circular. VF-PBRS is justified by Ng et al. (1999)'s theorem that any potential-based shaping reward preserves optimal policies; the choice of value functions from prior ABR iterations as potentials is legitimate algorithmic reuse, since PBRS invariance holds for arbitrary potentials and the empirical comparison against vanilla training is an independent test using unshaped mean episodic rewards, not a fitted prediction. The MRO extension is argued by the standard ADO/PSRO proof pattern: if the best response returned by each response set fails to improve on the current Nash mixture payoff, then the mixtures form a resource-bounded Nash equilibrium; this is a definitional termination condition, not a manufactured prediction, and the paper does not present the GE <= 0 check as an empirical discovery. The robustness conclusion ('Red agents struggle to find new successful TTPs') is an empirical search claim whose strength depends on the quality of the red oracles and the 100-episode payoff estimates, but that is a statistical under-determination concern, not circularity. The paper's self-citations (Palmer et al. 2023 survey; Miles et al. 2024 overview) are contextual and do not carry any load-bearing theorem or uniqueness argument; the convergence and invariance results are attributed to external sources (Ng et al., Oliehoek et al., McMahan et al.). No step reduces by construction to its own input, and no fitted parameter is renamed as a prediction. Score 2 reflects only the presence of minor, non-load-bearing self-citations.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central results rest on a handful of hand-chosen hyperparameters (tau, gamma_phi, step counts) and domain assumptions about CybORG fidelity and evaluation sufficiency. No entirely new physical or conceptual entities are introduced; MRO and VF-PBRS are algorithmic constructions.

free parameters (6)
  • Shaping scaling coefficient tau = 1 or 0.5
    Controls the magnitude of the VF-PBRS shaping reward (Equation 5). Chosen by hand and tested at two values.
  • Potential discount factor gamma_phi = 1
    Discount factor in the potential-based shaping term; set to 1 in experiments because gamma=0.99 often produced negative shaping rewards (Appendix C).
  • PTM sampling epsilon decay rate = 0.95
    Discount applied to epsilon after each ABR iteration in the epsilon-greedy pre-trained model sampling scheme (Section 4.3).
  • Response training steps = 1.5M PTM and 5M full
    Environment steps per ABR iteration; chosen to balance wall-time and convergence (Sections 7.1 and 7.2).
  • Evaluation episodes = 100
    Number of episodes used to estimate each payoff matrix cell; a fixed choice with no reported variance across multiple game runs.
  • Number of response oracles per side = 2 in CC2 and CC4 (VF-PBRS and vanilla)
    MRO uses two parallel response approaches per player; the set could be larger, and this choice affects game matrix size.
assumptions (6)
  • domain assumption CybORG CC2 and CC4 can be modelled as finite two-player zero-sum POMGs, and Red rewards are the negation of Blue rewards.
    Section 6 sets Red rewards to the negation of Blue rewards, making the game zero-sum; this is an assumption about the environment.
  • domain assumption The opponent mixture remains stationary while an agent computes a best response.
    Theorem 2 and the MRO proof rely on this standard DO idealization.
  • domain assumption Payoffs estimated from 100 evaluation episodes are sufficiently accurate for Nash solving of the empirical game.
    Section 7 uses 100 episodes per policy pair with no confidence intervals on the game value.
  • domain assumption DRL oracles (PPO, GPPO, AM-PPO) return approximate best responses of sufficient quality.
    ADO convergence to an epsilon-RBNE assumes oracles compute approximate best responses; the paper uses standard DRL without quality guarantees.
  • standard math Ng et al. potential-based reward shaping sufficiency extends to POMGs with function approximation.
    Used in Theorem 2 and Appendix A.3, depending on the reduction of the game to an MDP with a fixed opponent mixture.
  • domain assumption CybORG is a faithful enough simulator for conclusions about cyber-defence robustness to transfer.
    The paper's claims about robustness are made with respect to the simulator, not a real network.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Empirical Game-Theoretic Analysis of Autonomous Cyber-Defence Agents." pith.science (2026). https://pith.science/paper/27VNV4VT

@misc{pith2026250119206,
  author       = {Pith},
  title        = {Pith review of: An Empirical Game-Theoretic Analysis of Autonomous Cyber-Defence Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/27VNV4VT}},
  note         = {Machine review of arXiv:2501.19206}
}
read the original abstract

The recent rise in increasingly sophisticated cyber-attacks raises the need for robust and resilient autonomous cyber-defence (ACD) agents. Given the variety of cyber-attack tactics, techniques and procedures (TTPs) employed, learning approaches that can return generalisable policies are desirable. Meanwhile, the assurance of ACD agents remains an open challenge. We address both challenges via an empirical game-theoretic analysis of deep reinforcement learning (DRL) approaches for ACD using the principled double oracle (DO) algorithm. This algorithm relies on adversaries iteratively learning (approximate) best responses against each others' policies; a computationally expensive endeavour for autonomous cyber operations agents. In this work we introduce and evaluate a theoretically-sound, potential-based reward shaping approach to expedite this process. In addition, given the increasing number of open-source ACD-DRL approaches, we extend the DO formulation to allow for multiple response oracles (MRO), providing a framework for a holistic evaluation of ACD approaches.

Figures

Figures reproduced from arXiv: 2501.19206 by the authors.

Figure 1
Figure 1. A depiction of Blue and Red ABRs from an MRO run on CC2. Rewards are plotted from Blue’s perspective. Both agents are unable to find ABRs that significantly improve on the Nash payoff (the black line) between iterations 11 – 20. Red oracles use PTMs for the first 20 iterations, but subsequently switch to random initialisations (PTM=False). Initially this oracle setting finds a policy that significantly improves on t… view at source ↗
Figure 2
Figure 2. The box plot above compares VF-PBRS runs using VF ensembling (VFE) and individual VFs k (where µ k i > 0) against vanilla training runs (10 runs per setting). Statistical p-values of less than 0.05 and 0.01 are flagged with one and two asterisks respectively. Outliers are plotted as separate black circles. Final Mixture Composition [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 4
Figure 4. Depicted are the percentage of steps per episode where Red can obtain User and Privileged access on the listed nodes. We compare privileges obtained by Red against the original GPPO parameterisation, π o Blue, and the final mixture agent µBlue. The attacking policy is Red’s ABR against π o Blue: π 2 Red. 7.2. CybORG CAGE Challenge 4 For CC4 we launch an MRO run consisting of 2.5M step ABR iterations using PTMs. Blue… view at source ↗
Figures from the paper (12 more)
Figure 3
Figure 3. Figure 3: An illustration of CC2’s empirical game and Nash mix￾tures. Cells represent the mean episodic reward for each policy pairing plotted from Red’s perspective (100 evaluation episodes). X and Y ticks indicate the ABR iteration in which a response was learnt. We also inclu…
Figure 5
Figure 5. Figure 5: Action percentage comparison for CC4 agents tasked with defending restricted and operational zones A under the joint-policy profiles ⟨π o Blue, π o Red⟩ and ⟨π 1 Blue, π o Red⟩. Here, π o Blue are the original parameterisations from King (2024); π o Red Red’s ABR again…
Figure 6
Figure 6. Figure 6: CAGE Challenge 2 Network Layout. B.2. CAGE Challenge 4 [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: CAGE Challenge 4 Network Layout. C. Experiment Settings Below we provide an overview of our experiment settings. We begin by outlining the configurations that are used across our MRO runs. We subsequently provide details regarding CAGE Challenge 2 and 4 specific config…
Figure 8
Figure 8. Figure 8: In the plots above we illustrate the computational com￾plexities of MRO compared to the ADO algorithm with respect to the size of the increase in the size of the empirical payoff matrix (Figure 8a) and the number of new cells that need to be computed in each iteration …
Figure 9
Figure 9. Figure 9: We depict the impact of the number of methods used from computing responses within each response iteration on the size of the payoff matrix. F. Preliminary CAGE Challenge 2 Run During one of our early preliminary runs within CAGE Challenge 2 we evaluated the impact of …
Figure 10
Figure 10. Figure 10: A preliminary CAGE Challenge 2 run, comparing dif￾ferent types of oracles. While four oracles (VF-PBRS; PTMs; VF-PBRS & PTMs, and; Vanilla) are used in each iteration, we only depict the best performing response. We find the majority of ABRs were obtained by oracles u…
Figure 11
Figure 11. Figure 11: An example of ABRs from preliminary trial runs, which made noteworthy improvements after several million training steps. H. Policy Characteristics – CAGE Challenge 2 In this section we expand upon the comparison between the starting point of our training process, π o …
Figure 13
Figure 13. Figure 13: Policy characteristics, actions and targets from time￾steps 1 - 100 for the final Blue mixture agent µBlue against π 2 Red. 17 Unpublished Work. Copyright 2025 BAE Systems. All Rights Reserved [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 15
Figure 15. Figure 15: An illustration of CC4’s empirical game and Nash mix￾tures. Cells represent the mean episodic reward for each policy pairing (100 evaluation episodes). Darker cells represent match￾ups that are favorable for Red. X and Y ticks indicate the oracle used–VF-PBRS (PBRS) a…
Figure 16
Figure 16. Figure 16: This figure provides a pure strategy analysis for individual Blue (Figure 16a) and Red (Figure 16b) policies against the final Red and Blue mixture agents within CAGE Challenge 2. Within each sub-figure, the upper sub-plot illustrates the sampling probabilities for th…
Figure 17
Figure 17. Figure 17: As in [PITH_FULL_IMAGE:figures/full_fig_p021_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 39 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Reward Shaping for Happier Autonomous Cyber Security Agents

    Bates, E., Mavroudis, V., and Hicks, C. Reward Shaping for Happier Autonomous Cyber Security Agents . In Proc. of the 16th ACM Workshop on Artificial Intelligence and Security, AISec '23, pp.\ 221–232, 2023

    work page 2023

  3. [3]

    and Weibull, J

    Bj \"o rnerstedt, J. and Weibull, J. W. Nash equilibrium and evolution by imitation. Technical report, 1994

    work page 1994

  4. [4]

    On the complexity of iterated weak dominance in constant-sum games

    Brandt, F., Brill, M., Fischer, F., and Harrenstein, P. On the complexity of iterated weak dominance in constant-sum games. Theory of Computing Systems, 49: 0 162--181, 2011

    work page 2011

  5. [5]

    and Sandholm, T

    Conitzer, V. and Sandholm, T. Complexity of (iterated) dominance. In Proceedings of the 6th ACM Conference on Electronic Commerce, pp.\ 88--97, 2005

    work page 2005

  6. [6]

    D., and Kamhoua, C

    Fang, F., Liu, S., Basak, A., Zhu, Q., Kiekintveld, C. D., and Kamhoua, C. A. Introduction to Game Theory . Game Theory and Machine Learning for Cyber Security, pp.\ 21--46, 2021

    work page 2021

  7. [7]

    and Toni, F

    Gao, Y. and Toni, F. Potential based Reward Shaping for Hierarchical Reinforcement Learning . In IJCAI, 2015

    work page 2015

  8. [8]

    and Caarls, W

    Garcia, R. and Caarls, W. Online weighted Q-ensembles for reduced hyperparameter tuning in reinforcement learning . Soft Computing, 28 0 (13): 0 8549--8559, 2024

    work page 2024

Show all 47 references
  1. [9]

    Generative Adversarial Nets

    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative Adversarial Nets . In Proc. of NIPS , pp.\ 2672--2680, 2014

    2014

  2. [10]

    and Kudenko, D

    Grzes, M. and Kudenko, D. Learning shaping rewards in model-based reinforcement learning. In Proc. of AAMAS Workshop on Adaptive Learning Agents, volume 115, 2009 a

    2009

  3. [11]

    and Kudenko, D

    Grzes, M. and Kudenko, D. Theoretical and empirical analysis of reward shaping in reinforcement learning. In Proc. of ICMLA, pp.\ 337--344. IEEE, 2009 b

    2009

  4. [12]

    Pre-trained models: Past, present and future

    Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. Pre-trained models: Past, present and future. AI Open, 2: 0 225--250, 2021

    2021

  5. [13]

    CardiffUni - TTCP CAGE Challenge 2 Winning Submission

    Hannay, J. CardiffUni - TTCP CAGE Challenge 2 Winning Submission . https://github.com/john-cardiff/-cyborg-cage-2, 2022

    2022

  6. [14]

    Canaries and Whistles: Resilient Drone Communication Networks with (or without) Deep Reinforcement Learning

    Hicks, C., Mavroudis, V., Foley, M., Davies, T., Highnam, K., and Watson, T. Canaries and Whistles: Resilient Drone Communication Networks with (or without) Deep Reinforcement Learning . In Proc. of the 16th ACM Workshop on AI and Security, AISec '23, pp.\ 91–101, 2023

    2023

  7. [15]

    On autonomous agents in a cyber defence environment

    Kiely, M., Bowman, D., Standen, M., and Moir, C. On autonomous agents in a cyber defence environment. arXiv preprint arXiv:2309.07388, 2023

    2023 arXiv

  8. [16]

    KEEP: A GNN-based PPO Model for MARL

    King, I. KEEP: A GNN-based PPO Model for MARL . https://github.com/cybermonic/cage-4-submission, 2024

    2024

  9. [17]

    and Bowman, B

    King, I. and Bowman, B. CybORG CAGE Challenge 2 KEEP Agent . https://github.com/cybermonic/CAGE-2_KEEP-Agent, 2024

    2024

  10. [18]

    Kingma, D. P. and Ba, J. Adam: A Method for Stochastic Optimization . In Proc. of ICLR, 2014

    2014

  11. [19]

    Kipf, T. N. and Welling, M. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016

    2016 arXiv

  12. [20]

    On the elimination of dominated strategies in stochastic models of evolution with large populations

    Kuzmics, C. On the elimination of dominated strategies in stochastic models of evolution with large populations. Games and Economic Behavior, 72 0 (2): 0 452--466, 2011

    2011

  13. [21]

    A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning

    Lanctot, M., Zambaldi, V., Gruslys, A., Lazaridou, A., Tuyls, K., P \'e rolat, J., Silver, D., and Graepel, T. A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning . In Proc. of NIPS , pp.\ 4190--4203, 2017

    2017

  14. [22]

    Solving large-scale pursuit-evasion games using pre-trained strategies

    Li, S., Wang, X., Zhang, Y., Xue, W., C ern \`y , J., and An, B. Solving large-scale pursuit-evasion games using pre-trained strategies. In Proc. of AAAI, volume 37, pp.\ 11586--11594, 2023

    2023

  15. [23]

    Neu PL : Neural Population Learning

    Liu, S., Marris, L., Hennes, D., Merel, J., Heess, N., and Graepel, T. Neu PL : Neural Population Learning . Proc. of ICLR, 2022

    2022

  16. [24]

    and Llopis Sanchez, S

    Lopes Antunes, D. and Llopis Sanchez, S. The Age of fighting machines: the use of cyber deception for Adversarial Artificial Intelligence in Cyber Defence . In Proc. of the 18th International Conference on Availability, Reliability and Security, pp.\ 1--6, 2023

    2023

  17. [25]

    B., Gordon, G

    McMahan, H. B., Gordon, G. J., and Blum, A. Planning in the presence of cost functions controlled by an adversary. In Proc. of ICML, pp.\ 536--543, 2003

    2003

  18. [26]

    C., Gralewski, L., Menzies, R., et al

    Miles, I., Farmer, S., Foster, D., Harrold, D., Palmer, G., Parry, C., Willis, C., Mont, M. C., Gralewski, L., Menzies, R., et al. Reinforcement learning for autonomous resilient cyber defence. Presented at Black Hat USA, 2024

    2024

  19. [27]

    Non-Cooperative Games

    Nash, J. Non-Cooperative Games . Annals of mathematics, pp.\ 286--295, 1951

    1951

  20. [28]

    Y., Harada, D., and Russell, S

    Ng, A. Y., Harada, D., and Russell, S. Policy invariance under reward transformations: Theory and application to reward shaping . In Proc. of ICML, volume 99, pp.\ 278--287, 1999

    1999

  21. [29]

    V., Nguyen, H

    Nguyen, H. V., Nguyen, H. N., and Uehara, T. Multiple Level Action Embedding for Penetration Testing . In Proc. of ICFNDS, pp.\ 1--9, 2020

    2020

  22. [30]

    O'Driscoll, R., Hagen, C., Bater, J., and Adams, J. M. Multi-Objective Reinforcement Learning for Automated Resilient Cyber Defence . arXiv preprint arXiv:2411.17585, 2024

    2024 arXiv

  23. [31]

    A., Savani, R., Gallego, J., Pol, E

    Oliehoek, F. A., Savani, R., Gallego, J., Pol, E. v. d., and Gro , R. Beyond local Nash equilibria for adversarial networks . In Benelux Conference on Artificial Intelligence, pp.\ 73--89. Springer, 2018

    2018

  24. [32]

    J., and Willis, C

    Palmer, G., Parry, C., Harrold, D. J., and Willis, C. Deep Reinforcement Learning for Autonomous Cyber Operations: A Survey . arXiv preprint arXiv:2310.07745, 2023

    2023 arXiv

  25. [33]

    and Alstr m, P

    Randl v, J. and Alstr m, P. Learning to Drive a Bicycle Using Reinforcement Learning and Shaping. In Proc. of ICML, volume 98, pp.\ 463--471, 1998

    1998

  26. [34]

    Proximal Policy Optimization Algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal Policy Optimization Algorithms . arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  27. [35]

    Cyberbattlesim

    Seifert, C., Betser, M., Blum, W., Bono, J., Farris, K., Goren, E., Grana, J., Holsheimer, K., Marken, B., Neil, J., Nichols, N., Parikh, J., and Wei, H. Cyberbattlesim. https://github.com/microsoft/cyberbattlesim, 2021. Microsoft Defender Research Team

    2021

  28. [36]

    Adversarial agent-learning for cybersecurity: a comparison of algorithms

    Shashkov, A., Hemberg, E., Tulla, M., and O’Reilly, U.-M. Adversarial agent-learning for cybersecurity: a comparison of algorithms. The Knowledge Engineering Review, 38: 0 e3, 2023

    2023

  29. [37]

    Slantchev, B. L. Game theory: Dominance, nash equilibrium, symmetry. Department of Political Science, University of California--San Diego, 2008

    2008

  30. [38]

    J., Kim, J., and Marriott, D

    Standen, M., Lucas, M., Bowman, D., Richer, T. J., Kim, J., and Marriott, D. CybORG: A Gym for the Development of Autonomous Cyber Agents . In IJCAI-21 1st International Workshop on Adaptive Cyber Defense, 2021

    2021

  31. [39]

    Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018

    2018

  32. [40]

    Entity-based Reinforcement Learning for Autonomous Cyber Defence

    Symes Thompson, I., Caron, A., Hicks, C., and Mavroudis, V. Entity-based Reinforcement Learning for Autonomous Cyber Defence . In Proc. of the Workshop on Autonomous Cybersecurity, AutonomousCyber '24, pp.\ 56–67, 2024

    2024

  33. [41]

    Cascaded reinforcement learning agents for large action spaces in autonomous penetration testing

    Tran, K., Standen, M., Kim, J., Bowman, D., Richer, T., Akella, A., and Lin, C.-T. Cascaded reinforcement learning agents for large action spaces in autonomous penetration testing. Applied Sciences, 12 0 (21): 0 11265, 2022

    2022

  34. [42]

    TTCP CAGE Challenge 2

    TTCP CAGE Working Group . TTCP CAGE Challenge 2 . https://github.com/cage-challenge/cage-challenge-2, 2022

    2022

  35. [43]

    TTCP CAGE Challenge 4

    TTCP CAGE Working Group . TTCP CAGE Challenge 4 . https://github.com/cage-challenge/cage-challenge-4, 2023

    2023

  36. [44]

    Zur Theorie der Gesellschaftsspiele

    Von Neumann, J. Zur Theorie der Gesellschaftsspiele . Mathematische annalen, 100 0 (1): 0 295--320, 1928

    1928

  37. [45]

    and Morgenstern, O

    Von Neumann, J. and Morgenstern, O. Theory of games and economic behavior: 60th anniversary commemorative edition. In Theory of games and economic behavior. Princeton university press, 2007

    2007

  38. [46]

    Vyas, S., Hannay, J., Bolton, A., and Burnap, P. P. Automated cyber defence: A review. arXiv preprint arXiv:2303.04926, 2023

    2023 arXiv

  39. [47]

    Wellman, M. P. and Mayo, K. Navigating in a space of game views. JAAMAS, 38 0 (2): 0 31, 2024

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.