Pith. sign in

REVIEW 5 major objections 5 minor 48 references

PoolFlip: A Multi-Agent Reinforcement Learning Security Environment for Cyber Defense

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Population-based training makes FlipIt defenders roughly twice as effective against unseen attacker variants, the paper claims.

desk verdict A useful new FlipIt-style gym and a sensible PSRO application, but the 2x generalization claim needs seed-level statistics and stronger baselines before it can be taken at face value. read the letter →

arxiv 2508.19488 v1 pith:GRFLJ23F submitted 2025-08-27 cs.LG cs.AIcs.CR

Xavier Cadet , Simona Boboila , Sie Hendrata Dharmawan , Alina Oprea , Peter Chin This is my paper · ORCID
classification cs.LGcs.AIcs.CR
keywords FlipItmulti-agentreinforcementlearningpolicyspaceresponseoraclecyberdefensestealthytakeoverpopulation-basedtraininggeneralizationsecuritygames
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a defender for the stealthy-takeover game FlipIt can be trained to handle attackers it has never seen, provided it trains against a diverse population of attackers rather than a single opponent. To show this, the authors build PoolFlip, a multi-agent environment that adds a Check action and a zoo of parameterized heuristic attackers and defenders to the original FlipIt game. On top of it they run Flip-PSRO, a Policy-Space Response Oracle algorithm whose meta-strategy solver is driven by new ownership-based objectives, and report that the resulting defenders transfer to unseen attack variants roughly twice as effectively as Iterated Best Response baselines (average reward 32.3 versus 14.2). The reason this matters is that deployed cyber defenses face shifting adversaries and cannot relabel and retrain on each new attack family in real time.

What carries the argument

The central machinery is Flip-PSRO, an adaptation of the Policy-Space Response Oracle (PSRO) framework to the FlipIt game, with Proximal Policy Optimization (PPO) as the response oracle that iteratively learns a best response against opponents sampled from a growing pool. The new part is the meta-strategy solver: instead of a plain win-loss payoff, the meta-game utility matrix is filled with response objectives specific to stealthy takeovers, chiefly the win rate by ownership (the fraction of time steps a player controls the resource above a target t%) and the normalized performance gap against specialist policies. A softmax over these utilities turns raw scores into sampling probabilities, so each training round focuses the defender on the opponents it is worst at. This mechanism is what carries the generalization argument: prioritizing hard opponents during population training is what lets the final policy beat unseen variants that share structure with the pool.

What would settle it

Evaluate the trained MSS-O50% defender against a held-out set composed mostly of attack variants whose phases are not multiples of 4 (for example P(6), P(10), PAC(6), PAC(10), and odd-phase Burst variants) and measure the average reward against the IBR baseline; if the 2x advantage shrinks or reverses on this out-of-convex-hull set, the generalization claim as stated is falsified, and Table 5's near-zero or negative rewards on P(6) and PAC(6) are already a partial instance of that test.

Watch

Extended reading notes

Core claim

Flip-PSRO defenders, trained with a softmax meta-strategy solver that prioritizes opponents against whom the defender's ownership falls below a target (50% or 70% of time steps), generalize to attack variants not present in training. On the in-pool evaluation, the ownership-based metasolvers reach average rewards of 27–31, above the Iterated Best Response baseline (23.2) and the best heuristic (13.8); on held-out attack variants, the 50%-ownership variant reaches an average transfer reward of 32.3 versus 14.2 for IBR, a difference the paper describes as roughly 2x more effective. The same ownership objective raises the defender's average resource ownership from 76% to 81% as the target rises from 50% to 70%, while keeping reward high, showing that control and performance can be optimized together rather than traded off. The paper also finds that transfer works when the unseen opponent lies in the convex hull of the training strategies, and that out-of-hull phases such as P(6) and PAC(6) defeat both methods.

Load-bearing premise

The load-bearing premise is that a pool of five parameterized heuristics, plus the phase and burst variants used for transfer testing, fairly represents the distribution of adversaries a real defender will face; the paper itself concedes that transfer only works for opponents inside the convex hull of training strategies, and the reported 2x advantage does not hold for phases that are not multiples of 4, such as P(6) and PAC(6).

Editorial extensions

If this is right

  • Population-based training is a viable route to proactive defenses: a single Flip-PSRO policy can replace a fleet of specialists, each tuned to one known attack pattern.
  • Ownership-based objectives give operators a dial: raising the ownership target trades a little reward for substantially more resource availability, which matters for availability-critical systems.
  • Defenders trained this way are predictable in their failure modes: they generalize to unseen opponents inside the convex hull of training strategies, and degrade sharply outside it, so a deployment should audit the pool's coverage of anticipated attack variants.
  • The Check action becomes strategically decisive when it is cheap: at 20x lower check cost, the best heuristic switches to checking every turn and flipping immediately on loss of control, confirming that monitoring is the backbone of stealthy-takeover defense.
  • Self-play in this simple symmetric setting stalls into a check-flip stalemate, so the paper's own evidence suggests self-play pays off only in more complex action spaces, not in the current PoolFlip configuration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence the paper leaves implicit is that the 2x transfer advantage is specific to test sets whose phases are commensurate with the training phase of 4; a held-out set built mostly from non-multiple phases (6, 10, ...) should erode or erase the advantage, and Table 5's negative rewards on P(6) and PAC(6) already point that way.
  • The ownership objective can be read as a mechanism design trick: it encodes an availability constraint into the meta-game without changing the per-step reward, so the same idea should port to other resource-control games where uptime, not just reward, is the security goal.
  • Because the paper assumes a symmetric game with identical costs and gains for both players, the natural next test is asymmetric costs; an attacker with cheaper flips or a defender with expensive checks would break the shared-policy assumption and likely require separate populations, which the paper itself lists as future work.
  • Pool diversity, not raw training compute, is the lever that drives transfer; this suggests a practical benchmark recipe: measure a defense method by its performance on a held-out lattice of phase shifts rather than on the training opponents themselves.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces PoolFlip, a multi-agent reinforcement learning gym environment that extends the FlipIt stealthy-takeover game with a Check action, parameterized heuristic attackers/defenders, and support for trainable PPO agents. The authors also propose Flip-PSRO, a Policy-Space Response Oracle variant that trains a defender against a pool of heuristic opponents using several meta-strategy solvers, including reward, normalized performance gap, and ownership-target-based utilities. Experiments compare Flip-PSRO variants against specialists, Iterated Best Response (IBR), and heuristic baselines on in-pool reward (Table 3) and ownership (Table 4), and evaluate transfer to unseen attack variants (Table 5). The central claim is that Flip-PSRO defenders are roughly 2x more effective than IBR at generalizing to unseen heuristic attacks, based on an average transfer reward of 32.3 for MSS-O50% versus 14.2 for IBR.

Significance. If the central claim is substantiated, the paper would make a useful contribution by demonstrating that population-based training in the style of PSRO can produce defenders that transfer to shifted attacker behaviors in FlipIt-like games, and by releasing an open-source environment that future work can build on. The paper also introduces a novel response objective (ownership-based win rate) and thoughtfully discusses the convex-hull limitation on transfer, which is a strength. The reported experiments cover a reasonable set of heuristics and baselines, and the open-source code is a concrete asset. However, the headline quantitative claim currently rests on a single point estimate with no seed-level statistics, and the paper does not benchmark against the most closely related prior RL methods for FlipIt (QFlip and the DQN-based approach of Greige et al.), which tempers the significance as stated.

major comments (5)
  1. [Section 6, Q4, Table 5] The central 2x transfer claim is based on a single point estimate: Table 5 reports average rewards without standard errors, confidence intervals, or seed-level results. PPO training is stochastic, and the gap between MSS-O50% (32.3) and IBR (14.2) could be driven by initialization luck or the choice of evaluation episodes. The paper should report results over multiple random seeds, provide per-seed or interval estimates, and include a significance test (e.g., paired bootstrap or Wilcoxon signed-rank) for the comparison against IBR. Without this, the advertised factor is not statistically established.
  2. [Section 6, Q4, Table 5] The transfer set consists of seven variants of the same five heuristics used in training, and the paper itself states that transfer works best when unseen opponents lie in the convex hull of training strategies [47,48]. The results show that non-multiple phases fail: MSS-O70% scores 4.2 on P(6) and PAC(6), and MSS-O50% scores 4.2 on B(8,6). The abstract's unqualified '2x more effective' claim overstates the result. The authors should either qualify the claim to within-family phase-multiple shifts or include out-of-convex-hull transfer tests (e.g., phases not multiples of 4, or learned adaptive attackers).
  3. [Tables 3 and 5] The headline 2x improvement is achieved by MSS-O50%, but Table 5 shows MSS-O70% averages only 20.1, which is 5.9 above IBR (14.2), not 2x. The paper does not specify a principled criterion for selecting MSS-O50% as the representative Flip-PSRO variant; choosing the best-performing meta-solver after seeing the transfer results inflates the claim. The authors should either preselect the meta-solver, report all variants and adjust the claim accordingly, or justify why MSS-O50% is the appropriate deployment choice.
  4. [Section 5.2 and Table 4] The ownership-based response objective is, by construction, the utility optimized by MSS-Ot%; therefore, the high ownership percentages in Table 4 are partly a check that the optimizer is working, not an independent discovery. The claim that the ownership-based utility 'ensures that Flip-PSRO defenders maintain a high level of control while optimizing performance' needs a different kind of evidence, for example a comparison against a reward-only training objective matched for total reward, or a discussion of the trade-off frontier. As written, the ownership column is to some degree circular.
  5. [Sections 2 and 6] The paper motivates the work by limitations of prior RL approaches to FlipIt, namely QFlip (Oakley and Oprea [17]) and the DQN-based approach of Greige et al. [18], but no experiment compares Flip-PSRO against these methods. Since these are the most directly relevant baselines, the claim that Flip-PSRO improves upon prior RL work is not empirically supported. The authors should include these baselines or explicitly state why they are not comparable.
minor comments (5)
  1. [Section 6] The evaluation section says results are 'averaged over 100 episodes' but does not state the number of independent training runs/seeds; please specify the seed protocol.
  2. [Table 4] The average ownership for MSS-Gap is reported as 72.28, while other entries use at most one decimal place; please make the precision consistent.
  3. [Section 4] The naming is inconsistent: the text introduces 'Periodic Aggressive Check (PAC)' and later 'PAC(1)/UPAC', while Section 6 Q1 refers to 'PAC(1)'. Please unify the nomenclature.
  4. [Section 6, Q5] The self-play experiment uses a Check cost of 0.1, whereas the main experiments use Check cost 1.0; please explain why this cost is changed and how it affects the comparison.
  5. [Eq. (5)] The softmax meta-strategy solver in Eq. (5) is not defined with a temperature parameter; please specify whether a temperature is used and how the softmax is computed over the utility matrix.

Circularity Check

1 steps flagged · score 3.0 of 10

Main 2x generalization claim is not circular; one supporting ownership result is tautological because the ownership-based meta-solver optimizes the same ownership metric it reports.

  1. self definitional [Section 5.2 (Response Objectives), Section 5.3 Eq. (5), Section 6 Q3 / Table 4]
    "We denote by t% the required duration of ownership for the defender to be declared the winner. The win rate by ownership for the player i represents the fraction of games won by i, using a predefined ownership target of t%."

    The MSS-O meta-solver is defined to maximize exactly this win-rate-by-ownership objective: Eq. (5) sets sigma = softmax(U^Pi_RO)(Pi), and for MSS-O the response objective RO is the ownership-based win rate built from Eq. (4). Table 4 then reports the defender's ownership percentage as evidence that the ownership utility 'ensures high control.' That outcome is the optimized objective itself; observing high ownership under MSS-O is a sanity check on the optimizer, not an independent finding. The reward and transfer claims (Tables 3, 5) do not reduce to this objective and are evaluated against external baselines, so the central generalization result is not circular.

full rationale

The paper's headline generalization claim (2x reward vs IBR on unseen variants, Table 5) is an empirical comparison against an independent baseline and is not produced by fitting a parameter or by a self-citation chain. Training is on a fixed pool; transfer variants are constructed by changing phase/burst and are not used to select or tune the reported model. The paper even states the convex-hull caveat and reports poor transfer to P(6)/PAC(6), which cuts against cherry-picking. The only load-bearing step that reduces to its own input is the ownership-control result: the win-rate-by-ownership response objective is, by Eq. (4) and Eq. (5), the function MSS-O maximizes, so Table 4's high ownership partly re-states the training target. This is a secondary supporting claim, not the central derivation. Self-citations (Oakley/Oprea, Greige/Chin) are background only. Overall circularity is low but nonzero.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on hand-set environment parameters and on the representativeness of the heuristic pool. No physical constants or externally fitted parameters are involved; the free parameters listed below are experimental design choices that can change which method wins, such as Check cost determining whether PAC(1) is best and ownership target changing MSS behavior.

free parameters (8)
  • Action costs (Sleep=0.0, Check=1.0, Flip=2.0) = 0.0 / 1.0 / 2.0
    Chosen by hand in Section 6 PoolFlip settings; Section 6 Q1 shows the best heuristic changes when Check cost is lowered 20x, so the comparison is cost-dependent.
  • Resource gain per ownership time step = 1.0
    Chosen by hand; with c>g the agent must wait at least c/g steps between flips to earn positive reward (Section 3.3), so the gain value directly shapes viable strategies.
  • Game horizon T = 100 time steps
    Chosen in Section 6; rewards and ownership scores are accumulated over T, and a longer horizon could alter the relative advantage of checking and flipping strategies.
  • Heuristic pool default parameters = phase=4, random delay, burst=3 (phase=8 for Burst), awakening rate=0.05
    Default heuristic parameters in Section 6 define the training pool; Table 2 shows performance strongly varies with these parameters, and transfer is tested only on small variations of them.
  • Ownership win target t% = 50% and 70%
    Section 5.2 and Tables 3-4; t% defines the win-rate utility for the MSS-O variants, and changing it changes sampling weights, average reward (31.1 vs 27.4), and ownership (76.3 vs 81.1).
  • PPO hyperparameters and update schedule = LR=0.001, gamma=0.99, clip=0.2, 4 epochs, update every 10 episodes, entropy=0.01
    Section 6 Hyper-parameters in PPO; these are standard but hand-selected, and no sensitivity analysis is reported for the trained agents.
  • Observation memory limit M and PPO network architecture = not reported
    Section 3.2 defines observation size as 2+2M per resource, but the experimental section does not state M or the neural network architecture; these choices affect what policies can learn and are necessary to reproduce results.
  • Check cost in self-play experiment = 0.1
    Section 6 Q5 uses a different Check cost (0.1) than the main experiments (1.0), so the self-play results are not directly comparable to the other tables.
assumptions (5)
  • domain assumption FlipIt/PoolFlip dynamics are a faithful abstraction of stealthy takeover cyber defense.
    The paper's practical relevance claim depends on the game's costs, gains, and stealthy observations capturing real APT-style interactions; if the abstraction omits key operational details, the trained policies may not transfer to deployment.
  • ad hoc to paper The fixed pool of five heuristic attackers (Periodic, Burst, Awakening, Periodic Check, PAC) with default parameters represents a realistic distribution of unknown adversaries.
    Section 5.1 and Section 6 define the pool by hand with no external dataset or accepted benchmark; the transfer results in Q4 are conditional on this pool.
  • domain assumption Specialists S_H trained against individual heuristics are good approximations of best responses, so the normalized performance gap is a meaningful utility.
    Section 5.2 uses specialist rewards as reference values for the gap objective; if specialists are undertrained or non-unique, the MSS-Gap sampling is mis-calibrated.
  • domain assumption PPO serves as a sufficiently strong best-response oracle inside the PSRO loop.
    Algorithm 1 assumes the trained PPO policy approximates the best response to the sampled opponent mixture; no convergence or approximation bound is provided for this partially observed game.
  • ad hoc to paper The defender and attacker can be treated symmetrically despite the defender's initial ownership advantage.
    Section 5.1 asserts game symmetry, while Section 6 notes the defender starts with the resource; the self-play experiment in Q5 relies on this interchangeability even though rewards are not perfectly symmetric.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PoolFlip: A Multi-Agent Reinforcement Learning Security Environment for Cyber Defense." pith.science (2026). https://pith.science/paper/GRFLJ23F

@misc{pith2026250819488,
  author       = {Pith},
  title        = {Pith review of: PoolFlip: A Multi-Agent Reinforcement Learning Security Environment for Cyber Defense},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GRFLJ23F}},
  note         = {Machine review of arXiv:2508.19488}
}
abstract

Cyber defense requires automating defensive decision-making under stealthy, deceptive, and continuously evolving adversarial strategies. The FlipIt game provides a foundational framework for modeling interactions between a defender and an advanced adversary that compromises a system without being immediately detected. In FlipIt, the attacker and defender compete to control a shared resource by performing a Flip action and paying a cost. However, the existing FlipIt frameworks rely on a small number of heuristics or specialized learning techniques, which can lead to brittleness and the inability to adapt to new attacks. To address these limitations, we introduce PoolFlip, a multi-agent gym environment that extends the FlipIt game to allow efficient learning for attackers and defenders. Furthermore, we propose Flip-PSRO, a multi-agent reinforcement learning (MARL) approach that leverages population-based training to train defender agents equipped to generalize against a range of unknown, potentially adaptive opponents. Our empirical results suggest that Flip-PSRO defenders are $2\times$ more effective than baselines to generalize to a heuristic attack not exposed in training. In addition, our newly designed ownership-based utility functions ensure that Flip-PSRO defenders maintain a high level of control while optimizing performance.

Figures

Figures reproduced from arXiv: 2508.19488 by the authors.

Figure 1
Figure 1. Defender Reward (averaged over 100 episodes), where a heuristic defender is playing [PITH_FULL_IMAGE:figures/full_fig_p012_1.png] view at source ↗
Figure 2
Figure 2. Episode rewards for the trained agent (defender) against various strategies from the pool, [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Trained opponents playing against each other. We select an early variant produced before [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 38 canonical work pages

  1. [17]

    QFlip: An adaptive reinforcement learning strategy for the FlipIt security game

    Lisa Oakley and Alina Oprea. QFlip: An adaptive reinforcement learning strategy for the FlipIt security game. InDecision and Game Theory for Security, pages 364–384. Springer, 2019

    work page 2019

  2. [18]

    Deep reinforcement learning for FlipIt security game

    Laura Greige and Peter Chin. Deep reinforcement learning for FlipIt security game. InComplex Networks & Their Applications X, pages 831–843. Springer, 2022

    work page 2022

  3. [1]

    Deep learning methods in network intrusion detection: Asurveyandanobjectivecomparison

    Sunanda Gamage and Jagath Samarabandu. Deep learning methods in network intrusion detection: Asurveyandanobjectivecomparison. Journal of Network and Computer Applications, 169:102767, 2020

    work page 2020

  4. [2]

    Unsupervised anomaly detection in network intrusion detection using clusters

    Kingsly Leung and Christopher Leckie. Unsupervised anomaly detection in network intrusion detection using clusters. InACSC, page 333–342, 2005

    work page 2005

  5. [3]

    Beehive: large-scale log analysis for detecting suspicious activity in enterprise networks

    Ting-Fang Yen, Alina Oprea, Kaan Onarlioglu, Todd Leetham, William Robertson, Ari Juels, and Engin Kirda. Beehive: large-scale log analysis for detecting suspicious activity in enterprise networks. InACSAC, page 199–208, 2013

    work page 2013

  6. [4]

    Effectiveness of AI/ML in SOAR (Security Automation and Orchestration) Platforms

    Srihari Subudhi. Effectiveness of AI/ML in SOAR (Security Automation and Orchestration) Platforms. IJSR, 13(8):201–206, 2024. 16

    work page 2024

  7. [5]

    Advancing cybersecurity: a comprehensive review of ai-driven detection techniques.Journal of Big Data, 11(1):105, 2024

    Aya H Salem, Safaa M Azzam, Amr A Abohany, and O E Emam. Advancing cybersecurity: a comprehensive review of ai-driven detection techniques.Journal of Big Data, 11(1):105, 2024

    work page 2024

  8. [6]

    Cyber-security and reinforcement learning — a brief survey.Engineering Applications of Artificial Intelligence, 114:105116, 2022

    Giovanni Apruzzese, Pavel Laskov, and Johannes Schneider. Cyber-security and reinforcement learning — a brief survey.Engineering Applications of Artificial Intelligence, 114:105116, 2022

    work page 2022

Show all 48 references
  1. [7]

    Multi-agent reinforcement learning for cybersecurity: Classification and survey

    Multiple Authors. Multi-agent reinforcement learning for cybersecurity: Classification and survey. Journal of Network and Computer Applications, 2025

    2025

  2. [8]

    Optimal defender strategies for CAGE-2 using causal modeling and tree search.arXiv preprint arXiv:2407.11070, 2024

    Kim Hammar, Neil Dhir, and Rolf Stadler. Optimal defender strategies for CAGE-2 using causal modeling and tree search.arXiv preprint arXiv:2407.11070, 2024

    2024 arXiv

  3. [9]

    Finding the optimal security policies for autonomous cyber operations with competitive reinforcement learning.IEEE Access, 12:120292– 120305, 2024

    Garrett Mcdonald, Li Li, and Ranwa Al Mallah. Finding the optimal security policies for autonomous cyber operations with competitive reinforcement learning.IEEE Access, 12:120292– 120305, 2024

    2024

  4. [10]

    Deep reinforcement learning for cyber secu- rity

    Thanh Thi Nguyen and Vijay Janapa Reddi. Deep reinforcement learning for cyber secu- rity. IEEE Transactions on Neural Networks and Learning Systems, 2021. arXiv preprint arXiv:1906.05799

    2021 arXiv

  5. [11]

    Learning cyber defence tactics from scratch with multi-agent reinforcement learning, 2023

    Jacob Wiebe, Ranwa Al Mallah, and Li Li. Learning cyber defence tactics from scratch with multi-agent reinforcement learning, 2023. arXiv:2310.05939

    2023 arXiv

  6. [12]

    CybORG: A Gym for the Development of Autonomous Cyber Agents

    Maxwell Standen, Martin Lucas, Bowman David, Toby J˙Richer, Junae Kim, and Damian Marriott. CybORG: A Gym for the Development of Autonomous Cyber Agents. InIJCAI-21 1st International Workshop on Adaptive Cyber Defense.arXiv, 2021

    2021

  7. [13]

    CyberBattleSim - Microsoft Research

    Microsoft Research. CyberBattleSim - Microsoft Research. https://www.microsoft.com/ en-us/research/project/cyberbattlesim/, 2020. An experimentation and research platform to investigate the interaction of automated agents in an abstract simulated network environments

    2020

  8. [14]

    Marten van Dijk, Ari Juels, Alina Oprea, and Ronald L. Rivest. FlipIt: The game of stealthy takeover. Journal of Cryptology, 26(4):655–713, 2013

    2013

  9. [15]

    FlipThem: Modeling Targeted Attacks with FlipIt for Multiple Resources

    Aron Laszka, Gabor Horvath, Mark Felegyhazi, and Levente Buttyán. FlipThem: Modeling Targeted Attacks with FlipIt for Multiple Resources. InDecision and Game Theory for Security, pages 175–194, 2014

    2014

  10. [16]

    Are we compromised? modelling security assessment games

    Viet Pham and Carlos Cid. Are we compromised? modelling security assessment games. In Decision and Game Theory for Security, pages 234–247. Springer, 2012

    2012

  11. [19]

    Christopher J. C. H. Watkins and Peter Dayan. Q-learning.Machine Learning, 8(3):279–292, 1992. 17

    1992

  12. [20]

    Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, D

    Volodymyr Mnih, K. Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, D. Wierstra, and Martin A. Riedmiller. Playing Atari with deep reinforcement learning, 2013. arXiv:1312.5602

    2013 arXiv

  13. [21]

    Grandmaster level in StarCraft II using multi-agent reinforcement learning

    Oriol Vinyals and others. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782):350–354, November 2019

    2019

  14. [22]

    Mastering the game of go with deep neural networks and tree search.Nature, 529(7587):484–489, 2016

    David Silver et al. Mastering the game of go with deep neural networks and tree search.Nature, 529(7587):484–489, 2016

    2016

  15. [23]

    Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games

    Stephen Mcaleer, JB Lanier, Roy Fox, and Pierre Baldi. Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games. InNeurIPS, volume 33, pages 20238–20248, 2020

    2020

  16. [24]

    A unified game-theoretic approach to multiagent reinforcement learning, 2017

    Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Perolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning, 2017. arXiv:1711.00832

    2017 arXiv

  17. [25]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  18. [26]

    FlipNet: Modeling covert and persistent attacks on networked resources

    Sudip Saha, Anil Vullikanti, and Mahantesh Halappanavar. FlipNet: Modeling covert and persistent attacks on networked resources. InICDCS, pages 2444–2451, 2017

    2017

  19. [27]

    Flipleakage: A game-theoretic approach to protect against stealthy attackers in the presence of information leakage

    Sadegh Farhang and Jens Grossklags. Flipleakage: A game-theoretic approach to protect against stealthy attackers in the presence of information leakage. InProc. of GameSec, pages 195–214, 2016

    2016

  20. [28]

    Dynamic defense strategy against advanced persistent threat with insiders

    Pengfei Hu, Hongxing Li, Hao Fu, Derya Cansever, and Prasant Mohapatra. Dynamic defense strategy against advanced persistent threat with insiders. InINFOCOM, 2015

    2015

  21. [29]

    Max Olan Smith, Thomas Anthony, Yongzhao Wang, and Michael P. Wellman. Learning to play against any mixture of opponents, 2021. arXiv:2009.14180

    2021 arXiv

  22. [30]

    Human-level control through deep reinforcement learning.Nature, 518(7540):529–533, 2015

    Volodymyr Mnih et al. Human-level control through deep reinforcement learning.Nature, 518(7540):529–533, 2015

    2015

  23. [31]

    Hierarchical multi-agent reinforcement learning for cyber network defense

    Aditya Vikram Singh, Ethan Rathbun, Emma Graham, Lisa Oakley, Simona Boboila, Alina Oprea, and Peter Chin. Hierarchical multi-agent reinforcement learning for cyber network defense. arXiv preprint arXiv:2410.17351, 2024

    2024 arXiv

  24. [32]

    Oliehoek

    Ariyan Bighashdel, Yongzhao Wang, Stephen McAleer, Rahul Savani, and Frans A. Oliehoek. Policy space response oracles: a survey. InIJCAI, 2024

    2024

  25. [33]

    Comput- ing optimal equilibria and mechanisms via learning in zero-sum extensive-form games

    Brian Hu Zhang, Gabriele Farina, Ioannis Anagnostides, Federico Cacciamani, Stephen McAleer, Andreas Haupt, Andrea Celli, Nicola Gatti, Vincent Conitzer, and Tuomas Sandholm. Comput- ing optimal equilibria and mechanisms via learning in zero-sum extensive-form games. InNIPS, 2023

    2023

  26. [34]

    Deep reinforcement learning for green security games with real-time information

    Yufei Wang, Zheyuan Ryan Shi, Lantao Yu, Yi Wu, Rohit Singh, Lucas Joppa, and Fei Fang. Deep reinforcement learning for green security games with real-time information. InAAAI, 2019. 18

    2019

  27. [35]

    Robust reinforcement learning under minimax regret for green security, 2021

    Lily Xu, Andrew Perrault, Fei Fang, Haipeng Chen, and Milind Tambe. Robust reinforcement learning under minimax regret for green security, 2021. arXiv:2106.08413

    2021 arXiv

  28. [36]

    Game-theoretic robust reinforcement learning handles temporally-coupled perturbations, 2024

    Yongyuan Liang, Yanchao Sun, Ruijie Zheng, Xiangyu Liu, Benjamin Eysenbach, Tuomas Sandholm, Furong Huang, and Stephen McAleer. Game-theoretic robust reinforcement learning handles temporally-coupled perturbations, 2024. arXiv:2307.12062

    2024 arXiv

  29. [37]

    Patrol: Provable defense against adversarial policy in two-player games

    Wenbo Guo, Xian Wu, Lun Wang, Xinyu Xing, and Dawn Song. Patrol: Provable defense against adversarial policy in two-player games. InUSENIX Security Symposium, 2023

    2023

  30. [38]

    Efficient policy space response oracles, 2022

    Ming Zhou, Jingxiao Chen, Ying Wen, Weinan Zhang, Yaodong Yang, Yong Yu, and Jun Wang. Efficient policy space response oracles, 2022. arXiv:2202.00633

    2022 arXiv

  31. [39]

    A survey on self-play methods in reinforcement learning, 2025

    Ruize Zhang, Zelai Xu, Chengdong Ma, Chao Yu, Wei-Wei Tu, Wenhao Tang, Shiyu Huang, Deheng Ye, Wenbo Ding, Yaodong Yang, and Yu Wang. A survey on self-play methods in reinforcement learning, 2025. arXiv:2408.01072

    2025

  32. [40]

    Evolving diverse red-team language models in multi-round multi-agent games, 2024

    Chengdong Ma, Ziran Yang, Hai Ci, Jun Gao, Minquan Gao, Xuehai Pan, and Yaodong Yang. Evolving diverse red-team language models in multi-round multi-agent games, 2024. arXiv:2310.00322

    2024 arXiv

  33. [41]

    Finding needles in a moving haystack: Prioritizing alerts with adversarial reinforcement learning

    Liang Tong, Aron Laszka, Chao Yan, Ning Zhang, and Yevgeniy Vorobeychik. Finding needles in a moving haystack: Prioritizing alerts with adversarial reinforcement learning. AAAI, 34(01):946–953, Apr. 2020

    2020

  34. [42]

    Lee, Benjamin Lee, G

    Jiaxun Cui, Xiaomeng Yang, Mulong Luo, Geunbae Lee, Peter Stone, Hsien-Hsin S. Lee, Benjamin Lee, G. Edward Suh, Wenjie Xiong, and Yuandong Tian. Macta: A multi-agent reinforcement learning approach for cache timing attacks and detection. InICLR, May 2023

    2023

  35. [43]

    Oliehoek and Chris Amato

    Frans A. Oliehoek and Chris Amato. A concise introduction to decentralized POMDPs. In SpringerBriefs in Intelligent Systems, 2016

    2016

  36. [44]

    Scheduled Task/Job: Cron

    MITRE ATT&CK. Scheduled Task/Job: Cron. https://attack.mitre.org/techniques/ T1053/003/. [Online; accessed 4-June-2025]

    2025

  37. [45]

    On autonomous agents in a cyber defence environment, 2023

    Mitchell Kiely, David Bowman, Maxwell Standen, and Christopher Moir. On autonomous agents in a cyber defence environment, 2023. arXiv:2309.07388

    2023 arXiv

  38. [46]

    Algorithmic game theory.Commun

    Tim Roughgarden. Algorithmic game theory.Commun. ACM, 53(7):78–86, 2010

    2010

  39. [47]

    Fusion-PSRO: Nash policy fusion for policy space response oracles, 2025

    Jiesong Lian, Yucong Huang, Chengdong Ma, Mingzhi Wang, Ying Wen, Long Hu, and Yixue Hao. Fusion-PSRO: Nash policy fusion for policy space response oracles, 2025. arXiv:2405.21027

    2025

  40. [48]

    Policy space diversity for non-transitive games

    Jian Yao, Weiming Liu, Haobo Fu, Yaodong Yang, Stephen McAleer, Qiang Fu, and Wei Yang. Policy space diversity for non-transitive games. InNeurIPS, 2023. 19

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.