{"id":"de3a376a-03b1-4401-a69e-78531c0067ae","arxiv_id":"2501.19206","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An MRO framework with VF-PBRS reward shaping lets multiple DRL cyber agents be compared game-theoretically, yielding robust defenders in CybORG CAGE Challenges 2 and 4.","lead":"Researchers extended the double oracle algorithm so that multiple cyber-defence and attack learning agents can be evaluated together as an empirical game, and added value-function-based reward shaping to speed up training. The approach produced cyber-defence policies that withstood adaptive attackers in two CybORG CAGE challenges, offering a way to test and assure autonomous cyber defenders before deployment.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run, 100-episode payoff estimates leave the central robustness claim under-determined; a bootstrap or repeated-run check is needed before the empirical game value is treated as reliable.","rationale":"The reader's weakest assumption—that 100-episode payoff estimates and a single MRO run per environment are adequate for the equilibrium and robustness conclusions—is the most load-bearing concern I can identify. It directly affects the central claim that defenders trained with MRO/VF-PBRS are robust to learned attackers: if the payoff matrix is noisy or the run is unrepresentative, the reported equilibrium value and the exploitability-based termination criterion are not trustworthy. The paper does provide significant evidence in places, notably the significant VF-PBRS improvement for Blue GPPO in CC2 (Figure 2) and the clear behavioural analysis (e.g., BlockTrafficZone in CC4), so the work is not without support. However, the main game-theoretic numbers and the 'Red cannot find new TTPs' conclusion lack repeated-seed or bootstrap-based error quantification. I also note that Theorem 5 in Appendix A.2.2 is flawed: the assertion that ADO's mixture is a subset of MRO's mixture (µ'_p1 ⊂ µ_p1) is unjustified, since the algorithms follow different policy trajectories. But that theorem concerns a claimed advantage of MRO over ADO, not the convergence guarantee (which appears sound), so it is secondary to the empirical reliability issue. The conditional verdict is appropriate: the framework is promising and the theory mostly intact, but the central empirical claim needs the additional statistical support described above before it can be fully accepted.","tokens_in":30856,"tokens_out":7272,"duration_ms":70864,"concrete_test":"Bootstrap the 100 evaluation episodes per policy pair in the final CC2 empirical game (Figure 3) by resampling with replacement, and for each of at least 1000 bootstrap samples recompute the Nash equilibrium and the game value GBlue(µBlue, µRed). If the 95% confidence interval of the game value spans more than about ±2 points, or if the support of the equilibrium mixture changes across samples, then the single-run 100-episode payoff matrix is insufficient to support the robustness claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim—that final Blue mixtures obtained via MRO are robust to learned attackers—rests on the accuracy of payoff matrices estimated from 100 evaluation episodes per policy pair, and on a single MRO trajectory per environment (Sections 7.1 and 7.2, Appendix C). The algorithm's termination criterion and Nash solving treat these noisy estimates as exact. Table 1 reports standard errors of 0.5–3.7 for 100-episode evaluations, while the reported equilibrium differences that support 'no significant improvement' are often only a few points (e.g., Red VF-PBRS vs vanilla means 24.72/24.85 vs 24.22, explicitly not significant). Yet the main game values, GBlue(µBlue, µRed) = −29.63 (CC2) and −411.91 (CC4), and the conclusion that Red 'struggles to find new successful TTPs' are presented without any repeated-run error bars. The single MRO run per gym means the entire trajectory of which oracles produced which policies, whether Red found π21_Red, and whether Blue neutralized it, is one stochastic realization. If the payoff noise is of the same order as the gaps between candidate strategies, the Nash solver could select a spurious mixture and GE ≤ 0 could be triggered by estimation variance rather than true convergence. Thus the load-bearing premise is that a 100-episode, single-run empirical game is representative enough for equilibrium and exploitability claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multiple response oracles (MRO) extension of the approximate double oracle (ADO) algorithm for evaluating and combining deep reinforcement learning agents for autonomous cyber defence. It introduces value-function potential-based reward shaping (VF-PBRS) with mixture-weighted ensembles, together with pre-trained model sampling, and provides an empirical game-theoretic analysis in CybORG CAGE Challenges 2 and 4. The authors claim that MRO preserves ADO's convergence guarantees, that adversarial training produces Blue mixtures robust to learned Red agents, and that VF-PBRS yields significantly stronger policies than vanilla training in at least one setting.","tokens_in":31154,"tokens_out":11408,"duration_ms":108128,"significance":"If the MRO convergence result and the empirical findings hold, the paper offers a useful framework for comparing and ensembling heterogeneous ACD-DRL approaches, an area with little prior work. The paper's strengths include a natural and practically motivated extension of DO, a mostly correct convergence argument for MRO in Appendix A.2.1, a careful VF-PBRS design rooted in Ng et al.'s policy invariance theorem, and detailed appendices with hyperparameters, network diagrams, and action-level analyses. The VF-PBRS comparison for Blue GPPO in CC2 is supported by ten runs per setting with significance tests. However, the central empirical claims rest on a single MRO run per environment and on payoff matrices estimated from 100 evaluation episodes, and the claimed theoretical guarantee that MRO strictly improves on ADO is not proven. These issues currently limit the certainty with which the conclusions can be accepted.","major_comments":[{"comment":"The proof that MRO is guaranteed to produce stronger mixture agents than ADO is circular. Definition A.3 defines 'minimal viable support' but does not establish that the policy πi,n_p1 from Assumption A.1 belongs to such a support; Eq. (24) is essentially the assertion to be proved. Moreover, Eq. (26), 'µ′_p1 ⊂ µp1', need not hold: ADO and MRO are different algorithms that may generate different policy sets over the same iterations, so an ADO mixture can include policies not present in the MRO mixture. The claim in Section 4.2 that MRO will produce stronger mixtures is therefore unsupported as stated and should be removed or replaced with a correct theorem.","section":"Section 4.2 and Appendix A.2.2 (Theorem 5, Definition A.3, Eqs. 24-26)"},{"comment":"The central empirical claims—the final game values GBlue(µBlue, µRed) = −29.63 in CC2 and −411.91 in CC4, the statement that Red 'struggles to find new successful TTPs', and the convergence observations around iteration 20—are based on a single MRO run per environment, with payoff entries estimated from 100 evaluation episodes and no repeated-run or bootstrap intervals propagated into the Nash solver or the GE termination check. Table 1 reports standard errors of 0.4–3.7 for several responses; these are not reflected in the reported game values or in the Nash payoff line of Figure 1. Because the gaps supporting 'no significant improvement' are often small (e.g., Red VF-PBRS vs vanilla means 24.72/24.85 vs 24.22), the equilibrium mixture and the GE ≤ 0 conclusion could be artifacts of estimation noise. Please add repeated-seed MRO runs or, at minimum, bootstrap confidence intervals for the payoff matrix and game values, and qualify the convergence and robustness claims accordingly.","section":"Sections 7.1, 7.2 and Appendix C"},{"comment":"The abstract's claim that VF-PBRS oracles 'converge upon significantly stronger ACD policies compared to vanilla approaches' is overgeneralized. The significant improvement is demonstrated only for Blue GPPO responses in CC2; for Red in CC2 the differences (24.72/24.85 vs 24.22) are explicitly not significant, and in CC4 the paper reports no significant VF-PBRS advantage. Please restate the contribution as an empirical finding for the specific oracle and environment in which it is supported.","section":"Abstract and Section 7.1 (Table 1, Figure 2)"},{"comment":"The existence of an ABR policy in the response set Ri(µj) is assumed, not guaranteed. If none of the response oracles yields a policy that improves on the current mixture, GE ≤ 0 certifies only a resource-bounded equilibrium relative to the specific response functions, not that no approximate best response exists in the broader policy space. The paper should state this qualification explicitly when claiming the extension preserves ADO's theoretical properties, and Section 7.1's conclusion that Red 'struggles to find new successful TTPs' should be framed as a property of the tested response oracles.","section":"Section 4.2, Eq. (9), and Algorithm 2"}],"minor_comments":[{"comment":"The paper switches between γ_φ and γ when describing the potential discount factor; please use one symbol consistently, preferably γ_φ for the potential discount in Eq. (5).","section":"Appendix C"},{"comment":"The Nash payoff lines would be substantially more informative with pointwise confidence bands, especially where the text refers to 'significant' improvements or their absence.","section":"Figures 1 and 14"},{"comment":"The discussion of strictly and weakly dominated policies treats the estimated payoff matrix as exact; a sentence acknowledging that dominance relations inherit sampling noise from the 100-episode evaluations would make the pruning discussion more cautious.","section":"Section 8"},{"comment":"There are minor typographical errors, including 'V on Neumann' in the bibliography and 'Fale' in Figure 17; these should be corrected.","section":"References and Figure 17"},{"comment":"The notation π∗_Blue is used before being defined in the algorithm; please state explicitly that it denotes the best policy within R_Blue(µ_Red) according to Eq. (9).","section":"Algorithm 2, line 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable fit for an empirical game-theory or autonomous cyber-defence venue. The MRO extension is incremental but practically useful, and the VF-PBRS result for Blue GPPO in CC2 is a solid empirical contribution. However, the claimed theoretical advantage over ADO is currently unproven, and the single-run empirical basis for the main robustness conclusions needs strengthening. The revision should focus on the Theorem 5 proof and on propagating evaluation uncertainty into the reported game values; with those changes the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing you need to know: this is a worthwhile framework paper with one clear empirical overreach. The MRO extension and the VF-PBRS idea are real contributions; the central robustness claims, on the other hand, run ahead of the data.\n\nWhat is new is the application of empirical game theory with double oracles to CybORG CAGE Challenges 2 and 4—that combination is not in the prior literature. MRO is a natural extension of ADO: let each side return multiple response policies from several algorithms, then pick the best. The claim that convergence guarantees survive is basically right, because the best response in a superset is still a best response. VF-PBRS is a neat twist, using value functions from previous ABR iterations as potential functions. The theory is sound; Ng et al.'s PBRS result applies directly, and the Z-score normalization for ensembling is sensible. And the one carefully controlled result—Blue GPPO with VF-PBRS beating vanilla in CC2, with ten runs and p-values—is solid.\n\nThe soft spots are mostly in how the results are reported. Theorem 5's proof is flawed: it assumes ADO's mixture is a subset of MRO's policy set, which is not guaranteed, so the stronger-mixture claim is not established. That is a fixable hole in the theory section. The bigger issue is empirical. The payoff matrix comes from 100 evaluation episodes per policy pair, with standard errors around 0.5–3.7, and the whole CC2 and CC4 game values come from a single MRO run per environment. The termination criterion GE ≤ 0 treats those noisy estimates as exact. The claim that Red 'struggles to find new successful TTPs' is therefore supported by one stochastic trajectory. The abstract's claim that VF-PBRS 'converge upon significantly stronger ACD policies' overgeneralizes: the significant result is for Blue GPPO in CC2; Red's VF-PBRS is not significant and CC4 shows no significant differences. Also, no code or data is shipped, which limits reproducibility.\n\nWho should read it: anyone working on adversarial evaluation of cyber-defence agents, and people in empirical game theory looking at PSRO variants. They'll find the framework useful and the problem well-posed. It deserves a serious referee. I'd send it to review, with the expectation of heavy revision: fix Theorem 5, add repeated-run or bootstrap error bars on all game values, soften the abstract, and release code and data.","headline":"A worthwhile framework paper for adversarial evaluation of ACD agents, but the robustness claims run ahead of the evidence—single MRO runs and 100-episode payoffs are doing too much work.","tokens_in":31718,"tokens_out":2871,"would_cite":true,"duration_ms":29471,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a multi-oracle extension of the double-oracle algorithm, accelerated by value-function potential-based reward shaping, yields cyber-defence agents that hold their ground as adaptive attackers keep learning.","keywords":["autonomous cyber defence","empirical game theory","double oracle algorithm","multiple response oracles","potential-based reward shaping","deep reinforcement learning","Nash equilibrium","CAGE Challenge"],"falsifier":"Run the MRO loop repeatedly from different random seeds and fresh initialisations in CAGE Challenge 2, and then train a fresh attacker with a larger budget against the final Blue mixture; if the final game value varies across seeds by more than the reported standard errors, or if that fresh attacker improves on the Nash payoff by more than evaluation noise, the paper's robustness claim would be contradicted.","tokens_in":30644,"feed_emoji":"🛡️","tokens_out":10916,"duration_ms":96613,"temperature":0.7,"pith_summary":"The paper is trying to establish that adversarial learning via an empirical game-theoretic loop is the right organising principle for training and assuring autonomous cyber-defence agents. To make that practical, it introduces a multiple-response-oracles (MRO) extension of the double-oracle algorithm, where each side can draw approximate best responses from several independent learning approaches rather than one, and proves the convergence guarantees are unchanged. It also proposes value-function potential-based reward shaping (VF-PBRS), which reuses value functions from earlier response iterations as a shaping potential, making each response cheaper to compute and, for the defender, significantly stronger than vanilla training. In two standard cyber-defence benchmark environments (CAGE Challenges 2 and 4), defenders produced by the loop keep their ground as learning attackers try new tactics, and mixture play substantially reduces worst-case exposure compared to deploying any single open-source defender.","feed_headline":"Game-theoretic training shields cyber-defence from adaptive attackers","feed_subtitle":"New multi-oracle loop plus reward shaping produces defenders that hold their ground as attackers keep learning.","key_machinery":"The load-bearing object is the empirical matrix game, a finite two-player zero-sum normal-form game in which each action is a policy trained for one side, and each payoff entry is the mean episodic reward of one policy pair estimated from 100 evaluation episodes. The MRO algorithm replaces the single approximate-best-response oracle of the standard double-oracle loop with a response function that returns a set of candidate policies from several learning approaches, plus a selection step that keeps the best one; because the selected policy is still an approximate best response, the proof that exploitability $\\le 0$ implies a resource-bounded Nash equilibrium goes through unchanged. VF-PBRS shapes rewards with $F(s,s') = \\gamma\\Phi(s') - \\Phi(s)$, where the potential $\\Phi$ is the mixture-weighted, Z-score-normalised sum of value functions from earlier iterations. Since this is a potential-based shaping function, the optimal policy is invariant, so oracles using VF-PBRS still return approximate best responses. Pre-trained model sampling with $\\epsilon$-greedy decay balances reuse of past policies against exploration of new strategies.","core_discovery":"The central claim is that the approximate double-oracle procedure, extended to multiple response oracles and accelerated by VF-PBRS and pre-trained model sampling, converges to a resource-bounded Nash equilibrium of the empirical game. The argument is that potential-based reward shaping preserves the optimal policy, so a VF-PBRS oracle still returns an approximate best response, and that the best policy within a response set computed by MRO is itself an approximate best response. In the experiments, the value of the CC2 game stabilises around $-29.63$ from Blue's perspective after roughly 23 iterations, and a Red response trained from scratch initially jumps to $93.05$ but is neutralised to $29.75$ once Blue's mixture adjusts. The CC4 game value changes little after the first iteration, settling near $-411.91$, with the estimated value of the game essentially constant. The paper concludes that defenders obtained this way are robust to learning attackers, which struggle to find new successful tactics, techniques and procedures.","pith_inferences":["Inference: the MRO loop is a general evaluation harness for any domain where several candidate training recipes compete against an adaptive adversary; red-teaming of language models or fraud-detection systems could reuse the same structure, provided per-pair payoffs can be estimated reliably.","Inference: VF-PBRS helping the defender much more than the attacker suggests the shaping signal matters most when the reward is sparse and delayed, as in long attack chains; a targeted comparison of short versus long attack sequences would test this directly.","Inference: the reported game values come from a single MRO run per environment, so the variance of the equilibrium itself is not measured; re-running the loop across random seeds would show whether the robustness result is a property of the method or of one training trajectory."],"forward_implications":["A defender mixture produced by the CC2 MRO run held the Nash payoff to about $-29.63$; when a freshly initialised Red oracle found a response that briefly raised the payoff to $93.05$, the Blue mixture adjusted and pushed it back to $29.75$, and no later attacker response significantly moved the game value.","Under pre-trained initialisation in CC2, VF-PBRS with value-function ensembling significantly outperformed vanilla training for Blue responses (mean $-29.8 \\pm 1.2$ versus $-38.3 \\pm 2.7$), while for Red the shaping benefit was small and not statistically significant.","Because the selected response in MRO is still an approximate best response, the standard double-oracle termination condition applies: if the combined exploitability is at most zero, the algorithm has found a resource-bounded Nash equilibrium.","The final CC2 Blue mixture consisted of graph-based PPO policies, and a single Red policy that scored $-117.82$ against the original defender parameterisation scored only $-21.96$ against the mixture, illustrating that mixture play reduces worst-case exposure."],"supporting_citations":[{"why":"Supplies the potential-based reward shaping theorem that VF-PBRS relies on to preserve optimal policies.","marker":"(Ng et al., 1999)"},{"why":"Defines the double oracle algorithm whose convergence guarantees MRO extends and preserves.","marker":"(McMahan et al., 2003)"},{"why":"Provides the approximate double oracle formulation and resource-bounded Nash equilibrium used for ADO/MRO termination.","marker":"(Oliehoek et al., 2018)"},{"why":"Introduces policy-space response oracles and the exploitability metric used to measure deviations.","marker":"(Lanctot et al., 2017)"},{"why":"Shows the pre-trained model initialisation strategy that MRO oracles use to shorten response iterations.","marker":"(Li et al., 2023)"},{"why":"Introduces the gym environment that underlies the CAGE Challenge evaluation environments.","marker":"(Standen et al., 2021)"},{"why":"Defines CAGE Challenge 2, including the network scenario and rules-based attackers used as the CC2 environment.","marker":"(TTCP CAGE Working Group, 2022)"},{"why":"Defines CAGE Challenge 4, the multi-agent defence scenario used for the second evaluation.","marker":"(TTCP CAGE Working Group, 2023)"},{"why":"Supplies the open-source graph-based PPO defender used as a Blue oracle in CC2.","marker":"(King & Bowman, 2024)"},{"why":"Supplies the multi-agent graph-based PPO defender used as a Blue oracle in CC4.","marker":"(King, 2024)"}],"fun_headline_variants":["Double-oracle loop hardens cyber-defence against adaptive attackers","Multi-oracle game theory yields resilient autonomous defenders","Reward-shaped agents reach stable cyber-defence Nash equilibrium","Game-theoretic framework makes cyber-defence robust to learning attackers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The single load-bearing premise is that the empirical payoff matrix, estimated from 100 evaluation episodes per policy pair, is accurate enough for the Nash solver to identify a meaningful equilibrium, and that the one MRO run per environment is representative of the stochastic training process.","fun_headline_variants_meta":{"raw":{"variants":["Double-oracle loop hardens cyber-defence against adaptive attackers","Multi-oracle game theory yields resilient autonomous defenders","Reward-shaped agents reach stable cyber-defence Nash equilibrium","Game-theoretic framework makes cyber-defence robust to learning attackers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1379,"prompt_tokens":915,"completion_tokens":464,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":397}},"tokens_in":531,"tokens_out":464,"duration_ms":5420,"temperature":1.0,"reasoning_tokens":397,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T20:56:43.256603+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the MRO loop repeatedly from different random seeds and fresh initialisations in CAGE Challenge 2, and then train a fresh attacker with a larger budget against the final Blue mixture; if the final game value varies across seeds by more than the reported standard errors, or if that fresh attacker improves on the Nash payoff by more than evaluation noise, the paper's robustness claim would be contradicted.","supporting_citations":[{"cited_title":"B., Gordon, G","cited_arxiv_id":null,"evidence_quote":"Defines the double oracle algorithm whose convergence guarantees MRO extends and preserves."},{"cited_title":"A., Savani, R., Gallego, J., Pol, E","cited_arxiv_id":null,"evidence_quote":"Provides the approximate double oracle formulation and resource-bounded Nash equilibrium used for ADO/MRO termination."},{"cited_title":"A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning","cited_arxiv_id":null,"evidence_quote":"Introduces policy-space response oracles and the exploitability metric used to measure deviations."},{"cited_title":"J., Kim, J., and Marriott, D","cited_arxiv_id":null,"evidence_quote":"Introduces the gym environment that underlies the CAGE Challenge evaluation environments."},{"cited_title":"TTCP CAGE Challenge 2","cited_arxiv_id":null,"evidence_quote":"Defines CAGE Challenge 2, including the network scenario and rules-based attackers used as the CC2 environment."},{"cited_title":"TTCP CAGE Challenge 4","cited_arxiv_id":null,"evidence_quote":"Defines CAGE Challenge 4, the multi-agent defence scenario used for the second evaluation."}],"review_version":1}