{"id":"9e1818d9-bbe4-41cb-b38d-b3202216cdc3","arxiv_id":"2505.22531","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Diverse, dynamically ordered training tasks make network-defense RL agents generalize to unseen attacks better than single-task training.","lead":"Training RL agents to defend networks by learning across many attack scenarios, instead of one fixed scenario, produces agents that perform better on attacks they never trained on. The authors report this 'open-ended' approach also makes defenders more robust when adversaries change their behavior.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline outperformance claim in §4.2 rests on single-run comparisons without seed variance or confidence intervals; the OEL generalization result is not yet statistically supported.","rationale":"The reader's weakest assumption was FARLAND's fidelity to real enterprise networks. That is a legitimate external-validity concern, but it cannot be settled from the manuscript alone and is not the first thing that must be true for the central claim to hold. Even inside the simulator, the claim 'dynamic task selection outperforms single-task training' requires a statistically valid comparison. The paper's main evidence, Figures 5–8, shows point estimates without variance across training seeds, and the only described averaging is for one control baseline. Without repeated-seed evaluation, the reported advantage could be within-run noise, especially given stochastic PPO training and stochastic red/gray policies. This is an internal, addressable weakness, not a dispute about community consensus. I also credit the paper's concrete contributions: the PDDL-based task representation, action masking, and curriculum design are substantive and clearly described. However, the lack of released code and the post hoc selection of goal-metric pairs in §4.3 further prevent independent verification. My concern reinforces the reader's CONDITIONAL verdict rather than changing it: the evidence is plausible but incomplete until the comparison is shown to be robust across seeds and the artifacts are available.","tokens_in":22684,"tokens_out":6934,"duration_ms":84082,"concrete_test":"Request release of the training code and seed configuration, then rerun the Figure 5/8 comparison with at least 5 independent seeds per training condition (fixed DoS baseline and dynamic task selection), identical 5M-step budgets, and evaluate each resulting policy on the 200-host DoS testing task for 100 episodes per seed. Report per-seed means and 95% confidence intervals for episode reward and compromise rate; if the dynamic policy's CI overlaps the fixed policy's CI, the Section 4.2 outperformance claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—dynamic task selection yields more robust, generalizable defenders than single-task training—rests on Figures 5–8. Section 4.1 compares fixed vs. dynamic task selection after 5M training steps, and Section 4.2 reports that the dynamic policy outperforms a policy trained specifically to mitigate DoS, despite never seeing a DoS attack. The load-bearing problem is that these headline comparisons appear to come from single training runs per condition. The paper reports no error bars, no multiple training seeds, and no confidence intervals for the pivotal comparisons. The only place multiple seeds are mentioned is for the 'smooth changes' baseline in §4.1, which is averaged; the dynamic and fixed policies are not. Since PPO is stochastic and the red/gray agent policies are probabilistic, run-to-run variance in final policy quality can easily exceed the reported differences. This is compounded by two manuscript-internal limitations: §4.3 selects goal-metric pairs post hoc ('we selected 6 that the agent was able to reasonably master'), and the source code is repeatedly stated to be 'forthcoming,' so the numbers cannot be independently regenerated. Even granting full FARLAND realism, the empirical claim is not yet established from the reported evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an open-ended-learning-inspired training framework for network defense in the FARLAND simulator. It defines a universe of tasks (N,G) with consistent action/observation interfaces, PDDL-specified goals and metrics, and dynamic difficulty-based task selection. The central empirical claim, developed in Section 4, is that dynamic task selection outperforms fixed-task training and single-task DoS training, reaches higher performance in fewer steps, and generalizes to an unseen DoS attack and to deviations in red-agent behavior. The paper also compares task-selection strategies and goal-metric representations, and reports PPO hyperparameters.","tokens_in":22916,"tokens_out":6275,"duration_ms":72647,"significance":"If the empirical claims held, this would be a useful contribution to cyber-RL: it gives a concrete way to define broad, consistent task universes and shows that curricula with difficulty and diversity can help. The experimental design is thoughtful: comparing dynamic vs fixed, difficulty vs random vs smooth, and robustness to deviations are the right comparisons. The paper is candid about limitations of task-selection research. However, the central empirical support is currently weakened by missing seed variance and error bars on the pivotal figures, the private simulator with no external validation, and the unavailability of code; these are fixable but currently prevent full assessment.","major_comments":[{"comment":"The central comparison between dynamic and fixed task selection, and the generalization comparison against a DoS-trained policy, are reported without error bars, confidence intervals, or multiple training seeds. The only place multiple runs are mentioned is the 'smooth changes' baseline in Section 4.1 ('we train the policy several times and report the average'); the dynamic and fixed policies appear to be single runs. Because PPO is stochastic and the red and gray agent policies are probabilistic, run-to-run variance can easily exceed the reported differences. The headline claims ('dynamic task selection outperforms fixed...', 'dynamic ... outperforms the policy trained to mitigate DoS') therefore are not statistically supported as reported. Please provide at least 3–5 seeds for each condition and report means with variance or confidence intervals, or effect sizes.","section":"§4.1/§4.2, Figs. 5–8"},{"comment":"The manuscript states twice that source code is 'forthcoming' (Section 4.5 and Acknowledgments), and all experiments run inside the private FARLAND simulator (reference [29]) with no external validation. The dynamic task selection procedure in Section 3.2 also leaves unspecified the promotion/demotion thresholds, difficulty-level sampling ranges, and the exact selection criterion for goal-metric pairs. Together these make the central empirical claim impossible to reproduce or independently assess from the manuscript. At minimum, the paper should specify all curriculum parameters, release code or a public benchmark interface, and provide some validation of FARLAND against real network traffic or an existing public cyber-defense gym (e.g., CybORG, CyberBattleSim, or NASimEmu), or explicitly restrict the claims to the simulator.","section":"§4 overall; §4.5; Acknowledgments"},{"comment":"The six goal-metric pairs used to compare representations were selected post hoc ('we selected 6 that the agent was able to reasonably master'). This selection on the outcome variable makes Figures 10–11 uninterpretable as a comparison of goal representations, because the chosen subset is conditioned on agent performance. If this comparison is meant to support the framework's generality, all 11 evaluated pairs should be reported, or a pre-registered selection criterion should be stated; the text should not conclude that 'the one-hot representation results in higher performance' from this biased subset.","section":"§4.3"}],"minor_comments":[{"comment":"The evaluation section says 'First, Section 4.1 evaluates... Then, Section 4.1 compares different strategies'; the duplication of Section 4.1 is confusing and one of the two references should likely point to a later subsection or to a different part of Section 4.","section":"§4, opening paragraph"},{"comment":"The sparse reward formula is ambiguous in the rendering '1 2ex2 + 0.5'; please clarify whether the exponent is x, x/2, or x^2, since the stated asymptotic behavior (reward converging to 0.5 for satisfied goals as penalties grow) is only consistent with an exponent that decays as x becomes more negative.","section":"§3.1.3"},{"comment":"The appendix contains placeholder citations '[?]' for the claims that optimal behavior may be achieved via multiple rewards and that poorly designed dense rewards can be worse than sparse rewards; these need concrete references.","section":"Appendix 6.1"},{"comment":"The glossary defines 'declared-victory', but the PDDL listings for goals 42 and 43 use 'red-declared-victory'; align these terms so the appendix is internally consistent.","section":"Appendix 6.2, PDDL terms"},{"comment":"The sentence 'the agent training via dynamic task selection is ultimately trained against a more sophisticated red TTP' should read 'the agent trained via dynamic task selection...'.","section":"§4.2"},{"comment":"The column header 'hyperparamters' is a typo for 'hyperparameters'.","section":"Table 1"},{"comment":"Figure 5 is referenced in both Section 4.1 and Section 4.2 for different claims, but the figure does not appear to have labeled panels; add clear panel labels so the reader can tell which curve supports which comparison.","section":"Figs. 5–8"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a cyber-RL or general RL venue, but its load-bearing evidence is single-run and its simulator is private; I would not accept without seeds and code. Note also the heavy reliance on self-citation [29] for FARLAND and the repeated 'forthcoming' code statements; these should be checked at revision. My concerns are statistical support and reproducibility, not the OEL paradigm itself."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read. What's actually new: applying open-ended-learning ideas to network defense with a concrete PDDL-based goal/metric encoding and PDDL-driven action masking, plus a dynamic difficulty curriculum. That is a useful recipe for cyber-defense gyms, and the paper shows that a dynamic-curriculum agent outperforms a fixed-task agent on a held-out DoS task it never saw. That's a genuine result worth taking seriously.\n\nThe paper does several things well. The task universe is described concretely with the parameters that vary, and the PDDL goal-metric pairs are listed in full in the appendix. The comparisons include fixed vs dynamic, random vs difficulty vs smooth, and robustness to red-agent deviation. The observation and action normalization across tasks is thoughtful. The writing is honest about what is not solved.\n\nThe soft spots are real and they coincide with the stress-test. The main learning curves in Figures 5-8 appear to come from single training runs per condition; the only averaged baseline is the 'smooth changes' one. With PPO and probabilistic red/gray agents, run-to-run variance can easily be as large as the reported gaps. There are no error bars or confidence intervals. Section 4.3 also says the 6 goal-metric pairs were selected post hoc, which is fine for a proof of concept but weakens the representation comparison. The whole evaluation runs inside FARLAND, the authors' own simulator, with no validation against public benchmarks and code 'forthcoming.' None of these are fatal individually, but together they mean the central claim—that dynamic task selection gives more generalizable defenders—is plausible, not yet established.\n\nThe paper is still worth engaging. The target audience is researchers building cyber-defense benchmarks; they should read it before designing their next gym. It deserves a serious peer review, with the requirement that the authors release code, seeds, and error bars. If those hold up, the result is an important one.","headline":"Real contribution in the PDDL task encoding and dynamic curriculum, but the headline generalization result is not yet statistically supported because the main comparisons are single runs without error bars.","tokens_in":23478,"tokens_out":1736,"would_cite":true,"duration_ms":20154,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a single reinforcement-learning agent on a broad universe of network-defense tasks produces defenders that generalize to attacks never seen during training, outperforming specialists trained on the attack itself.","keywords":["open-ended learning","reinforcement learning","cybersecurity","network defense","curriculum learning","task universe","action representations","generalization"],"falsifier":"Run the identical dynamic-task-selection curriculum in a second, independent network-defense environment, for example an emulation built from real hosts, real services, and live traffic, or an unrelated public benchmark, and give the resulting policy the same held-out DoS test. If the broadly trained agent no longer matches or beats a DoS-trained specialist on that independent ground, the reported generalization advantage is a property of the simulator's structure rather than of open-ended training.","tokens_in":22481,"feed_emoji":"🛡️","tokens_out":12313,"duration_ms":120260,"temperature":0.7,"pith_summary":"This paper tries to establish that open-ended learning transfers from robotics and games to network defense: training one reinforcement-learning agent on a diverse, procedurally generated universe of defense tasks, rather than on a single attack scenario, yields defenders that learn faster and handle conditions they never met. The key evidence is a held-out generalization result: a policy trained with dynamic task selection, which never faced a denial-of-service attack, still outperformed a policy trained specifically to mitigate DoS when both were tested on a 200-host network under that attack. The paper also reports that a difficulty-driven curriculum reaches higher performance with fewer training steps than fixed-task training, and that task diversity protects performance when the adversary masks its behavior. If the claim holds, it gives the cyber-defense gym community a concrete template: keep action spaces, observation spaces, and reward interfaces consistent across a broad task universe, and let the curriculum expand the task distribution as the agent improves. The authors further recommend that benchmark builders adopt broad task universes rather than a few 'representative' scenarios.","feed_headline":"Training on diverse defense tasks beats single-attack training","feed_subtitle":"An agent that never saw a DoS attack during training still outperforms a policy trained specifically to stop one.","key_machinery":"The load-bearing object is the task pair $(N, G)$: $N$ is a network's dynamics (initial configuration, gray-agent user policies, red-agent attacker policies, and a function $f$ that converts abstract defender actions into executable parametric actions), and $G$ is a boolean goal plus a numeric metric written in PDDL, turned into a bounded sparse reward $r(x)=\\tfrac{1}{2}e^{x/2}+0.5$ when the goal is satisfied and $r(x)=\\tfrac{1}{2}e^{x/2}$ when it is not. Two design choices make the universe learnable as a single curriculum: fixed action and observation spaces across all tasks, with a small set of non-parametric action representations standing in for thousands of parametric actions; and PDDL goal-metric pairs (43 of them) that keep the reward semantics consistent as goals change. The third piece is the outer training loop, dynamic task selection, which raises cardinality parameters such as host counts, subnet counts, and red-and-gray behavior variability as the agent's mean score crosses a threshold and lowers them when the score falls, producing a difficulty-driven curriculum.","core_discovery":"The paper's central claim is that a single RL agent trained by dynamic task selection over a universe of network-defense tasks, each specified as a pair $(N, G)$ of network dynamics and a PDDL-encoded security-and-QoS goal, learns a policy that generalizes to defense tasks it never trained on. In the main experiment, an agent whose curriculum covered exfiltration and ransomware attacks, plus variations in network scale, user behavior, and red-agent behavior, but never a DoS attack, outperformed at DoS mitigation a policy trained exclusively on DoS. The authors read this as evidence that open-ended learning principles carry over to cyber defense: broad capability built across related tasks beats narrow competency tuned to a single attack. They further report that the difficulty-driven curriculum reaches higher reward with fewer training steps than fixed-task training, and that training against red agents that interleave gray-like behavior produces policies that do not fall apart when the adversary deviates from its basic tactics. The paper frames these results as a demonstration that the task universe, not the single environment, should be the unit of design for RL-based network defense.","pith_inferences":["The recipe should transfer to other blue action sets and other attack categories: nothing in the argument depends on the five defender actions tested, so a direct testable extension is holding out an entire tactic family, exactly as the paper holds out DoS, and checking that the advantage persists.","The paper's evaluation suggests a concrete reporting standard for cyber-defense RL: report performance on a held-out task from the same universe, not just on held-out seeds of the training task; the authors argue for this but do not state it as a formal evaluation rule.","Because all comparisons run inside one simulator, the strongest open question is external validity: running the identical curriculum in an emulation-based environment with real hosts and traffic would show whether the generalization benefit survives the simulation-to-reality gap, a test the paper does not provide.","The appendix's curriculum findings imply that task-selection heuristics are as consequential as the RL algorithm itself: front-loading the hardest goal-metric pairs, or exposing the agent only to an inactive adversary, derails training, so a principled theory of when to demote tasks would directly improve defender training."],"forward_implications":["A defender whose curriculum includes exfiltration and ransomware tasks, but never DoS, will beat a DoS-trained specialist at DoS mitigation when both face the same held-out network under attack.","Dynamic, difficulty-driven task selection reaches higher performance with fewer training steps than fixed-task training, and it masters harder tasks than uniform-random or unguided smooth curricula under the same computing budget.","Replacing the full parametric action space with a small set of action representations prevents learning stagnation, so curriculum training scales to networks whose raw action spaces would be thousands of actions.","Training against red agents that interleave gray-like (user-mimicking) behavior yields policies that maintain performance when the adversary deviates from its basic behavior.","Cyber-defense gym and benchmark builders should define broad task universes with consistent goal, reward, and action interfaces, because evaluating on small sets of representative tasks misleads assessments of defender robustness."],"supporting_citations":[{"why":"FARLAND, the simulator whose network, gray-agent, and red-agent dynamics define every task in the universe and host every experiment in the paper.","marker":"[29]"},{"why":"The open-ended learning training scheme (worlds and agent goals with smoothness, vastness, and diversity) that the paper adapts to network defense.","marker":"[41]"},{"why":"Automated domain randomization and difficulty escalation, cited as the basis for the dynamic task-selection curriculum.","marker":"[32]"},{"why":"Action representations, the mechanism that keeps the action space fixed and small across the whole task universe.","marker":"[11]"},{"why":"The conceptual account of open-ended learning that motivates training for broad capability over narrow competency.","marker":"[34]"}],"fun_headline_variants":["Broad task training beats focused DoS defense","Never trained on DoS, but beats DoS specialist","Open-ended learning sharpens network defense","Diverse tasks, robust defenders: OEL for cyber","Task universe beats single-attack training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the FARLAND simulator, with its gray-agent user model, red-agent tactics, QoS event model, and host-and-subnet dynamics, faithfully represents realistic enterprise networks, because every training run, comparison, and generalization test takes place inside it and nothing validates the simulator against real traffic or independent benchmarks.","fun_headline_variants_meta":{"raw":{"variants":["Broad task training beats focused DoS defense","Never trained on DoS, but beats DoS specialist","Open-ended learning sharpens network defense","Diverse tasks, robust defenders: OEL for cyber","Task universe beats single-attack training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000163,"raw_usage":{"total_tokens":1258,"prompt_tokens":976,"completion_tokens":282,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":212}},"tokens_in":592,"tokens_out":282,"duration_ms":3539,"temperature":1.0,"reasoning_tokens":212,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:04:33.720576+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical dynamic-task-selection curriculum in a second, independent network-defense environment, for example an emulation built from real hosts, real services, and live traffic, or an unrelated public benchmark, and give the resulting policy the same held-out DoS test. If the broadly trained agent no longer matches or beats a DoS-trained specialist on that independent ground, the reported generalization advantage is a property of the simulator's structure rather than of open-ended training.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The open-ended learning training scheme (worlds and agent goals with smoothness, vastness, and diversity) that the paper adapts to network defense."},{"cited_title":"2019.Solving Rubik’s Cube with a Robot Hand","cited_arxiv_id":null,"evidence_quote":"Automated domain randomization and difficulty escalation, cited as the basis for the dynamic task-selection curriculum."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Action representations, the mechanism that keeps the action space fixed and small across the whole task universe."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The conceptual account of open-ended learning that motivates training for broad capability over narrow competency."}],"review_version":1}