Pith. sign in

REVIEW 2 major objections 6 minor 1 cited by

Learning Robust Penetration Testing Policies under Partial Observability: A systematic evaluation

T0 review · 2 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that a simple element-wise maximum over past observations outperforms LSTM and transformer architectures in partially observable penetration testing, converging up to four times faster.

desk verdict A careful empirical comparison that makes a plausible case for cheap history augmentation over learned memory in a useful new benchmark, though the headline generalization is confined to monotone environments and the code is missing. read the letter →

arxiv 2509.20008 v2 pith:OMHJLO54 submitted 2025-09-24 cs.LG cs.CR

Raphael Simon , Pieter Libin , Wim Mees This is my paper
classification cs.LGcs.CR
keywords partialobservabilityreinforcementlearningpenetrationtestingobservationaugmentationPPOLSTMtransformerpolicygeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that for automated penetration testing, partial observability does not require expensive learned memory. It introduces StochNASim, a stochastic network simulator that regenerates network topologies and sizes every episode, and systematically compares PPO variants with frame-stacking, LSTM, transformer, and an augmented-observation wrapper. The central finding is that the augmented observation method—which keeps an element-wise maximum of all past observations—learns the best and most human-like policies, converges two to four times faster, and generalizes across network sizes. The authors also show that policies trained on fixed networks transfer poorly, while training on regenerated networks yields robust policies, and that qualitatively different behaviors can produce similar scores.

What carries the argument

The central object is the augmented-observation wrapper, which maintains an aggregation matrix O_aug_t = max(O_aug_{t-1}, O_t) taking the element-wise maximum of the current observation and all previous observations. This matrix is stacked below the latest observation and fed to a standard feedforward PPO policy. The mechanism works because the environment's task-relevant state is monotone within an episode: once a host is discovered, compromised, or accessed, that information never expires, so the running maximum is a sufficient statistic for the hidden state. It allows a stateless policy to behave as if it had memory, without recurrent or attention machinery.

What would settle it

Run PPO-AO against LSTM/Transformer baselines in a version of StochNASim where, mid-episode, a host can be rebooted (losing access) or a firewall rule can block a previously reachable host. If PPO-AO's reward advantage shrinks or reverses, the paper's central claim is contingent on monotonic network dynamics, as the authors themselves caution.

Watch

Extended reading notes

Core claim

The paper's central claim is that simple observation augmentation significantly outperforms complex architectures (LSTMs, transformers), challenging conventional assumptions about memory mechanisms in this domain. Concretely, PPO with augmented observations (PPO-AO) converges twice as fast as the transformer variant and four times faster than the remaining methods, while achieving the largest cumulative reward across all tested network sizes. Manual inspection of the learned action sequences shows PPO-AO systematically scans, retains discovered information, and exploits efficiently, whereas the transformer learns a brute-force strategy and the LSTM fails to build a useful history representat

Load-bearing premise

The load-bearing premise is that within an episode the network state only ever becomes more known: once a host is discovered, compromised, or accessed, that fact never changes or becomes obsolete. If hosts can go offline, get patched, or have access revoked mid-episode, the element-wise max history will keep stale info and the simple augmentation loses its edge.

Editorial extensions

If this is right

  • For penetration-testing tasks where the hidden state is monotonically accumulating information, learned memory architectures are unnecessary; a stateless policy over an aggregated observation is sufficient.
  • Training on freshly generated network permutations per episode—rather than on a fixed set of scenarios—produces policies that generalize to unseen network configurations.
  • Algorithms that reach similar mean rewards can learn qualitatively different behaviors (e.g., systematic scanning vs. brute-force exploit attempts), so evaluating policies solely on return can be misleading.
  • The action-type distribution and per-step sequences provide a practical way to audit what an RL pentesting agent actually does.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same element-wise max trick might transfer to other information-gathering tasks where facts accumulate and never expire (e.g., vulnerability discovery, puzzle solving), but it would need an explicit forget mechanism for non-monotone environments.
  • The authors' Section 6 caveat implies that in live adversarial settings—where defenders actively patch or cut paths—the advantage of PPO-AO over LSTM/Transformer policies may reverse, making learned memory still relevant for real-world deployment.
  • The paper's result suggests a design heuristic: before adding recurrent or attention-based policy networks, try a task-specific state summarization; the summary may carry enough information and train faster.
  • The observed 'brute-force' behavior of TrXL suggests that powerful sequence models can overfit to the mechanics of the action set, ignoring the information-gathering structure; an interesting extension would be to add an explicit information-gathering reward to those baselines and see if they then match PPO-AO.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces StochNASim, a stochastic extension of the NASim penetration-testing simulator. StochNASim regenerates network topologies, host properties, and action spaces at every episode and supports networks of 5–8 hosts, in contrast to the fixed configurations of NASim. Using StochNASim, the authors compare five PPO-based agents for partial observability: vanilla PPO, frame-stacking (PPO-FS), observation augmentation with an element-wise max over history (PPO-AO), PPO with an LSTM, and PPO with a Transformer-XL memory. Hyperparameters are tuned per algorithm with 250 Optuna trials and validated on five control seeds. The main reported result is that PPO-AO converges fastest and attains the highest cumulative reward, learning an efficient 'scan then exploit' policy, while LSTM and TrXL variants underperform; PPO-TrXL learns a brute-force no-scan policy. The paper also reports a transfer experiment showing that policies trained on fixed NASim scenarios generalize worse to unseen networks than policies trained on StochNASim, and a qualitative action-sequence analysis of the learned policies.

Significance. If the reported results hold, the paper makes a useful contribution to automated penetration testing. The main practical finding—that a hand-crafted element-wise maximum over observation history can outperform recurrent and transformer-based memory in this environment—is interesting and challenges a default preference for expressive sequence models in POMDPs. The paper's methodological strengths are notable for this literature: a per-algorithm hyperparameter search with 250 trials, control-seed validation, evaluation on freshly generated environments, IQM-based aggregation, and manual policy inspection. The StochNASim environment, with per-episode network regeneration and variable sizes, is a concrete step toward studying generalization in this domain. The transfer experiment, although limited to a single algorithm, illustrates the benefit of stochastic training environments.

major comments (2)
  1. [Section 4.2, Eq. (2); Section 6] The PPO-AO aggregation O_aug_t = max(O_aug_{t-1}, O_t) is a sufficient statistic only while all state variables that matter are monotone within an episode: hosts become discovered/compromised/reachable and access levels only increase; nothing is ever reset. Under such dynamics, the element-wise max preserves all relevant information, which explains why PPO-AO outperforms learned memory. The paper itself concedes in Section 6 that in dynamic environments (hosts shutting down, firewall rules updated, paths becoming unavailable) 'hand-crafted observation augmentations might become brittle and infeasible to maintain.' Since the title promises 'robust' policies and Contribution 3 states the superiority claim in general terms, the headline result is currently established only for a monotone-state POMDP. Please either restrict the claims to this setting, or add an experiment in which state can
  2. [Abstract, Conclusion, and Section 5.2 (Fig. 3)] The abstract and conclusion state that PPO-AO converges 'three times faster', but Section 5.2 reports that it 'converges twice as fast as PPO-TrXL, and four times faster than the remaining methods.' These statements are mutually inconsistent unless a precise definition of convergence speed (e.g., environment steps to reach X% of final reward) is given. Please specify the convergence criterion, report the actual values, and make the numbers consistent across the paper. As written, the headline speed-up claim is ambiguous.
minor comments (6)
  1. [Section 3] The sentence 'Li et al. incorporate reward machines (RM), a form of [23]' is incomplete. Please complete the sentence or the reference.
  2. [Section 4.1] The observation size is first given as (m_c+1)×n and later as (m+1)×n; clarify the padding convention.
  3. [Eq. (2)] The three matrices labelled O_aug_1..3 do not by themselves show the stacking of latest observation and aggregated history; add labels or a schematic.
  4. [Section 5.2] Typo: 'such a larg amount' should be 'such a large amount'.
  5. [Section 5.4] Typo: 'one millions total steps' should be 'one million total steps'.
  6. [Tables 1 and 4] Table 1 has 'Succcess Prob.' and Table 4 uses '0,995' instead of '0.995'; unify decimal notation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical comparison, and its central claims rest on experiments rather than on equations or self-citations that reduce to their own inputs.

full rationale

The claimed contributions are empirical: PPO-AO's faster convergence and higher reward are measured against baselines on a stochastic simulator, not derived from a fitted quantity or from the definition of the method. The observation-augmentation rule in Section 4.2, O_aug_t = max(O_aug_{t-1}, O_t), is a design choice, and the paper is explicit that in StochNASim only four host values change and that repeating a successful action has no further effect. This makes the max-history a reasonable sufficient statistic for the environment as defined, but the paper does not equate the method's success with the definition of the augmentation; it tests it empirically. Hyperparameters are optimized per algorithm and then verified on five control seeds with fresh evaluation environments, so the headline result is not a fitted parameter renamed as a prediction. Self-citations (e.g., Libin et al. for pandemic RL background) are not load-bearing for any central claim, and no uniqueness theorem is imported from the authors' prior work. The Section 6 limitation about dynamic environments where host states can change is an acknowledged scope restriction, not a circular step. Therefore no circularity is present.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper is an empirical comparison; the central result depends on environment design constants and per-algorithm tuning, not on a derivation. Key hand-set quantities are the reward weights and action costs, the step limit, and the per-method hyperparameters selected by Optuna on the evaluation metric. The comparison inherits NASim's stationary-agent simplification and the generator's guarantee of exploitable paths. StochNASim is a software benchmark, not an invented physical or theoretical entity; no new particles, forces, mediators, or conserved quantities are introduced.

free parameters (5)
  • Per-algorithm PPO hyperparameters (learning rate, entropy coefficient, clip range, network architecture, etc.) = Optuna-selected over 250 trials per algorithm; tables 2-4 list best values (e.g., PPO-AO LR 3e-4, entropy 1e-2; PPO-FS f
    Fitted to maximize mean undiscounted reward on freshly generated evaluation environments (Section 4.3); guarded by 5 control seeds in Section 5.2.
  • Frame stack count f_n (PPO-FS) = 8 (selected from {4, 8, 16, 32}; Table 2)
    Tuned hyperparameter; sensitivity analysis shows it explains under 5% of variance.
  • Reward shape constants = Host value 5; sensitive host value 100; scan cost 1; exploit/priv-esc cost 3; success probability 0.9
    Hand-chosen in Section 5.1 to create a reward hierarchy; these constants affect the trade-off between scanning and exploiting, hence all reported rewards.
  • Episode step limit = 5000
    Set from random-agent mean episode length multiplied by 10 (Section 5.1); an ad hoc bound that affects which policies count as solving the task.
  • PPO-TrXL rollout configuration = 768 rollout steps, 8 environments (Appendix A)
    Chosen from GPU memory constraints (A100/H100); differs from the other methods' rollout settings, a confound for the TrXL comparison.
assumptions (5)
  • domain assumption The network generator guarantees at least one viable attack path to sensitive hosts (each subnet has a vulnerable host, sensitive hosts are root-exploitable, firewall rules allow a vulnerable service between zones).
    Section 4.1, Network Generation. Ensures every generated episode is solvable; the difficulty distribution and all inter-algorithm comparisons depend on it.
  • domain assumption Task-relevant observable features are monotone within an episode, so the element-wise maximum of observations is a sufficient history representation.
    Section 4.2, PPO-AO (Eq. 2). Load-bearing premise for why PPO-AO works; authors concede in Section 6 that dynamic environments would break it.
  • domain assumption Penetration testing in these networks is adequately modeled with a stationary agent that never moves between hosts, only escalates access and scans remotely.
    Section 4.1, State/Initial State. Inherited from NASim; narrows the claim of modeling real penetration testing.
  • standard math PPO's clipped surrogate objective and GAE advantage estimates constitute a sound optimization target in this POMDP setting.
    Section 4.2, Baseline. Standard result assumed without proof.
  • domain assumption Mean undiscounted cumulative reward over 100 fresh environments is an adequate measure of policy quality for comparison.
    Sections 4.3 and 5.2. The chosen objective is also the optimization target for hyperparameter tuning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Robust Penetration Testing Policies under Partial Observability: A systematic evaluation." pith.science (2026). https://pith.science/paper/OMHJLO54

@misc{pith2026250920008,
  author       = {Pith},
  title        = {Pith review of: Learning Robust Penetration Testing Policies under Partial Observability: A systematic evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OMHJLO54}},
  note         = {Machine review of arXiv:2509.20008}
}
read the original abstract

Penetration testing, the simulation of cyberattacks to identify security vulnerabilities, presents a sequential decision-making problem well-suited for reinforcement learning (RL) automation. Like many applications of RL to real-world problems, partial observability presents a major challenge, as it invalidates the Markov property present in Markov Decision Processes (MDPs). Partially Observable MDPs require history aggregation or belief state estimation to learn successful policies. We investigate stochastic, partially observable penetration testing scenarios over host networks of varying size, aiming to better reflect real-world complexity through more challenging and representative benchmarks. This approach leads to the development of more robust and transferable policies, which are crucial for ensuring reliable performance across diverse and unpredictable real-world environments. Using vanilla Proximal Policy Optimization (PPO) as a baseline, we compare a selection of PPO-based variants designed to mitigate partial observability, including frame-stacking, augmenting observations with historical information, and employing LSTM or TrXL architectures. We conduct a systematic empirical analysis of these algorithms across different host network sizes. We find that this task greatly benefits from history aggregation. Converging up to four times faster than other approaches. Manual inspection of the learned policies by the algorithms reveals clear distinctions and provides insights that go beyond quantitative results.

Figures

Figures reproduced from arXiv: 2509.20008 by the authors.

Figure 1
Figure 1. The agent–environment interaction in reinforcement learning. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Example network topology in StochNASim showing five hosts across [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. All results are aggregated over 5 seeds. (a) Learning curves of selected [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Performance sensitivity analysis across network sizes. Box plots show [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Action type distribution per algorithm, revealing distinct learned [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Representative action sequences on a 5-host network. Colours indicate [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Training and evaluation performance comparison of PPO-AO trained [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Hyperparameter importance analysis using fANOVA framework [16]. [PITH_FULL_IMAGE:figures/full_fig_p027_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecurity

    cs.LG 2026-02 accept novelty 6.0 of 10

    Across 66 DRL-for-cybersecurity papers, the authors identify 11 recurring methodological pitfalls—averaging 5.8 per paper—and demonstrate their impact in four environments.

Reference graph

Works this paper leans on

60 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    Agarwal, M

    R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare. Deep reinforcement learning at the edge of the statistical precipice.Advances in neural information processing systems, 34:29304–29320, 2021

    2021

  2. [2]

    Akiba, S

    T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next- generation hyperparameter optimization framework. InThe 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2623– 2631, 2019

    2019

  3. [3]

    Avalos, F

    R. Avalos, F. Delgrange, A. Nowe, G. Perez, and D. M. Roijers. The wasserstein believer: Learning belief updates for partially observable environments through 19 reliable latent space models. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  4. [4]

    Algorithmsforhyper-parameter optimization.Advances in neural information processing systems, 24, 2011

    J.Bergstra, R.Bardenet, Y.Bengio, andB.Kégl. Algorithmsforhyper-parameter optimization.Advances in neural information processing systems, 24, 2011

    2011

  5. [5]

    Berner, G

    C. Berner, G. Brockman, B. Chan, V. Cheung, P. Dębiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, et al. Dota 2 with large scale deep reinforcement learning.arXiv preprint arXiv:1912.06680, 2019

    arXiv 1912

  6. [6]

    Leveragingproceduralgeneration to benchmark reinforcement learning

    K.Cobbe, C.Hesse, J.Hilton, andJ.Schulman. Leveragingproceduralgeneration to benchmark reinforcement learning. InProceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020

    2020

  7. [7]

    Quantifyinggeneraliza- tion in reinforcement learning

    K.Cobbe, O.Klimov, C.Hesse, T.Kim, andJ.Schulman. Quantifyinggeneraliza- tion in reinforcement learning. InInternational conference on machine learning, pages 1282–1289. PMLR, 2019

    2019

  8. [8]

    M. E. Consens, C. Dufault, M. Wainberg, D. Forster, M. Karimzadeh, H. Goodarzi, F. J. Theis, A. Moses, and B. Wang. Transformers and genome language models.Nature Machine Intelligence, pages 1–17, 2025

    2025

Show all 60 references
  1. [9]

    Degrave, F

    J. Degrave, F. Felici, J. Buchli, M. Neunert, B. Tracey, F. Carpanese, T. Ewalds, R. Hafner, A. Abdolmaleki, D. de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning.Nature, 602(7897):414–419, 2022

    2022

  2. [10]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Un- terthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  3. [11]

    Dulac-Arnold, N

    G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. Challenges of real-world reinforcement learning: definitions, bench- marks and analysis.Machine Learning, 110(9):2419–2468, 2021

    2021

  4. [12]

    Ghosh, J

    D. Ghosh, J. Rahme, A. Kumar, A. Zhang, R. P. Adams, and S. Levine. Why gen- eralization in rl is difficult: Epistemic pomdps and implicit partial observability. Advances in neural information processing systems, 34:25502–25515, 2021

    2021

  5. [13]

    Hausknecht and P

    M. Hausknecht and P. Stone. Deep Recurrent Q-Learning for Partially Observable MDPs, Jan. 2017. arXiv:1507.06527 [cs]

    2017 arXiv

  6. [14]

    Hochreiter and J

    S. Hochreiter and J. Schmidhuber. Long short-term memory.Neural computation, 9(8):1735–1780, 1997

    1997

  7. [15]

    Huang, R

    S. Huang, R. F. J. Dossa, C. Ye, J. Braga, D. Chakraborty, K. Mehta, and J. G. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms.Journal of Machine Learning Research, 23(274):1–18, 2022

    2022

  8. [16]

    Hutter, H

    F. Hutter, H. Hoos, and K. Leyton-Brown. An efficient approach for assessing hyperparameter importance. InProceedings of the 31st International Conference on Machine Learning, volume 32, pages 754–762, Bejing, China, 22–24 Jun 2014. PMLR

    2014

  9. [17]

    R. T. Icarte, T. Q. Klassen, R. Valenzano, and S. A. McIlraith. Reward ma- chines: Exploiting reward function structure in reinforcement learning.Journal of Artificial Intelligence Research, 73:173–208, 2022

    2022

  10. [18]

    Cybersecurity workforce study

    ISC2. Cybersecurity workforce study. 2023

    2023

  11. [19]

    Janisch, T

    J. Janisch, T. Pevný, and V. Lisý. NASimEmu: Network Attack Simulator & Emulator for Training Agents Generalizing to Novel Scenarios, Aug. 2023. arXiv:2305.17246 [cs]

    2023 arXiv

  12. [20]

    L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains.Artificial intelligence, 101(1-2):99–134, 1998. 20

    1998

  13. [21]

    Q. Li, M. Hu, H. Hao, M. Zhang, and Y. Li. INNES: An intelligent network pene- tration testing model based on deep reinforcement learning.Applied Intelligence, 53(22):27110–27127, Nov. 2023

    2023

  14. [22]

    Q. Li, R. Wang, D. Li, F. Shi, M. Zhang, and A. Chattopadhyay. Dynpen: Auto- mated penetration testing in dynamic network scenarios using deep reinforcement learning.IEEE Transactions on Information Forensics and Security, 2024

    2024

  15. [23]

    Y. Li, H. Dai, and J. Yan. Knowledge-informed auto-penetration testing based on reinforcement learning with reward machine. In2024 International Joint Con- ference on Neural Networks (IJCNN), pages 1–9. IEEE, 2024

    2024

  16. [24]

    Z. Li, Q. Zhang, and G. Yang. EPPTA: Efficient partially observable reinforce- ment learning agent for penetration testing applications. n/a:e12818. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/eng2.12818

    doi

  17. [25]

    M. C. Libicki, L. Ablon, and T. Webb.The Defender’s Dilemma: Charting a Course Toward Cybersecurity. Number RR-1024-JNI in Research Report. RAND Corporation, 2015. Accessed: May 21, 2025

    2015

  18. [26]

    P. J. Libin, A. Moonens, T. Verstraeten, F. Perez-Sanjines, N. Hens, P. Lemey, and A. Nowé. Deep reinforcement learning for large-scale epidemic control. In European Conference in Machine Learning 2020, Ghent, Belgium, pages 155–170. Springer, 2021

    2020

  19. [27]

    Macaulay

    T. Macaulay. The danger of critical infrastructure interdependency.Centre for International Governance Innovation, 2019. Accessed: May 21, 2025

    2019

  20. [28]

    V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning.Nature, 518(7540):529–533, 2015

    2015

  21. [29]

    Technical guide to information security testing and assessment

    National Institute of Standards and Technology. Technical guide to information security testing and assessment. Technical Report SP 800-115, NIST, September 2008

    2008

  22. [30]

    Assessing security and privacy controls in information systems and organizations

    National Institute of Standards and Technology. Assessing security and privacy controls in information systems and organizations. Technical Report SP 800-53A Rev. 5, NIST, 2022

    2022

  23. [31]

    T. Ni, B. Eysenbach, and R. Salakhutdinov. Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs. InProceedings of the 39th International Conference on Machine Learning, pages 16691–16723. PMLR, June 2022. ISSN: 2640-3498

    2022

  24. [32]

    Oesch, A

    S. Oesch, A. Chaulagain, B. Weber, M. Dixson, A. Sadovnik, B. Roberson, C. Watson, and P. Austria. Towards a high fidelity training environment for autonomous cyber defense agents. InProceedings of the 17th Cyber Security Experimentation and Test Workshop, CSET ’24, pages 91–9...

    2024

  25. [33]

    Parisotto, F

    E. Parisotto, F. Song, J. Rae, R. Pascanu, C. Gulcehre, S. Jayakumar, M. Jader- berg, R. L. Kaufman, A. Clark, S. Noury, et al. Stabilizing transformers for reinforcement learning. InInternational conference on machine learning, pages 7487–7498. PMLR, 2020

    2020

  26. [34]

    Patterson, S

    A. Patterson, S. Neumann, M. White, and A. White. Empirical design in rein- forcement learning.Journal of Machine Learning Research, 25(318):1–63, 2024

    2024

  27. [35]

    Pleines, M

    M. Pleines, M. Pallasch, F. Zimmer, and M. Preuss. Memory gym: Towards endless tasks to benchmark memory capabilities of agents.Journal of Machine Learning Research, 26(6):1–40, 2025. 21

    2025

  28. [36]

    M. L. Puterman. Chapter 8 markov decision processes. InStochastic Models, volume 2 ofHandbooks in Operations Research and Management Science, pages 331–434. Elsevier, 1990

    1990

  29. [37]

    A. Raffin. Rl baselines3 zoo.https://github.com/DLR-RM/rl-baselines3-zoo, 2020

    2020

  30. [38]

    Raffin, A

    A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines3: Reliable reinforcement learning implementations.Journal of Machine Learning Research, 22(268):1–8, 2021

    2021

  31. [39]

    Automatedpenetrationtestingbasedonlstm and advanced curiosity exploration

    Q.Ren, J.Liu, X.Xiong, andC.Lu. Automatedpenetrationtestingbasedonlstm and advanced curiosity exploration. In2024 IEEE 5th International Conference on Pattern Recognition and Machine Learning (PRML), pages 150–156. IEEE, 2024

    2024

  32. [40]

    Sarraute, O

    C. Sarraute, O. Buffet, and J. Hoffmann. Penetration Testing == POMDP Solv- ing?, June 2013. arXiv:1306.4714 [cs]

    2013 arXiv

  33. [41]

    Schulman, S

    J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. InInternational conference on machine learning, pages 1889–1897. PMLR, 2015

    2015

  34. [42]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal Policy Optimization Algorithms, Aug. 2017. arXiv:1707.06347 [cs]

    2017 arXiv

  35. [43]

    Schwartz and H

    J. Schwartz and H. Kurniawatti. Nasim: Network attack simulator.https: //networkattacksimulator.readthedocs.io/, 2019

    2019

  36. [44]

    Silver, J

    D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge.nature, 550(7676):354–359, 2017

    2017

  37. [45]

    Standen, M

    M. Standen, M. Lucas, D. Bowman, T. J. Richer, J. Kim, and D. Marriott. CybORG: A Gym for the Development of Autonomous Cyber Agents, Aug. 2021. arXiv:2108.09118 [cs]

    2021 arXiv

  38. [46]

    B. E. Strom, A. Applebaum, D. P. Miller, K. C. Nickels, A. G. Pennington, and C. B. Thomas. Mitre att&ck: Design and philosophy. InTechnical report. The MITRE Corporation, 2018

    2018

  39. [47]

    R. S. Sutton and A. G. Barto.Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018

    2018

  40. [48]

    M. D. R. Team. Cyberbattlesim.https://github.com/microsoft/ cyberbattlesim, 2021. Created by Christian Seifert, Michael Betser, William Blum, JamesBono, KateFarris, EmilyGoren, JustinGrana, KristianHolsheimer, Brandon Marken, Joshua Neil, Nicole Nichols, Jugal Parikh, Haoran Wei

    2021

  41. [49]

    Terranova, A

    F. Terranova, A. Lahmadi, and I. Chrisment. Leveraging deep reinforcement learning for cyber-attack paths prediction: Formulation, generalization, and eval- uation. InProceedings of the 27th International Symposium on Research in At- tacks, Intrusions and Defenses, pages 1–16, 2024

    2024

  42. [50]

    K. Tran, M. Standen, J. Kim, D. Bowman, T. Richer, A. Akella, and C.-T. Lin. Cascaded Reinforcement Learning Agents forLarge Action Spaces in Autonomous Penetration Testing.Applied Sciences, 12(21):11265, Jan. 2022. Number: 21 Publisher: Multidisciplinary Digital Publishing Institute

    2022

  43. [51]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  44. [52]

    R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning.Machine learning, 8:229–256, 1992. 22

    1992

  45. [53]

    Global cybersecurity outlook 2023.Insight Report, 2023

    World Economic Forum. Global cybersecurity outlook 2023.Insight Report, 2023

    2023

  46. [54]

    Yang and X

    Y. Yang and X. Liu. Behaviour-Diverse Automatic Penetration Testing: A Curiosity-Driven Multi-Objective Deep Reinforcement Learning Approach, Feb

  47. [55]

    Zhang, O

    C. Zhang, O. Vinyals, R. Munos, and S. Bengio. A study on overfitting in deep reinforcement learning.arXiv preprint arXiv:1804.06893, 2018

    2018 arXiv

  48. [56]

    Zhang, J

    Y. Zhang, J. Liu, S. Zhou, D. Hou, X. Zhong, and C. Lu. Improved Deep Recur- rent Q-Network of POMDPs for Automated Penetration Testing.Applied Sci- ences, 12(20):10339, Jan. 2022. Number: 20 Publisher: Multidisciplinary Digital Publishing Institute

    2022

  49. [57]

    S. Zhou, J. Liu, D. Hou, X. Zhong, and Y. Zhang. Autonomous Penetration Testing Based on Improved Deep Q-Network.Applied Sciences, 11(19):8823, Jan

  50. [2021]

    23 Table 2: Hyperparameter search ranges for PPO, PPO-FS and PPO-AO

    Number: 19 Publisher: Multidisciplinary Digital Publishing Institute. 23 Table 2: Hyperparameter search ranges for PPO, PPO-FS and PPO-AO. Best performing parameter values for PPO are marked inbold, underlinedfor PPO- FS and initalicsfor PPO-AO. Hyperparameter Values Batch Siz...

  51. [2022]

    arXiv:2202.10630 [cs]

    arXiv

  52. [7602]

    The specific version we used for thestable-baselines3 framework is 2.4

    We simply pass them as an argument (–seed) to the scripts fromcleanRL andstable-baselines3. The specific version we used for thestable-baselines3 framework is 2.4. This holds for both the algorithm implementations as well as their hyperparameter tuning framework inrl-baselines...

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.