Pith. sign in

REVIEW 4 major objections 5 minor 28 references

An 8-billion-parameter LLM, steered only by a prompt, outperforms a reinforcement-learning baseline in a cyber-defense simulation, and its defensive policy can be distilled into a 64,910-parameter agent in about 240 episodes.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 01:35 UTC pith:C7QJ7UVP

load-bearing objection A transparent, well-scoped empirical study: the distillation works on CybORG reward, but 'defensive capability' is only as strong as the reward signal the authors themselves question. the 4 major comments →

arxiv 2607.28826 v1 pith:C7QJ7UVP submitted 2026-07-30 cs.LG

Distilling Knowledge from Large Language Models into Lightweight Reinforcement Learning Agents for Autonomous Cyber Operations

Konur Tholl , Fran\c{c}ois Rivest , Mariam El Mezouar , Adrian Taylor , Ranwa Al Mallah This is my paper
classification cs.LG
keywords autonomous cyber operationspolicy distillationlarge language modelsreinforcement learningprompt engineeringteacher-guided RLCybORGCAGE Challenge 2
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that a large language model pretrained on cybersecurity text can serve as a defensive decision-maker in a simulated enterprise network, and that its policy can be compressed into a tiny reinforcement-learning agent without losing effectiveness. The central comparison: an 8-billion-parameter LLM, steered only by a carefully engineered prompt, earns higher CybORG reward than a PPO baseline trained from scratch; then an online distillation procedure transfers that behavior into a 64,910-parameter agent in about 240 episodes, with lower run-to-run variance. The authors also test the approach across 4- to 12-host network topologies and find it transfers reasonably, with one exception. Finally, they show that seven teacher-guided stabilization tricks fail to consistently beat the teacher, which they read as evidence that reward-driven RL and teacher-guided strategies are policy-misaligned in this environment. The paper's stated motivation: a practical path from expensive frontier models to lightweight, deployable cyber defenders.

Core claim

On the paper's own terms, the central claim is that a frozen, cybersecurity-pretrained 8-billion-parameter LLM, optimized only by prompt engineering, can outperform a baseline PPO agent in a modified CybORG CAGE Challenge 2 environment, and that the LLM's policy can be transferred online into a 64,910-parameter RL agent using action masking and a teacher-log-likelihood loss. The distilled agent, trained for 240 episodes, matches the teacher's reward level and shows lower standard error than the PPO baseline run for 50,000 episodes. The authors further claim that the method transfers across 4- to 12-host scenarios, with only the 7-host environment clearly favoring the baseline, and that none

What carries the argument

Two equations carry the argument. Equation (1) is action masking: at each step, every action not recommended by the LLM teacher gets probability zero, so the student is forced to act like the teacher from the first episode. Equation (2) is a teacher-derived loss: the negative log probability the student assigns to the teacher's recommended action, which is optimized for 240 episodes before the student acts alone. This combination turns the LLM's prompt-conditioned recommendations into a supervised imitation signal, replacing environmental reward as the training objective during distillation. The teacher itself remains frozen; only the prompt changes, using a structured chain-of-thought scaff

Load-bearing premise

Every headline comparison is measured by the simulator's single reward number, which the authors admit may not reflect real defensive skill, and the teacher prompt was tuned to improve that same number, so the teacher's edge is not independently validated.

What would settle it

Evaluate both the 8-billion-parameter teacher and the 64,910-parameter distilled agent on a defense-quality signal independent of CybORG's scalar reward, such as the fraction of critical hosts compromised before containment or a different network simulator with the same action set, and check whether the distilled agent still matches the teacher. A second falsifier: run the PPO baseline with substantially more episodes or better hyperparameters; if its mean reward stabilizes above the distilled agent's mean, the central performance claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A 64,910-parameter agent trained for about 240 episodes can match the reward of its 8-billion-parameter teacher, making LLM-derived defensive policies deployable in resource-constrained settings at roughly 0.0008% of the teacher's parameter count.
  • Because the LLM teacher is frozen and needs no fine-tuning, the only per-environment cost of the method is prompt design and the short distillation phase.
  • In CybORG, the method transfers across 4- to 12-host topologies: the distilled agent matches or beats baseline mean reward in most scenarios and exhibits substantially lower variance; the 7-host case is the clear exception.
  • Reward-driven RL optimization after teacher guidance tends to degrade toward baseline performance, and none of the seven stabilization techniques consistently exceed the teacher, suggesting a policy-alignment ceiling in this environment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the prompt was selected by iterating against the same CybORG reward, the teacher's advantage is partly a selected property of that metric; an independent measure of defensive quality would be needed to know whether the distilled agent is genuinely better at defense, not just at scoring.
  • The same masked behavioral-cloning recipe could in principle be applied to any sequential task where an LLM can express a policy but is too large to deploy, suggesting a general 'prompt, then distill' route for LLM-guided control.
  • The failure of the seven stabilization variants hints that mixing teacher-derived loss with PPO's own objective is the problem; a testable alternative is to keep the teacher loss dominant or use a KL constraint to the teacher during post-distillation fine-tuning, rather than decaying it away.
  • The 7-host anomaly and the topology-dependent distillation cutoff suggest that distillation duration should be adaptively tied to student-teacher agreement or action-space size, not fixed at 240 episodes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes an online policy-distillation framework in which a frozen 8-billion-parameter cybersecurity LLM, optimized only through prompt engineering, is used as a teacher for a 64,910-parameter RL student in a modified CybORG CAGE Challenge 2 environment. The teacher's action is used to mask the student's action distribution and to provide a cross-entropy loss (Eq. 2) for 240 episodes; the student then acts independently. The authors report that the prompt-optimized LLM outperforms a PPO baseline, that the distilled agent maintains comparable performance with lower variance, that the approach transfers across 4-12-host scenarios, and that none of seven teacher-guided stabilization variants consistently surpasses the optimized teacher policy. Limitations acknowledge that CybORG is a simulation and that the scalar reward may not fully capture optimal defensive behavior.

Significance. If the central claims hold, the paper offers a practical compression pathway for deploying large cybersecurity LLMs in resource-constrained settings, and it provides a useful negative result about policy-alignment limitations in teacher-guided RL. The work is strengthened by its reproducible artifacts (code, configurations, prompts in the accompanying repository), by reporting means with ±1 standard error over 10 runs, and by honestly disclosing the 7-host exception and baseline parity in several transfer scenarios. The main significance is currently qualified by the fact that all headline comparisons are scored on the same CybORG scalar reward used to iteratively select the prompt, and by the authors' own admission that this reward may not capture optimal defensive behavior.

major comments (4)
  1. [§4.1, §5.2] The headline claim that the LLM 'can outperform a baseline RL agent' is established only against CybORG's scalar reward, which is the same objective used to select the optimized prompt ('iterative process ... until superior performance was observed', §4.1). Since §5.2 concedes that the reward 'may not fully capture optimal defensive behavior', the teacher's superiority is a selected property of the evaluation metric rather than an independent measure of cyber-defense quality, and the subsequent distillation and transferability results inherit this limitation. Please either add an independent validation of policy quality (e.g., mission success, host-compromise counts, expert assessment on held-out scenarios) or explicitly scope all conclusions to 'reward under the CybORG objective'.
  2. [§4.3, Fig. 7] The transferability conclusion is stronger than the evidence. The text reports that in the 5-, 6-, 8-, and 9-host scenarios the baseline achieves similar performance by roughly episode 1250, and in the 7-host scenario the baseline outperforms the distilled agent by roughly episode 700. Only a subset of the nine scenarios is discussed, with no aggregate statistics or significance tests. This does not support 'transfers reasonably well' as stated. Please report all scenarios (4-12 hosts), include per-scenario final rewards with confidence intervals, and either soften the transferability claim or identify conditions under which the distilled agent is competitive.
  3. [§4.2, Fig. 4] The claim that the distilled agent 'outperforms' the baseline PPO agent relies on visual comparison of means with ±1 SE and on the assertion that the baseline mean 'never appears to stabilize beyond' the distilled agent. No statistical test is reported, and the text notes that individual baseline runs surpass the distilled agent and that the baseline mean temporarily surpasses it after 23,000 episodes. Provide a formal comparison of final-performance distributions (e.g., bootstrap CI, Mann-Whitney, or effect size) and state the comparison horizon over which 'outperforms' is claimed.
  4. [§3.1, Eq. (2)] The distillation claim is not directly validated against the teacher. The paper compares the distilled agent to untrained PPO and to teacher-guided RL, but no experiment reports how closely the student's action distribution or episode reward matches the LLM teacher's policy. Since the framework rests on the assumption that 240 episodes of the loss in Eq. (2) transfer the teacher's policy, please report student-teacher agreement (e.g., action-match rate, reward gap on a held-out set) and show sensitivity to the distillation horizon.
minor comments (5)
  1. [Fig. 7] The legend and axis labels are not legible in the manuscript; include a table of per-scenario mean rewards with standard errors.
  2. [Eq. (3)] The notation Mt(ai) overloads the masking matrix introduced in Eq. (1). In Eq. (3) it appears to be a scalar indicator for teacher-recommended actions; please clarify the notation to avoid ambiguity.
  3. [§4.4, Figs. 5-6] State the number of runs and the hyperparameter values used for each of the seven stabilization variants. The figures show standard errors but the run count and exact schedules are not specified.
  4. [§3.2] The term 'chain-of-thought scaffolding' is new; define it precisely and consider including the full prompt in an appendix rather than only in the repository.
  5. [Throughout] Minor typographical issues include 'B-line agent' and 'to optimize its attack trajectory'; these should be corrected in a final pass.

Circularity Check

1 steps flagged

Optimized-prompt superiority is selected on the evaluation reward; distilled-vs-baseline comparison remains externally grounded.

specific steps
  1. fitted input called prediction [Section 4.1, 'Prompt Engineering' (see also Section 5.2 Limitations)]
    "This was an iterative process of incrementally modifying the chain-of-thought scaffolding to guide the LLM's decision-making until superior performance was observed. ... The metric used to evaluate the performance of the prompts is the reward obtained by the LLM with respect to CybORG's reward signals."

    The optimized prompt is the fitted object: the scaffolding was modified until it produced high CybORG reward, and that same scalar reward is then used as evidence that the LLM outperforms the baseline. The reported ~35% improvement is the selection criterion, not an independent out-of-sample prediction of LLM defensive capability. Section 5.2 concedes the reward 'may not fully capture optimal defensive behavior,' so the teacher-quality premise that anchors the distillation is partly constructed from the evaluation metric rather than independently demonstrated. The distilled-agent-vs-PPO comparison itself is external and not forced, keeping the circularity partial.

full rationale

The paper's central distillation comparison is genuinely external: the 64,910-parameter student is compared with a from-scratch PPO baseline over 50,000 episodes, so the student's relative performance is not true by construction. The Eq. 1 action-masking / Eq. 2 teacher-loss distillation is a standard behavioral-cloning objective; the student acts without masking after episode 240, so parity with the teacher is not definitional. The citations to the authors' prior work ([21], [23], [22]) supply the baseline prompt, masking technique, and environment modifications, but the current paper re-evaluates the teacher and student against an external baseline, so those self-citations are not load-bearing. The one substantive circular step is the teacher-quality claim: the prompt was iteratively tuned on the same CybORG reward that is then reported as the demonstration of LLM superiority, and the paper's own limitations section admits that this reward may not fully capture defensive quality. Thus the teacher's 'superiority' is a selected property of the evaluation metric, partially circular, while the core distillation comparison retains independent content.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claim rests on two tuned quantities (per-scenario distillation cut-off, iteratively selected prompt), an uncalibrated baseline, and an unverified reward proxy. The axioms list states the domain premises that must hold for the comparisons to mean what the abstract says.

free parameters (3)
  • Distillation cut-off (episodes) = 240 episodes for the base scenario; adjusted per topology (not tabulated)
    The only hyperparameter changed across scenarios (§3.3); chosen per environment because 'it was found that less time for knowledge transfer was required for smaller network topologies.' The transferability results depend on this per-scenario fit.
  • Optimized prompt structure (chain-of-thought scaffolding, reasoning rules) = Not specified numerically; final prompt in GitHub repo
    Selected by iterative search against CybORG reward in §4.1 ('incrementally modifying the chain-of-thought scaffolding ... until superior performance was observed'). The teacher's headline 35% improvement over the baseline prompt is a selection result, not an independent estimate.
  • Learning-rate schedule and decay (stabilization variants) = Critic 1.6e-3 to 3.2e-3 to 0.8e-3; actor 1.6e-3 to 0.8e-3; multiplicative decay 0.99
    Hand-chosen for the seven teacher-guided stabilization experiments (§4.4). These feed the negative result about failing to consistently surpass the teacher.
axioms (5)
  • domain assumption CybORG's scalar reward is a valid and sufficient measure of defensive capability for comparing teacher, student, and baseline.
    Every quantitative claim (Figs 2–7) is scored on CybORG rewards; the authors themselves note in §5.2 that the reward 'may not fully capture optimal defensive behavior.'
  • domain assumption The LLM's extracted action (regex, falling back to semantic similarity) faithfully represents the LLM's recommendation.
    Action extraction reuses the procedure from [23] (§3.2); no error rate or failure analysis of the extraction is reported.
  • domain assumption The modified red-side B-line agent provides a representative and fair adversary across all scenarios.
    The red agent was 'modified ... to optimize its attack trajectory' per environment (§3.3, §4.3); the modification procedure is underspecified, so adversary strength is uncontrolled between scenarios.
  • domain assumption A single in-house PPO agent is an adequate baseline for the claim of outperforming 'a baseline RL agent.'
    No comparison to published CAGE Challenge 2 results or other CybORG agents (§4.2), so the strength of the baseline is uncalibrated.
  • ad hoc to paper 240 episodes of teacher cross-entropy loss under action masking converges the student to the teacher's policy.
    No student-teacher action-agreement metric is reported; distillation success is inferred only from reward parity (§3.1, §4.2).

pith-pipeline@v1.3.0-alltime-deepseek · 15902 in / 19979 out tokens · 210132 ms · 2026-08-03T01:35:05.921225+00:00 · methodology

0 comments
read the original abstract

Autonomous Cyber Operations (ACO) are increasingly important for defending enterprise networks as cyber threats continue to evolve in sophistication. ACO applications commonly employ Reinforcement Learning (RL) agents to learn defensive behaviors through interaction with environments. However, RL agents typically require extensive exploration during training, often resulting in unstable behavior and poor initial decision-making before converging toward effective defense strategies. In this work, we investigate the use of a Large Language Model (LLM) to improve autonomous defensive decision-making within an ACO environment. Through prompt engineering rather than fine-tuning, we demonstrate that an 8-billion parameter LLM pretrained on cybersecurity data can outperform a baseline RL agent in a modified CybORG CAGE Challenge 2 environment. We then propose an online policy distillation framework that transfers the LLM's defensive policy into a lightweight RL agent containing only 64,910 parameters, reducing model size by several orders of magnitude while maintaining effective defensive capabilities. This provides a pathway toward operationalizing frontier cybersecurity models within lightweight, deployable agents. To evaluate transferability, we construct CybORG scenarios ranging from 4 to 12 hosts and assess the approach across varying network configurations. We also evaluate teacher-guided RL stabilization strategies and observe that none consistently surpass the optimized teacher policy, suggesting policy-alignment limitations between reward-driven RL optimization and teacher-guided defense strategies. Our results demonstrate the potential of cybersecurity-focused LLMs as sources of expertise for autonomous cyber defense, while policy distillation provides a practical path toward operationalizing frontier cybersecurity models within efficient, scalable agents.

Figures

Figures reproduced from arXiv: 2607.28826 by Adrian Taylor, Fran\c{c}ois Rivest, Konur Tholl, Mariam El Mezouar, Ranwa Al Mallah.

Figure 1
Figure 1. Figure 1: Overview of transforming CybORG’s state space into a coherent prompt and extracting an executable action from the LLM. 3.3 Transferability To assess the transferability of the proposed approach across different environ￾ments, we created nine additional scenarios ranging in complexity from 4 to 12 hosts. A similar B-line agent was used on the red side to infiltrate the networks, with minor modifications to … view at source ↗
Figure 2
Figure 2. Figure 2: Evaluation of the Standard and Optimized Prompt across 10 independent runs for 50 episodes. Per-episode mean reward with a ±1 standard error [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparing distillation with teacher-guided RL. Per-episode mean reward with a ±1 standard error for 2,000 episodes. 0 10000 20000 30000 40000 50000 Episode 0 10 20 30 40 50 60 70 Reward Running Mean - 10 Episodes ± 1 SE Baseline LLM [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparing the distilled agent against the baseline PPO agent across 10 inde￾pendent runs using a 10-episode running average with a ±1 standard error for 50,000 episodes [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: First four attempts to increase performance beyond the LLM with the optimized prompt. Techniques include: adding the LLM’s feedback to the critic’s loss, initializing the critic as a pretrained model over 10,000 episodes, dynamically changing the actor and critic LRs, and adding extra learning epochs to the critic during the transition from teacher-guided to independent RL. Plots show a mean reward after a… view at source ↗
Figure 6
Figure 6. Figure 6: Next three attempts to increase performance beyond the LLM with the opti￾mized prompt. Techniques include: incorporating the LLM’s guidance as a distribution (instead of a single action), stopping the critic learning after the transition to inde￾pendent RL, and decaying the teacher’s impact by a multiplicative factor instead of a linear constant. Plots show a mean reward after applying a 10-episode running… view at source ↗
Figure 7
Figure 7. Figure 7: Evaluation of the LLM-distilled agent against the baseline agent across different CybORG scenarios ranging from 4 to 12 hosts. The dotted line denotes the point at which the distilled agent has transitioned to acting independently of the LLM. The results are across 10 independent runs, with a 10-episode running average and a ± 1 standard error [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 5 canonical work pages · 1 internal anchor

  1. [1]

    In: 12th International Conference on Learning Representations, ICLR 2024, May 7, 2024 - May 11, 2024

    Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Ramos, S., Geist, M., Bachem, O.: ON-POLICY DISTILLATION OF LANGUAGE MODELS: LEARNING FROM SELF-GENERATED MISTAKES. In: 12th International Conference on Learning Representations, ICLR 2024, May 7, 2024 - May 11, 2024. p. et al; Google Deepmind; Google Research; Meta; Microsoft. 12th International Confere...

    2024

  2. [2]

    https://doi.org/10.48550/arXiv.2309.16021, http://arxiv.org/abs/2309.16021, arXiv:2309.16021 [cs]

    Ali, T., Kostakos, P.: HuntGPT: Integrating Machine Learning-Based Anomaly Detection and Explainable AI with Large Language Models (LLMs) (Sep 2023). https://doi.org/10.48550/arXiv.2309.16021, http://arxiv.org/abs/2309.16021, arXiv:2309.16021 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2309.16021 2023

  3. [3]

    https://doi.org/10.48550/arXiv.2002.10667

    Baillie, C., Standen, M., Schwartz, J., Docking, M., Bowman, D., Kim, J.: CybORG: An Autonomous Cyber Operations Research Gym (Feb 2020). https://doi.org/10.48550/arXiv.2002.10667

    Pith/arXiv arXiv doi:10.48550/arxiv.2002.10667 2020

  4. [4]

    In: TA-Explore: Teacher-Assisted Exploration for Facilitating Fast Reinforcement Learning

    Beikmohammadi, A., Magnusson, S.: TA-Explore: Teacher-Assisted Exploration for Facilitating Fast Reinforcement Learning. In: TA-Explore: Teacher-Assisted Exploration for Facilitating Fast Reinforcement Learning. 2023 International Foun- dation for Autonomous Agents and Multiagent Systems, London, United Kingdom (May 2023)

    2023

  5. [5]

    Generalized Kullback-Leibler Divergence Loss

    Cui, J., Zhu, B., Xu, Q., Tian, Z., Qi, X., Yu, B., Zhang, H., Hong, R.: Generalized Kullback-Leibler Divergence Loss (Mar 2025). https://doi.org/10.48550/arXiv.2503.08038, http://arxiv.org/abs/2503.08038, arXiv:2503.08038 [cs]

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2503.08038 2025

  6. [6]

    In: Chen, Y., Lin, C.W., Chen, B., Zhu, Q

    Guastalla, M., Li, Y., Hekmati, A., Krishnamachari, B.: Application of Large Language Models to DDoS Attack Detection. In: Chen, Y., Lin, C.W., Chen, B., Zhu, Q. (eds.) Security and Privacy in Cyber-Physical Systems and Smart Vehicles, vol. 552, pp. 83–99. Springer Nature Switzerland (2024). https://doi.org/10.1007/978-3-031-51630-6_6, series Title: Lectu...

    doi:10.1007/978-3-031-51630-6_6 2024

  7. [7]

    IJRDO -JOURNAL OF MATHEMATICS9, 1–5 (Sep 2023)

    Jawad, E.: THE DEEP NEURAL NETWORK-A REVIEW. IJRDO -JOURNAL OF MATHEMATICS9, 1–5 (Sep 2023). https://doi.org/10.53555/m.v9i9.5842

    work page doi:10.53555/m.v9i9.5842 2023

  8. [8]

    arXiv preprint arXiv:2309.07388 (2023)

    Kiely, M., Bowman, D., Standen, M., Moir, C.: On autonomous agents in a cyber defence environment. arXiv preprint arXiv:2309.07388 (2023)

    Pith/arXiv arXiv 2023

  9. [9]

    In: 2024 Inter- national Conference on Military Communication and Information Systems (ICM- CIS)

    Loevenich, J.F., Adler, E., Mercier, R., Velazquez, A., Lopes, R.R.F.: Design of an Autonomous Cyber Defence Agent using Hybrid AI models. In: 2024 Inter- national Conference on Military Communication and Information Systems (ICM- CIS). pp. 1–10 (Apr 2024). https://doi.org/10.1109/ICMCIS61231.2024.10540988, https://ieeexplore.ieee.org/document/10540988/?a...

    arXiv 2024

  10. [10]

    IEEE Computer Society (Sep 2024), https://ieeexplore-ieee- org.journal.rmc.ca/document/10645591

    Matthew, S., Matthew, S.: Prompt Engineering ChatGPT for Code- names. IEEE Computer Society (Sep 2024), https://ieeexplore-ieee- org.journal.rmc.ca/document/10645591

    arXiv 2024

  11. [11]

    IEEE Ac- cess12, 120292–120305 (2024)

    Mcdonald, G., Li, L., Mallah, R.A.: Finding the Optimal Security Policies for Au- tonomous Cyber Operations With Competitive Reinforcement Learning. IEEE Ac- cess12, 120292–120305 (2024). https://doi.org/10.1109/ACCESS.2024.3446310, https://ieeexplore.ieee.org/document/10639381/

    arXiv 2024

  12. [12]

    In: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)

    Nashid, N., Sintaha, M., Mesbah, A.: Retrieval-Based Prompt Selection for Code- Related Few-Shot Learning. In: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). pp. 2450–2462. IEEE Computer Society, Mel- bourne, Australia (May 2023). https://doi.org/10.1109/ICSE48619.2023.00205, https://ieeexplore.ieee.org/document/10172590/

    arXiv 2023

  13. [13]

    https://doi.org/10.48550/arXiv.2310.07745, http://arxiv.org/abs/2310.07745, arXiv:2310.07745 [cs]

    Palmer, G., Parry, C., Harrold, D.J.B., Willis, C.: Deep Reinforce- ment Learning for Autonomous Cyber Operations: A Survey (Sep 2024). https://doi.org/10.48550/arXiv.2310.07745, http://arxiv.org/abs/2310.07745, arXiv:2310.07745 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.07745 2024

  14. [14]

    IEEE Robotics and Automation Letters3(4), 4423–4430 (Oct 2018)

    Pfeiffer, M., Shukla, S., Turchetta, M., Cadena, C., Krause, A., Sieg- wart, R., Nieto, J.: Reinforced Imitation: Sample Efficient Deep Re- inforcement Learning for Mapless Navigation by Leveraging Prior Demonstrations. IEEE Robotics and Automation Letters3(4), 4423–4430 (Oct 2018). https://doi.org/10.1109/LRA.2018.2869644, https://ieeexplore.ieee.org/doc...

    arXiv 2018

  15. [15]

    https://doi.org/10.1007/s00521- 025-11162-0, https://doi.org/10.1007/s00521-025-11162-0

    Pozzi, A., Incremona, A., Tessera, D., Toti, D.: Mitigating exposure bias in large languagemodeldistillation:animitationlearningapproach.NeuralComputingand Applications37(18), 12013–12029 (Jun 2025). https://doi.org/10.1007/s00521- 025-11162-0, https://doi.org/10.1007/s00521-025-11162-0

    doi:10.1007/s00521- 2025

  16. [16]

    In: 2025 IEEE International Conference on Fuzzy Systems (FUZZ)

    Santos, F.M., Gonçalves, A., Sousa, J.M., Vieira, S.M.: Distilling Knowl- edge from Deep Neural Networks to Neuro-Fuzzy Inference Systems. In: 2025 IEEE International Conference on Fuzzy Systems (FUZZ). pp. 1–6 (Jul 2025). https://doi.org/10.1109/FUZZ62266.2025.11152050, https://ieeexplore.ieee.org/document/11152050, iSSN: 1558-4739

    arXiv 2025

  17. [17]

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal Policy Optimization Algorithms (Aug 2017), http://arxiv.org/abs/1707.06347, arXiv:1707.06347 [cs]

    Pith/arXiv arXiv 2017

  18. [18]

    In: 2025 IEEE Inter- national Conference on Consumer Electronics (ICCE)

    Son, M., Lee, S.: Performance Analysis of Prompt-Engineering Techniques for Large Language Model. In: 2025 IEEE Inter- national Conference on Consumer Electronics (ICCE). pp. 1– 5 (Jan 2025). https://doi.org/10.1109/ICCE63647.2025.10930066, https://ieeexplore.ieee.org/document/10930066/, iSSN: 2158-4001

    arXiv 2025

  19. [19]

    MIT Press, Cambridge, MA, 2nd edn

    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 2nd edn. (2014) Distilling LLM Knowledge into Lightweight RL Agents 19

    2014

  20. [20]

    https://github.com/Poly-AIvsAI/LLMDistillationACO (2026)

    Tholl, K.: Llmdistillationaco: Distilling knowledge from large language models into lightweight reinforcement learning agents for autonomous cyber operations. https://github.com/Poly-AIvsAI/LLMDistillationACO (2026)

    2026

  21. [21]

    In: 2025 IEEE Annual Congress on Artificial Intelligence of Things (AIoT)

    Tholl, K., El Mezouar, M., Al Mallah, R.: A Comparative Evaluation of Teacher-Guided Reinforcement Learning Techniques for Autonomous Cyber Op- erations. In: 2025 IEEE Annual Congress on Artificial Intelligence of Things (AIoT). pp. 845–849 (Dec 2025). https://doi.org/10.1109/AIoT66900.2025.00136, https://ieeexplore.ieee.org/document/11416350

    arXiv 2025

  22. [22]

    https://doi.org/10.48550/arXiv.2508.19278, http://arxiv.org/abs/2508.19278, arXiv:2508.19278 [cs]

    Tholl, K., Mezouar, M.E., Mallah, R.A.: Towards Production- Worthy Simulation for Autonomous Cyber Operations (Aug 2025). https://doi.org/10.48550/arXiv.2508.19278, http://arxiv.org/abs/2508.19278, arXiv:2508.19278 [cs]

    work page doi:10.48550/arxiv.2508.19278 2025

  23. [23]

    https://doi.org/10.48550/arXiv.2509.05311, http://arxiv.org/abs/2509.05311, arXiv:2509.05311 [cs]

    Tholl, K., Rivest, F., Mezouar, M.E., Taylor, A., Mallah, R.A.: Large Language Model Integration with Reinforcement Learning to Aug- ment Decision-Making in Autonomous Cyber Operations (Feb 2026). https://doi.org/10.48550/arXiv.2509.05311, http://arxiv.org/abs/2509.05311, arXiv:2509.05311 [cs]

    doi:10.48550/arxiv.2509.05311 2026

  24. [24]

    Vanessasml: https://huggingface.co/Vanessasml/cyber-risk-llama-3-8b (2024)

    2024

  25. [25]

    IEEE Robotics and Automation Letters 10(1), 612–619 (Jan 2025)

    Wang, J., Wang, T., Cai, W., Xu, L., Sun, C.: Boosting Effi- cient Reinforcement Learning for Vision-and-Language Navigation With Open-Sourced LLM. IEEE Robotics and Automation Letters 10(1), 612–619 (Jan 2025). https://doi.org/10.1109/LRA.2024.3511402, https://ieeexplore.ieee.org/document/10777561, conference Name: IEEE Robotics and Automation Letters

    arXiv 2025

  26. [26]

    Algorithms17(2), 60 (Feb 2024)

    Wang, Z., Li, X., Sun, L., Zhang, H., Liu, H., Wang, J.: Learning State-Specific Action Masks for Reinforcement Learning. Algorithms17(2), 60 (Feb 2024). https://doi.org/10.3390/a17020060, https://www.mdpi.com/1999-4893/17/2/60, number: 2 Publisher: Multidisciplinary Digital Publishing Institute

    work page doi:10.3390/a17020060 2024

  27. [27]

    https://doi.org/10.48550/arXiv.2310.05939, http://arxiv.org/abs/2310.05939, arXiv:2310.05939 [cs]

    Wiebe, J., Mallah, R.A., Li, L.: Learning Cyber Defence Tactics from Scratch with Multi-Agent Reinforcement Learning (Aug 2023). https://doi.org/10.48550/arXiv.2310.05939, http://arxiv.org/abs/2310.05939, arXiv:2310.05939 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.05939 2023

  28. [28]

    https://doi.org/10.48550/arXiv.2505.20335, http://arxiv.org/abs/2505.20335, arXiv:2505.20335 [cs] version: 4 20 K

    Yu, Z., Li, S., Zhang, X.: Language Model Distillation: A Tem- poral Difference Imitation Learning Perspective (Jan 2026). https://doi.org/10.48550/arXiv.2505.20335, http://arxiv.org/abs/2505.20335, arXiv:2505.20335 [cs] version: 4 20 K. Tholl et al. A Transferability As discussed in Sections 3 and 4, we evaluated the transferability of the distilled agen...

    work page doi:10.48550/arxiv.2505.20335 2026