Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Online Incident Response Planning under Model Misspecification through Bayesian Learning and Belief Quantization

T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Model-misspecified incident response learned online with a bounded error

desk verdict A solid, honest combination of Bayesian model learning and belief quantization for cyber incident response, but the headline robustness bound depends on an unmeasured constant and the CAGE-2 win is suggestive, not conclusive. read the letter →

arxiv 2508.14385 v1 pith:W6OOBC3A submitted 2025-08-20 cs.LG cs.AIcs.CRcs.SYeess.SY

Kim Hammar , Tao Li This is my paper
classification cs.LGcs.AIcs.CRcs.SYeess.SY
keywords modelmisspecificationBayesianlearningbeliefquantizationPOMDPincidentresponsecybersecuritydynamicprogrammingCAGE-2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents MOBAL, an online incident-response method that does not require a correct model of the attack. MOBAL maintains a Bayesian distribution over possible attack-transition parameters, updates this distribution from security alerts, samples a model conjecture, and quantizes its belief space into a finite Markov decision process so a response can be computed by dynamic programming. The paper proves that the posterior concentrates on consistent conjectures and bounds the gap between MOBAL's cost and the true optimal cost as ε/(1−γ) + γαc_max/(1−γ)², separating the error of belief quantization from the error of model misspecification. On the CAGE-2 benchmark, MOBAL remains competitive when the model is correct and substantially reduces cost when decoy-related attack probabilities are misspecified, where non-adaptive baselines degrade.

What carries the argument

The key machinery is the loop: a Bayesian posterior ρ_t over model parameters θ, updated via Eq. (6); a sampled conjecture θ_t ∼ ρ_t; a quantization map Φ projecting the continuous belief space B onto a finite grid B̃ of r-level representative beliefs (Eqs. 12–13); and a quantized MDP whose optimal value V* is computed by value iteration and lifted back to the original belief space via V*(Φ(b)). The load-bearing identity is Theorem 1: ||J̃ − J*||_∞ ≤ ε/(1−γ) + γαc_max/(1−γ)², which decomposes the total suboptimality into a quantization error ε and a transition-misspecification error α.

What would settle it

In the CAGE-2 misspecification scenario, set true decoy attack probabilities to values not in {0, 0.5, 1}, e.g., 0.2 and 0.35, run MOBAL, and track the estimated total-variation distance α_t between the sampled conjecture's transition probabilities and the true simulator. If α_t does not decrease and cost rises toward the baseline range, the robustness claim is scenario-specific rather than a general consequence of the bound.

Watch

Extended reading notes

Core claim

The central claim is that effective incident response planning can be performed without a correctly specified system model. The method treats the transition parameters as unknown, maintains a posterior ρ_t, updates it by Bayesian learning from observations, samples a parameter θ_t, and plans with a quantized MDP over a finite grid of representative beliefs. The paper establishes that (i) the posterior concentrates on parameter vectors minimizing the KL discrepancy to the true observation process; (ii) using a misspecified model costs at most γαc_max/(1−γ)², with α the total variation gap of transition probabilities; (iii) the quantization error vanishes as r→∞; and (iv) the combined suboptim

Load-bearing premise

The theorem's guarantee is only meaningful if the total-variation gap α between the learned conjecture's and true transitions is small, and the paper does not prove that Bayesian learning shrinks α when the true parameter lies outside the conjecture set; Section 4.2 explicitly allows the most consistent conjecture to deviate significantly from the true model.

Editorial extensions

If this is right

  • Incident-response tools can be deployed without a simulator calibrated to the exact environment; the model can be learned online from security alerts.
  • Because the suboptimality bound separates ε and α, an operator can reduce cost either by increasing quantization resolution r (trading compute) or by expanding/improving the conjecture set Θ so Bayesian learning brings α closer to zero.
  • As r→∞ and if the learned conjecture approaches the true transition model, MOBAL's strategy converges to the true optimal incident-response policy.
  • On the CAGE-2 benchmark, in the misspecified scenario, MOBAL's mean cost is 35.91±9.01 versus 91–124 for non-adaptive baselines, indicating that the adaptation mechanism, not a better static model, is responsible for robustness.
  • MOBAL runs fully online without an offline training phase, with a per-time-step planning time around 8.5 minutes on the tested hardware, which the paper notes is acceptable for many but not time-critical scenarios.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This loop is not specific to cybersecurity: any finite-state POMDP with unknown transition parameters—such as medical triage or infrastructure monitoring—could use the same Bayesian-conjecture-plus-belief-quantization scheme, although the paper evaluates only CAGE-2.
  • The main open question the theoretical part leaves is whether the misspecification gap α actually decays when the true parameter lies outside Θ; a natural deployment-time check is to estimate α from prediction errors of the sampled conjecture and use it as a warning that Θ should be expanded.
  • Planning on a single sampled conjecture discards posterior uncertainty; one could extend MOBAL to sample multiple hypotheses and select actions by Thompson-sampling or Bayes-adaptive planning, which may reduce the effective α at the price of more computation.
  • The error bound suggests a concrete design principle: quantization resolution and model-conjecture quality are interchangeable to first order, so when compute is cheap, one can compensate partly for a poor initial model by increasing r; the CAGE-2 experiment uses only r=1 after feature reduction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes MOBAL, an online method for incident-response planning in POMDPs under model misspecification. At each step, a belief over the system state is estimated (Eqs. 4–5), a Bayesian posterior over conjectured model parameters is updated from observations (Eq. 6), a model is sampled and its belief space is quantized into a finite MDP (Eqs. 12–14), and the resulting strategy is computed by dynamic programming. The paper claims: (i) asymptotic posterior concentration on observation-consistent conjectures (Prop. 1); (ii) a bound on the cost suboptimality due to transition misspecification in terms of a total-variation constant α (Prop. 2); (iii) a quantization error bound controlled by the oscillation ε of the conjectured optimal cost over quantization cells (Prop. 3); and (iv) the combined sub-optimality bound ε/(1-γ)+γα c_max/(1-γ)^2 (Thm. 1). Experiments on CAGE-2 show MOBAL is competitive in the correctly specified case and substantially better than cardiff, c-pomcp, pomcp, and PPO under a misspecified decoy-probability scenario.

Significance. If the claims were fully established, the paper would make a useful step: it explicitly decomposes the sub-optimality into quantization and misspecification terms, and it demonstrates empirically that online Bayesian adaptation can help in a challenging cyber-response benchmark. The use of a standard benchmark, comparison with strong baselines, and an illustrative example with posterior evolution are assets. The theoretical decomposition in Thm. 1 is appealing, and the authors are to be credited for attempting a finite-horizon-agnostic error analysis of a misspecified online POMDP planner. However, as written, the theoretical guarantees are conditional: α is not bounded or measured, the quantization error bound in Prop. 3 appears to omit a transition-mismatch term, and the belief estimator's dependence on the unknown model is under-specified. The experimental evidence is promising but does not by itself establish the claimed principled robustness guarantee.

major comments (5)
  1. [Thm. 1 / Eq. (10) / Sec. 4.2] The robustness guarantee is conditional on α, the total-variation distance between the conjectured and true transition kernels, but α is never bounded or measured. Prop. 1 establishes posterior concentration w.r.t. the observation-space discrepancy K (Eq. 7), not w.r.t. the transition TV in Eq. (10). The paper itself concedes in Sec. 4.2 that even the most consistent conjecture may yield a transition model that deviates significantly from the true model. In the illustrative example the posterior concentrates on θ=0 while the true θ*=0.2, so α is plausibly not small; with γ=0.99 the second term in Thm. 1 is about 2000·c_max, making the bound vacuous. A proof that α→0 under stated conditions, or an experimental measurement/bound on α, is necessary for the paper's central robustness claim.
  2. [Prop. 3 / Eqs. (13)–(16)] The stated bound |~J(b)-J*(b)| ≤ ε/(1-γ) does not follow from the definitions given. The quantized transition kernel p̂_θ(·|~b,a) is evaluated at the representative ~b=Φ(b), while J* is evaluated at b. When ~b≠b, the kernel mismatch induces an additional term involving the total-variation distance between p_θ(·|~b,a) and p_θ(·|b,a) times ||J*||∞; this term is not controlled by ε. Aggregation theorems of this type typically require an explicit Lipschitz/transition-consistency assumption and yield a bound with an extra transition-mismatch term. As written, Prop. 3—and therefore the quantization component of Thm. 1—is not established.
  3. [Appendix C / Prop. 2] The first equality in the proof of Prop. 2 writes the Bellman equations for both value functions J* and J* using the same action a. A single action is not generally the argmin for both value functions, so the displayed equality is not valid for either J* or J*. The standard simulation-lemma proof can be repaired by taking the optimal action of one model and using the Bellman inequality for the other, but the proof as printed is incorrect. Since Prop. 2 is load-bearing for Thm. 1, the proof must be corrected.
  4. [Sec. 4.1 / Eqs. (4)–(5)] The belief estimator is defined using transition probabilities p_{ss'}(a) and observation probabilities z(o|s') with no dependence on the conjectured parameter θ. If these are the true dynamics, the algorithm assumes access to the very model that is supposed to be misspecified. If they are the conjectured dynamics, the dependence on θ_t must be made explicit, and the optimality/consistency analysis must account for the fact that the belief is computed under a possibly wrong model. The paper does not resolve this ambiguity, which is central to the claim of operating under model misspecification.
  5. [Appendix B / Prop. 1] The proof of Prop. 1 relies on [21, Lemma 7] and [21, Thm. 3] for the critical almost-sure convergence of Z_t to K, and on Assumption 2 (regularity conditions). The assumptions are not verified for the particle-filtered, feature-reduced, quantized implementation used in the CAGE-2 experiment, and the theorem statement in the main text refers only to 'suitable regularity conditions.' The paper should either state these conditions explicitly as assumptions of Prop. 1 or verify them for the practical instantiation, otherwise the consistency claim is not self-contained.
minor comments (5)
  1. [Eq. (7)] The likelihood ratio in the definition of K is written with identical-looking numerator and denominator. Please clarify which factor uses the conjectured parameter θ and which uses the true parameter θ*, e.g., by using explicit superscripts.
  2. [Appendix E / Prop. 4] After deriving max ||b-b'||∞ ≤ 2n/r, the text says 'Choose any r such that 1/r < δ'; the choice should be 2n/r < δ to match the displayed bound. The result is not affected, but the argument should be consistent.
  3. [Table 3 / Sec. 6] The reproducibility appendix does not list the quantization resolution r used in CAGE-2 (stated as r=1 in Sec. 6), the number of Monte-Carlo samples for the approximate Bayesian update, or details of the feature reduction. These parameters should be included in Table 3.
  4. [Prop. 3 statement] Typo: 'conjetured' should be 'conjectured.'
  5. [Table 1] The text says results within the margin of statistical equivalence are highlighted in bold, but no bold entries appear in the table as typeset. If this formatting is missing, it should be restored or the sentence removed.

Circularity Check

2 steps flagged · score 3.0 of 10

Theorem 1 is a non-circular conditional simulation-lemma bound, but the consistency proposition's proof is deferred to the authors' own [21], and the CAGE-2 'misspecification' scenario contradicts the paper's own definition (true θ=0.5 lies in mobal's Θ={0,0.5,1}).

  1. self citation load bearing [Appendix B (Proof of Proposition 1), Section 4.2]
    "Due to page restrictions, we present only the main proof steps here. See our earlier work [21, Thm. 3] for technical details. To begin with, we invoke two lemmas from [21, Lemma 8, 9] that ensure the regularity of the belief space and the integrand in (9)."

    The paper's stated contribution 'we prove that Bayesian learning is asymptotically consistent' is discharged by referring to the authors' own [21, Thm. 3] and importing [21, Lemmas 6-9] for the key steps (the well-defined measure of Lemma 3 and the Z_t-to-K equivalence of Lemma 7). Because Kim Hammar and Tao Li are authors of both works, the proof of Prop. 1 reduces, as presented, to a self-citation rather than a proof contained in this paper. Mitigating factors: [21] is a peer-reviewed IEEE TIFS publication and thus real independent evidence, and Prop. 1 is not used in the proof of Theorem 1 (which uses only Props. 2-3), so the central sub-optimality bound is not carried by this citation.

  2. other [Section 6, 'Evaluation scenarios', item (2) misspecification; Table 2]
    "(2) misspecification: In this scenario, we consider the case where the model is misspecified, i.e., ρ0(θ) = 0. ... For the misspecification scenario, we run these methods on a simulator of cage-2 where all of the probability parameters listed in Table 2 are fixed to 0.5. (Table 2: Conjectures Θ = {0, 0.5, 1})"

    Section 3 defines misspecification as 'the conjecture distribution ρ_t is misspecified if θ ∉ Θ', yet the 'misspecification' experiment fixes the true parameter to 0.5 and Table 2 gives mobal's conjecture set as Θ={0,0.5,1} for every configuration. Since the prior ρ0 is supported on Θ, ρ0(0.5)>0, so the setup is correctly specified for mobal's learned parameter: the Bayes update can in principle drive the Eq. (10) constant α to 0. The observed advantage (35.91 vs 94.28/92.71/91.51/124.38) is therefore the advantage of an adaptive learner whose hypothesis class contains the truth over baselines deliberately given a wrong fixed model; it does not exercise the θ*∉Θ regime that Theorem 1's misspecification term addresses, which Section 4.2 concedes can be arbitrarily bad. The robustness conclu

full rationale

The derivation chain of Theorem 1 is not circular: its proof is a triangle inequality over Prop. 2 (a standard simulation lemma, proved in Appendix C via the textbook recursion) and Prop. 3 (explicitly attributed to the external Tsitsiklis–van Roy [53, Thm. 1], with a generalization in [39, Prop. 3]). The misspecification constant α (Eq. 10) is a genuine hypothesis, not the defined value of the error; the ε/(1−γ) term is the standard horizon-amplified aggregation error. The paper itself flags the key limitation in Section 4.2: 'if the true parameter vector θ lies outside the set Θ, then even the most consistent conjecture may yield a transition model p_θ(b'|b,a) that deviates significantly from the true model.' That admitted gap—Prop. 1 gives KL consistency of the posterior, but Theorem 1 needs total-variation closeness of transition kernels—makes the robustness guarantee conditional, a correctness risk rather than a circularity. Two items nevertheless raise the score above 0. First, Prop. 1, claimed as a contribution, is proven by deferral to the authors' earlier [21, Thm. 3] and Lemmas 6–9; this is load-bearing self-citation for that proposition, though not for Theorem 1, and [21] is peer-reviewed support. Second, the CAGE-2 'misspecification' scenario is labeled 'misspecified, i.e., ρ0(θ)=0' while Table 2 sets Θ={0,0.5,1} and the evaluation simulator fixes the true parameter at 0.5∈Θ, so mobal is correctly specified for the parameter it learns; the experiment compares an adaptive learner whose hypothesis class contains the truth with baselines given a wrong model, and it does not test the θ*∉Θ regime of Theorem 1. The illustrative example does test that regime (θ*=0.2∉Θ, Fig. 10 shows large misspecification error), which highlights the benchmark mismatch. Net: the central theoretical derivation is independent; the consistency contribution leans on the authors' prior theorem; the flagship empirical claim rests on a scenario that fails the paper's own misspecification definition. Score: 3.

Assumptions & free parameters 5 free parameters · 7 assumptions · 0 invented entities

The paper's contribution rests on several user-chosen hyperparameters (r, M, feature set, Θ) and on assumptions about the regularity of the POMDP and the smallness of α. The most unquantified is α, which determines whether the misspecification bound has any bite.

free parameters (5)
  • Quantization resolution r = 5 (illustrative); 1 (CAGE-2)
    Controls the approximation error ε in Prop. 3 and the size of the quantized MDP; set by the user, not learned.
  • Number of particles M = 50 (CAGE-2); up to 18 (illustrative)
    Trade-off between belief estimation accuracy and computational cost; M=50 is used in evaluation.
  • Discount factor γ = 0.99
    Standard discount factor used in Eq. (2) and in all bounds.
  • Conjecture set Θ = {0, 0.5, 1} for each decoy configuration
    Defines the support of the prior and the extent of misspecification; true CAGE-2 parameters are outside Θ, so the paper chooses this set rather than learning it.
  • Belief feature reduction = attacker state, next target, decoy configuration
    Reduces belief dimension to 427,500 in CAGE-2; taken from [22], affects what can be learned.
assumptions (7)
  • domain assumption The incident response problem is a POMDP with finite state, action, and observation spaces (Section 3).
    Foundation of the model; if the real system is not a finite POMDP, the formalism and bounds do not apply.
  • domain assumption The true transition model is parameterized by θ*, and the conjecture set Θ is compact with prior ρ0 having full support (Assumption 1).
    Needed for Prop. 1's consistency result; not verified for CAGE-2.
  • domain assumption Regularity conditions in Assumption 2: log-likelihood is Lipschitz in the belief w.r.t. Wasserstein distance and dominated by an integrable function.
    Assumed for Prop. 1; the paper does not check these for the cyber use case.
  • standard math The particle filter estimate converges to the true belief as M→∞ (Section 4.1).
    Law of large numbers; the paper uses M=50 and assumes the approximation is adequate.
  • standard math The optimal cost function J★ of the conjectured POMDP is uniformly continuous (used in Prop. 4, cited [60]).
    Required for the quantization to achieve asymptotic optimality.
  • ad hoc to paper The misspecified bound Eq. (10) holds for some finite α.
    The bound in Thm. 1 is only as useful as α is small; the paper neither bounds α from the learning procedure nor measures it.
  • ad hoc to paper The reduced belief features are sufficient for good decisions.
    Feature set from [22] is assumed to preserve enough information for response planning on CAGE-2; no analysis is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Online Incident Response Planning under Model Misspecification through Bayesian Learning and Belief Quantization." pith.science (2026). https://pith.science/paper/W6OOBC3A

@misc{pith2026250814385,
  author       = {Pith},
  title        = {Pith review of: Online Incident Response Planning under Model Misspecification through Bayesian Learning and Belief Quantization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W6OOBC3A}},
  note         = {Machine review of arXiv:2508.14385}
}
read the original abstract

Effective responses to cyberattacks require fast decisions, even when information about the attack is incomplete or inaccurate. However, most decision-support frameworks for incident response rely on a detailed system model that describes the incident, which restricts their practical utility. In this paper, we address this limitation and present an online method for incident response planning under model misspecification, which we call MOBAL: Misspecified Online Bayesian Learning. MOBAL iteratively refines a conjecture about the model through Bayesian learning as new information becomes available, which facilitates model adaptation as the incident unfolds. To determine effective responses online, we quantize the conjectured model into a finite Markov model, which enables efficient response planning through dynamic programming. We prove that Bayesian learning is asymptotically consistent with respect to the information feedback. Additionally, we establish bounds on misspecification and quantization errors. Experiments on the CAGE-2 benchmark show that MOBAL outperforms the state of the art in terms of adaptability and robustness to model misspecification.

Figures

Figures reproduced from arXiv: 2508.14385 by the authors.

Figure 1
Figure 1. Our method (mobal) for incident response planning under model misspecification. At each time step, we estimate a belief about the system’s security state and use it to update a conjecture about the system model through Bayesian learning. We then use this conjecture to sample a specific Markov model, whose belief space is quantized. Finally, we use the quantized model to efficiently compute an effective response thro… view at source ↗
Figure 2
Figure 2. The actors and systems involved in the incident response use case. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. mobal: an iterative method for online learning of incident response strategies under model misspecification. The figure illustrates a time step during which (i) the posterior distribution over possible system models is updated via Bayesian learning based on feedback from the system; (ii) a conjectured model is sampled from the posterior and quantized into a computationally tractable mdp; and (iii) a response strateg… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The system in the illustrative example [22]. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Observation distribution per component𝑙 in the illustrative example. The transition probabilities 𝑝𝑠𝑠′ (𝑎) are defined as follows. If com￾ponent𝑙 is compromised (𝑠 𝑙 = 1), then it remains so until recovery is applied (𝑎 𝑙 = 1), at which point the state 𝑠 𝑙 is set to 0.…
Figure 8
Figure 8. Figure 8: Comparison between the theoretical error bound in Prop. 3 and the [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 10
Figure 10. Figure 10: Comparison between the optimal cost function [PITH_FULL_IMAGE:figures/full_fig_p006_10.png]
Figure 6
Figure 6. Figure 6: Expected error of the particle filter for the illustrative example in [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Online Identification of IT Systems through Active Causal Learning

    cs.LG 2025-09 conditional novelty 4.0 of 10

    Online active causal learning with GP regression and rollout intervention selection identifies IT system causal functions with lower loss than passive monitoring.

Reference graph

Works this paper leans on

66 extracted references · 41 canonical work pages · cited by 1 Pith paper

  1. [1]

    Aliprantis and Kim C

    Charalambos D. Aliprantis and Kim C. Border. 2006.Infinite Dimensional Analysis. Springer Berlin, Heidelberg. doi:10.1007/3-540-29587-9

    doi:10.1007/3-540-29587-9 2006

  2. [2]

    Tansu Alpcan and Tamer Basar. 2010. Network Security: A Decision and Game- Theoretic Approach (1st ed.). Cambridge University Press, USA

    work page 2010

  3. [3]

    Eitan Altman, Konstantin Avrachenkov, and Andrey Garnaev. 2007. A Jamming Game in Wireless Networks with Transmission Cost. In NET-COOP

    work page 2007

  4. [4]

    Anderson

    James P. Anderson. 1980. Computer Security Threat Monitoring and Surveillance . Technical Report Contract 79F26400. James P. Anderson Co., Fort Washington, PA. Prepared for the United States Air Force

    work page 1980

  5. [5]

    Andy Applebaum, Camron Dennler, Patrick Dwyer, Marina Moskowitz, Harold Nguyen, Nicole Nichols, Nicole Park, Paul Rachwalski, Frank Rau, Adrian Webster, and Melody Wolk. 2022. Bridging Automated to Autonomous Cyber Defense: Foundational Analysis of Tabular Q-Learning. In Proceedings of the 15th ACM Workshop on Artificial Intelligence and Security . doi:10...

    work page arXiv 2022

  6. [6]

    Andy Applebaum, Shawn Johnson, Michael Limiero, and Michael Smith. 2018. Playbook Oriented Cyber Response. In 2018 National Cyber Summit (NCS) . 8–15. doi:10.1109/NCS.2018.00007

    arXiv 2018

  7. [8]

    Yahuza Bello and Ahmed Refaey Hussein. 2024. Dynamic Policy Decision/En- forcement Security Zoning Through Stochastic Games and Meta Learning. IEEE Transactions on Network and Service Management (2024), 1–1. doi:10 .1109/ TNSM.2024.3481662

    work page arXiv 2024

  8. [9]

    Robert H. Berk. 1966. Limiting Behavior of Posterior Distributions when the Model is Incorrect. The Annals of Mathematical Statistics 37, 1 (1966), 51 – 58. doi:10.1214/aoms/1177699597

    arXiv 1966

Show all 66 references
  1. [10]

    Dimitri Bertsekas. 2019. Feature-based aggregation and deep reinforcement learn- ing: A survey and some new implementations. IEEE/CAA Journal of Automatica Sinica 6, 1 (2019), 1–31. doi:10.1109/JAS.2018.7511249

    2019

  2. [11]

    Dimitri Bertsekas. 2019. Reinforcement learning and optimal control . Athena Scientific

    2019

  3. [12]

    Dimitri Bertsekas. 2025. A Course in Reinforcement Learning . Athena Scientific. 2nd edition

    2025

  4. [13]

    Levente Buttyán and Hubaux Jean-Pierre. 2001. Rational Exchange - A Formal Model Based on Game Theory. In Electronic Commerce, Ludger Fiege, Gero Mühl, and Uwe Wilhelm (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 114–126

    2001

  5. [14]

    CAGE. 2022. TTCP CAGE Challenge 2. In AAAI-22 Workshop on Artificial Intelli- gence for Cyber Security (AICS)

    2022

  6. [15]

    Alberto Castagnaro, Mauro Conti, and Luca Pajola. 2024. Offensive AI: Enhancing Directory Brute-forcing Attack with the Use of Language Models. In Proceedings of the 2024 Workshop on Artificial Intelligence and Security (Salt Lake City, UT, USA) (AISec ’24). Association for Co...

    2024

  7. [16]

    Castro, Roberto Campbell, Nancy Lau, Octavio Villalobos, Jiaqi Duan, and Alvaro A

    Sebastián R. Castro, Roberto Campbell, Nancy Lau, Octavio Villalobos, Jiaqi Duan, and Alvaro A. Cardenas. 2025. Large Language Models are Autonomous Cyber Defenders. arXiv:2505.04843 [cs.AI] https://arxiv .org/abs/2505.04843

    2025 arXiv

  8. [17]

    Yunfei Ge, Tao Li, and Quanyan Zhu. 2023. Scenario-Agnostic Zero-Trust Defense with Explainable Threshold Policy: A Meta-Learning Approach. InIEEE INFOCOM 2023 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS). 1–6. doi:10.1109/infocomwkshps57453.2023.10225816

    2023

  9. [18]

    James Cannady Georgia. 2000. Next Generation Intrusion Detection: Autonomous Reinforcement Learning of Network Attacks. InIn Proceedings of the 23rd National Information Systems Secuity Conference . 1–12

    2000

  10. [19]

    Kim Hammar, Tansu Alpcan, and Emil C. Lupu. 2025. Incident Response Plan- ning Using a Lightweight Large Language Model with Reduced Hallucination. arXiv:2508.05188 [cs.CR] https://arxiv.org/abs/2508.05188

    2025 arXiv

  11. [20]

    Kim Hammar, Neil Dhir, and Rolf Stadler. 2024. Optimal Defender Strategies for CAGE-2 using Causal Modeling and Tree Search. arXiv:2407.11070 [cs.LG] https://arxiv.org/abs/2407.11070

    2024 arXiv

  12. [21]

    Kim Hammar, Tao Li, Rolf Stadler, and Quanyan Zhu. 2025. Adaptive Secu- rity Response Strategies Through Conjectural Online Learning. IEEE Transac- tions on Information Forensics and Security 20 (2025), 4055–4070. doi:10.1109/ TIFS.2025.3558600

    2025

  13. [22]

    Lupu, and Dimitri Bertsekas

    Kim Hammar, Yuchao Li, Tansu Alpcan, Emil C. Lupu, and Dimitri Bertsekas

  14. [23]

    Kim Hammar and Rolf Stadler. 2024. Intrusion Tolerance for Networked Systems through Two-Level Feedback Control. In2024 54th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) . 338–352. doi:10.1109/ DSN58291.2024.00042

    2024

  15. [24]

    Yi Han, Tansu Alpcan, Jeffrey Chan, Christopher Leckie, and Benjamin I. P. Rubinstein. 2016. A Game Theoretical Approach to Defend Against Co-Resident Attacks in Cloud Computing: Preventing Co-Residence Using Semi-Supervised Learning. IEEE Transactions on Information Forensics...

    2016

  16. [25]

    Yi Han, Benjamin I. P. Rubinstein, Tamas Abraham, Tansu Alpcan, Olivier De Vel, Sarah Erfani, David Hubczenko, Christopher Leckie, and Paul Montague. 2018. Reinforcement Learning for Autonomous Defence in Software-Defined Network- ing. In Decision and Game Theory for Security,...

    2018

  17. [26]

    Chris Hicks, Vasilios Mavroudis, Myles Foley, Thomas Davies, Kate Highnam, and Tim Watson. 2023. Canaries and Whistles: Resilient Drone Communication Networks with (or without) Deep Reinforcement Learning. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and ...

    2023

  18. [27]

    Zequan Huang, Jacques Robin, Nicolas Herbaut, Nourhène Ben Rabah, and Bénédicte Le Grand. 2025. Toward an Intent-Based and Ontology-Driven Auto- nomic Security Response in Security Orchestration Automation and Response. arXiv:2507.12061 [cs.CR] https://arxiv.org/abs/2507.12061

    2025 arXiv

  19. [28]

    Eric M Hutchins, Michael J Cloppert, Rohan M Amin, et al. 2011. Intelligence- driven computer network defense informed by analysis of adversary campaigns and intrusion kill chains.Leading Issues in Information Warfare & Security Research 1, 1 (2011), 80

    2011

  20. [29]

    Stefano Iannucci, Qian Chen, and Sherif Abdelwahed. 2016. High-Performance Intrusion Response Planning on Many-Core Architectures. In International Con- ference on Computer Communication and Networks (ICCCN) . AISec ’25, October 13–17, 2025, Taipei, Taiwan Hammar and Li

    2016

  21. [30]

    IBM Security and Ponemon Institute. 2024. Cost of a Data Breach Report 2024 . Technical Report 19. IBM, Cambridge, MA. Based on breaches at 524 organi- zations across 17 industries in 16 countries between March 2023 and February 2024

    2024

  22. [31]

    C. T. Ionescu-Tulcea. 1949. Mesures dans les espaces produits. Atti della Ac- cademia Nazionale dei Lincei, Rendiconti Classe di Scienze Fisiche, Matematiche e Naturali 7 (1949), 208–211

    1949

  23. [32]

    Michael Kearns and Satinder Singh. 2002. Near-Optimal Reinforcement Learning in Polynomial Time. Machine Learning 49, 2 (01 Nov 2002), 209–232. doi:10.1023/ A:1017984413808

    2002

  24. [33]

    Vikram Krishnamurthy. 2016. Partially Observed Markov Decision Processes: From Filtering to Controlled Sensing . Cambridge University Press. doi:10.1017/ CBO9781316471104

    2016

  25. [34]

    Mehmet Necip Kurt, Oyetunji Ogundijo, Chong Li, and Xiaodong Wang. 2019. Online Cyber-Attack Detection in Smart Grid: A Reinforcement Learning Ap- proach. IEEE Transactions on Smart Grid 10, 5 (2019), 5174–5185. doi:10.1109/ TSG.2018.2878570

    2019

  26. [35]

    Tao Li, Kim Hammar, Rolf Stadler, and Quanyan Zhu. 2024. Conjectural Online Learning with First-order Beliefs in Asymmetric Information Stochastic Games. In 2024 IEEE 63rd Conference on Decision and Control (CDC) . 6780–6785. doi:10.1109/ CDC56724.2024.10886479

    2024

  27. [36]

    Tao Li, Yunian Pan, and Quanyan Zhu. 2024. Decision-Dominant Strategic De- fense Against Lateral Movement for 5G Zero-Trust Multi-Domain Networks. In Network Security Empowered by Artificial Intelligence . Springer Nature Switzer- land, Cham, Switzerland, 25–76. doi:10.1007/97...

    2024 doi

  28. [37]

    Tao Li, Ya-Ting Yang, Yunian Pan, and Quanyan Zhu. 2025. From Texts to Shields: Convergence of Large Language Models and Cybersecurity. arXiv preprint arXiv:2505.00841 (2025). arXiv:2505.00841

    2025 arXiv

  29. [38]

    Tao Li and Quanyan Zhu. 2025. Symbiotic game and foundation models for cyber deception operations in strategic cyber warfare. In Foundations of Cyber Deception. Springer Cham, Cham, Switzerland. [Online] Available at https: //arxiv.org/pdf/2403.10570

    2025 arXiv

  30. [39]

    Yuchao Li, Kim Hammar, and Dimitri Bertsekas. 2025. Feature-Based Belief Aggregation for Partially Observable Markovian Decision Problems. https: //arxiv.org/abs/2507.04646

    2025 arXiv

  31. [40]

    Erik Miehling, Mohammad Rasouli, and Demosthenis Teneketzis. 2018. A POMDP Approach to the Dynamic Defense of Large-Scale Cyber Networks. IEEE Transactions on Information Forensics and Security 13, 10 (2018). doi:10.1109/ TIFS.2018.2819967

    2018

  32. [41]

    Shana Moothedath, Dinuka Sahabandu, Joey Allen, Andrew Clark, Linda Bushnell, Wenke Lee, and Radha Poovendran. 2020. A Game-Theoretic Approach for Dynamic Information Flow Tracking to Detect Multistage Advanced Persistent Threats. IEEE Trans. Automat. Control 65, 12 (2020), 52...

    2020

  33. [42]

    Thanh Thi Nguyen and Vijay Janapa Reddi. 2023. Deep Reinforcement Learning for Cyber Security. IEEE Transactions on Neural Networks and Learning Systems 34, 8 (2023), 3779–3795. doi:10.1109/TNNLS.2021.3121870

    2023

  34. [43]

    Phan and Thomas Bauschert

    Trung V. Phan and Thomas Bauschert. 2022. DeepAir: Deep Reinforcement Learning for Adaptive Intrusion Response in Software-Defined Networks. IEEE Transactions on Network and Service Management (2022), 1–1. doi:10 .1109/ TNSM.2022.3158468

    2022

  35. [44]

    Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. 2021. Stable-Baselines3: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research 22, 268 (2021), 1–8. http://jmlr.org/papers/v22/20-1364.html

    2021

  36. [45]

    Catania, and Sebastian Garcia

    Maria Rigaki, Ondřej Lukáš, Carlos A. Catania, and Sebastian Garcia. 2023. Out of the Cage: How Stochastic Parrots Win in Cyber Security Environments. arXiv:2308.12086 [cs.CR]

    2023 arXiv

  37. [46]

    Naci Saldi, Serdar Yüksel, and Tamás Linder. 2017. On the Asymptotic Optimality of Finite Approximations to Markov Decision Processes with Borel Spaces. Math. Oper. Res. 42, 4 (Nov. 2017), 945–978. doi:10.1287/moor.2016.0832

    2017

  38. [47]

    Daniel Schlette, Philip Empl, Marco Caselli, Thomas Schreck, and Günther Pernul

  39. [48]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  40. [49]

    Arturo Servin and Daniel Kudenko. 2008. Multi-agent Reinforcement Learning for Intrusion Detection. In Adaptive Agents and Multi-Agent Systems III. Adaptation and Multi-Agent Learning

    2008

  41. [50]

    David Silver et al. 2016. Mastering the Game of Go with Deep Neural Networks and Tree Search. Nature 529, 7587 (Jan. 2016), 484–489. doi:10.1038/nature16961

    2016 doi

  42. [51]

    David Silver and Joel Veness. 2010. Monte-Carlo Planning in Large POMDPs. In Advances in Neural Information Processing Systems , Vol. 23

    2010

  43. [52]

    Rock Stevens, Daniel Votipka, Josiah Dykstra, Fernando Tomlinson, Erin Quar- tararo, Colin Ahern, and Michelle L. Mazurek. 2022. How Ready is Your Ready? Assessing the Usability of Incident Response Playbook Frameworks. In Proceed- ings of the 2022 CHI Conference on Human Fact...

    2022

  44. [53]

    Tsitsiklis and Benjamin van Roy

    John N. Tsitsiklis and Benjamin van Roy. 1996. Feature-based methods for large scale dynamic programming. Machine Learning 22, 1 (01 Mar 1996), 59–94. doi:10.1007/BF00114724

    1996 doi

  45. [54]

    Sanyam Vyas, John Hannay, Andrew Bolton, and Professor Pete Burnap. 2023. Automated Cyber Defence: A Review. arXiv:2303.04926 [cs.CR] https://arxiv.org/ abs/2303.04926, code: https://github.com/john-cardiff/-cyborg-cage-2

    2023 arXiv

  46. [55]

    Stephen Walker and Nils Lid Hjort. 2001. On Bayesian consistency.Journal of the Royal Statistical Society: Series B (Statistical Methodology) 63, 4 (2001), 811–821. doi:10.1111/1467-9868.00314

    2001

  47. [56]

    Anwar, Charles A

    Zelin Wan, Jin-Hee Cho, Mu Zhu, Ahmed H. Anwar, Charles A. Kamhoua, and Munindar P. Singh. 2022. Foureye: Defensive Deception Against Advanced Persistent Threats via Hypergame Theory. IEEE Transactions on Network and Service Management 19, 1 (2022), 112–129. doi:10.1109/TNSM.2...

    2022

  48. [57]

    Wazuh Inc. 2022. Wazuh - The Open Source Security Platform. https: //wazuh.com/

    2022

  49. [58]

    Melody Wolk, Andy Applebaum, Camron Dennler, Patrick Dwyer, Marina Moskowitz, Harold Nguyen, Nicole Nichols, Nicole Park, Paul Rachwalski, Frank Rau, and Adrian Webster. 2022. Beyond CAGE: Investigating Generalization of Learned Autonomous Network Defense Policies. doi:10.4855...

    2022 doi arXiv

  50. [59]

    Xin Xu and Tao Xie. 2005. A Reinforcement Learning Approach for Host-Based Intrusion Detection Using Sequences of System Calls. In Advances in Intelligent Computing

    2005

  51. [60]

    Huizhen Yu. 2006. Approximate solution methods for partially observable markov and semi-markov decision processes . Ph. D. Dissertation. Massachusetts Institute of Technology, USA. Advisor(s) Bertsekas, Dimitri

    2006

  52. [61]

    Huizhen Yu and Dimitri Bertsekas. 2004. Discretized approximations for POMDP with average cost. InProceedings of the 20th Conference on Uncertainty in Artificial Intelligence (Banff, Canada) (UAI ’04). AUAI Press, Arlington, Virginia, USA, 619–627

    2004

  53. [62]

    Lefeng Zhang, Tianqing Zhu, Farookh Khadeer Hussain, Dayong Ye, and Wanlei Zhou. 2022. Defend to Defeat: Limiting Information Leakage in Defending against Advanced Persistent Threats. IEEE Transactions on Information Forensics and Security (2022), 1–1. doi:10.1109/TIFS.2022.3229595

    2022

  54. [63]

    Zonouz, Himanshu Khurana, William H

    Saman A. Zonouz, Himanshu Khurana, William H. Sanders, and Timothy M. Yardley. 2009. RRE: A game-theoretic intrusion Response and Recovery Engine. In 2009 IEEE/IFIP International Conference on Dependable Systems & Networks . 439–448. doi:10.1109/DSN.2009.5270307

    2009

  55. [64]

    Karl Johan Åström. 1965. Optimal control of Markov processes with incomplete state information. J. Math. Anal. Appl. 10, 1 (1965), 174–205. doi:10.1016/0022- 247X(65)90154-X A Experimental Setup All computations are performed on an m2-ultra processor. The hy- perparameters are...

    1965 doi

  56. [2017]

    CoRR (2017)

    Proximal Policy Optimization Algorithms. CoRR (2017). arXiv:1707.06347 http://arxiv.org/abs/1707.06347 http://arxiv.org/abs/1707.06347

    2017 arXiv

  57. [2024]

    In 2024 IEEE Symposium on Security and Privacy (SP)

    Do You Play It by the Books? A Study on Incident Response Playbooks and Influencing Factors. In 2024 IEEE Symposium on Security and Privacy (SP) . 3625–3643. doi:10.1109/SP54263.2024.00060

    2024

  58. [2025]

    https://arxiv.org/abs/2507.15163

    Adaptive network security policies via belief aggregation and rollout. https://arxiv.org/abs/2507.15163

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.