{"id":"d3a2dee1-6576-4486-a8c0-f4e122c5b493","arxiv_id":"2412.02875","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A probabilistic neural network monitors a reinforcement-learning cyber-defense agent and flags transitions never or rarely seen during training as out-of-distribution.","lead":"This paper adds an out-of-distribution monitor to an RL-based cyber-defense agent, flagging state transitions that were rarely seen during training. The monitor is tested in a simulated network defense game and detects when the attacker switches strategy, triggering a safe reset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The safety claim is unsupported because ρ=0 makes the OOD detector a pure novelty detector: any transition seen at least once in training is in-distribution, so dangerous transitions that occurred during training cannot be flagged, and rare safe transitions trigger false alarms; no false-negative…","rationale":"The paper's stated contribution is an OOD monitor for discrete-state RL agents, integrated into an EBT, and the conclusion claims it 'effectively detects OOD situations under all settings and hence can assure safety.' I read the argument in good faith: the monitor is a simple and plausible novelty detector based on empirical transition frequencies, and the strategy-switch experiments do show that transitions generated by an unseen adversarial policy are flagged. The load-bearing step, however, is the identification of 'low training probability' with 'unsafe situation.' That step is weakest at ρ=0, the value actually used. The paper's own Table I shows the detector is extremely sensitive to the threshold, yet the threshold is selected on the test episodes with no held-out validation. The internal justification in Section V-B speaks of 'significantly large number of times,' but ρ=0 admits any single occurrence, so the threshold choice is not supported by the stated rationale. The reader's weakest_assumption identifies the same issue—the unvalidated premise that low empirical probability means unsafe. My concrete test would settle it: if in-distribution but safety-critical transitions are flagged rarely or never, then the central safety claim fails. I do not see this as a novelty or consensus problem; it is a correctness risk in the argument from Definition 1 to Section VIII. Because the reader already marked the paper CONDITIONAL on essentially this concern, my read does not require changing the verdict; it sharpens the condition that must be met.","tokens_in":11548,"tokens_out":3926,"duration_ms":62269,"concrete_test":"Run a held-out false-negative test. Train the PNN on episodes 1–9000 (or a random 90% split), reserving the remaining episodes. From the reserved set, label transitions by a safety-relevant outcome—e.g., whether the next state contains a compromised operational server or whether the episode reward falls below a threshold. Restrict to reserved transitions whose (s_{t-1},a_{t-1},s_t) triple also appears at least once in the training set. Feed these through Algorithm 1 with ρ=0 and count the fraction flagged OOD. If that fraction is not zero or near-zero, the monitor fails to flag safety-critical transitions that are in-distribution, directly refuting the 'assure safety' claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Definition 1 defines OOD by Pr((s_{t-1},a_{t-1})→s_t) < ρ, and Section VII-B sets ρ=0 after Table I shows ρ=10^{-5} flags all 1000 episodes. With ρ=0, Algorithm 1's condition is equivalent to 'the transition count in D_train is zero': any transition observed even once is accepted, regardless of how rarely or how unsafe it is. This contradicts the motivation in Section V-B that confidence requires the transition to occur 'a significantly large number of times.' The central safety claim in Section VIII therefore rests on an unvalidated equivalence between novelty and unsafety. Two concrete failure modes follow. (1) False negatives: an unsafe transition that occurred in training—e.g., an Impact action that compromises the operational server but was part of the training episodes—has positive empirical probability and will never be flagged. Such transitions are arguably the most safety-critical, and the paper provides no evidence about them. (2) False positives: any safe but rare transition absent from D_train is flagged and triggers GetSafeAction!, which restores a host/server to a safe state and may itself disrupt operations; with ρ=0 there is no way to distinguish a genuinely dangerous novel transition from a benign one merely absent from a finite training set. No held-out validation is reported for the threshold choice, and no precision, recall, coverage, or ground-truth OOD labels are given. Thus the experimental demonstrations that strategy switches are detected only show that unseen transitions are flagged—a property of any novelty detector—not that the monitor assures safety.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an out-of-distribution (OOD) monitoring algorithm for reinforcement-learning-based agents with discrete states and discrete actions. The algorithm learns transition probabilities from training data using a probabilistic neural network (PNN), and at runtime flags a transition (s_{t-1}, a_{t-1}) -> s_t as OOD if its estimated training probability is below a threshold rho. The authors integrate this monitor into an evolving behavior tree (EBT) cyber-defense agent and evaluate it in the CybORG CAGE Challenge Scenario 2 simulator under Meander, Blines, and RedSwitch adversarial strategies. The main claims are that the algorithm detects OOD situations under these settings, that the added GetSafeAction! behavior restores safety after a strategy switch, and that the approach 'can assure safety' in general.","tokens_in":11817,"tokens_out":3349,"duration_ms":37210,"significance":"The problem is well motivated: autonomous cyber-defense agents trained in simulation need runtime monitors that detect situations outside their training distribution and hand them to human experts. The proposed method is simple and, for the discrete-state setting, differentiates this work from continuous-state OOD detectors for RL. The strategy-switch experiments, especially the comparison with and without GetSafeAction!, provide a concrete and suggestive demonstration that a monitor can trigger a recovery action after an adversary changes behavior. However, the paper's central safety claim is not supported by the evidence presented. There is no ground-truth OOD labeling, no false-positive or false-negative analysis, no coverage or generalization analysis, and no baseline comparison. The choice rho=0 turns the detector into a pure novelty detector, which may be a reasonable starting point for detecting distribution shift but is not by itself a safety guarantee. The manuscript also ships a complete algorithm description and experimental setup, which is helpful for reproducibility, although no code is provided.","major_comments":[{"comment":"The threshold rho is selected from the test data after observing that rho=1e-5 flags 1000 out of 1000 episodes for Meander and 782 out of 1000 for Blines; the paper then sets rho=0 for all subsequent experiments. This is a post hoc choice on the evaluation set, with no held-out validation, no theory, and no sensitivity analysis. Because rho=0 makes Algorithm 1's condition equivalent to 'the transition count in D_train is zero', any transition observed even once is considered in-distribution. This directly contradicts the motivation in Section V-B that confidence requires a transition to occur 'a significantly large number of times.' The authors should either justify rho=0 with a separate validation procedure, report precision/recall over a range of rho on held-out data, or both.","section":"Section VII-B, Table I"},{"comment":"The conclusion states that the proposed algorithm 'effectively detects OOD situations under all settings and hence can assure safety,' but the paper provides no evidence that low empirical transition probability is equivalent to unsafety. Two concrete failure modes are not addressed: (1) false negatives, where a dangerous transition that occurred in training has positive empirical probability and will never be flagged; and (2) false positives, where a benign rare transition absent from the finite training set is flagged and triggers GetSafeAction!, possibly disrupting operations. The problem statement asks 'Can we assure safety if the system transitions to any state s' such that Pr((s,a)->s') < rho?', but no answer in the form of a guarantee, bound, or even an empirical safety metric is given. The authors should either provide a rigorous safety argument with explicit assumptions or replace the safety-assurance claim with a more limited claim about detecting transitions unseen in training.","section":"Section VIII and Problem Statement in Section V-A"},{"comment":"The evaluation of OOD episodes is circular: Definition 2 defines an episode as OOD if it contains a transition with Pr < rho, where Pr is the same estimated transition probability used by Algorithm 1. Thus the reported detection counts (15/1000 for Meander and 1/1000 for Blines at rho=0) are largely a restatement of the threshold definition rather than an independent measure of detection quality. The strategy-switch experiments provide a more independent grounding because the RedSwitch scenario uses a held-out red strategy, but these results are reported only qualitatively for five episodes and through aggregate counts in Figures 8-10. The authors should quantify detection delay, false-alarm rate, and accuracy against a labeled set of strategy-switch episodes, and should report results over multiple random seeds with error bars.","section":"Section VII-B, Definition 2"},{"comment":"The experimental methodology lacks several standard elements needed to support the paper's claims. No baseline detector is compared (for example, a simple empirical frequency table, a likelihood-based method, or an existing OOD detector for RL); no variance or confidence intervals are reported despite the 1000-episode evaluation; and no separate train/test split for the threshold choice is described. Given the central role of the safety claim, the absence of false-positive and false-negative rates, and the absence of any measure of coverage of the state space by D_train, make it impossible to distinguish the proposed method's behavior from a naive count-based novelty detector. Adding baselines, multiple seeds, and precision/recall or ROC-style evaluation would substantially strengthen the paper.","section":"Section VII, Experimental Evaluation"}],"minor_comments":[{"comment":"The PNN input layer is described as having size 1, but the input is the pair (s_{t-1}, a_{t-1}); the figure and text should clarify how the state-action pair is encoded and matched against the pattern layer.","section":"Section V-B, Algorithm 1 and Figure 3"},{"comment":"The red agent names 'Meander' and 'Blines' are typeset with an extra space in several places (e.g., 'M eander' and 'B line'); this should be corrected for readability.","section":"Throughout"},{"comment":"The text states that a transition is considered in-distribution if the associated probability is 'greater than' rho, while Algorithm 1 uses '>' in the condition; if rho=0, an observed transition with probability exactly 0 is OOD, which is consistent, but the strictness should be stated consistently and the case of probability exactly rho should be addressed explicitly.","section":"Section V-B, paragraph after Definition 1"},{"comment":"The in-text citation for the Haider et al. work on probabilistic dynamics models appears as [20] in Section V-B, but reference [19] in the bibliography appears to be the Haider et al. AAMAS 2023 paper; the citation numbering should be checked and corrected.","section":"References"},{"comment":"The captions refer to median values 'marked in red' in the reward-distribution plots; please ensure the figures actually display these marks clearly and describe the box-plot or violin-plot semantics in the captions.","section":"Figures 6 and 7"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely application of OOD detection to neurosymbolic cyber agents, and the integration of the monitor with the behavior tree is a nice system contribution. My main concern is that the safety claim in Section VIII goes well beyond what the experiments establish, and the rho=0 choice makes the detector equivalent to a novelty detector based on finite training counts. This is not, by itself, a fatal flaw: the strategy-switch experiments suggest the approach can be useful. However, the manuscript needs a substantially strengthened evaluation (baselines, error bars, held-out validation of rho, and a clear separation between novelty detection and safety) and a reworded conclusion before it is suitable for publication. I would not recommend rejection because the core idea is reasonable and the defects are fixable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a look for the system integration, but the central safety claim does not hold up. The authors adapt model-based OOD detection to discrete-state RL agents using a PNN to estimate transition probabilities, and they integrate this monitor into an evolving behavior tree with a GetSafeAction! fallback. That is a sensible, concrete engineering contribution, and they evaluate it in CybORG CAGE 2 with a substantial training dataset (10,000 episodes, 27 hours). The strategy-switch experiments are suggestive: the monitor does flag unseen attacker behavior, and the GetSafeAction behavior appears to reduce the number of OOD transitions per episode. That part is legitimate. The problem is the definition of OOD and the threshold choice. Definition 1 says a transition is OOD if its training probability is less than rho. Section VII-B then sets rho=0, after observing that rho=1e-5 flags nearly everything. With rho=0, the condition reduces to \"this transition was never seen in training.\" That makes the detector a pure novelty detector, not a safety monitor. Two failure modes follow. First, false negatives: a dangerous transition that happened to appear in training, say an Impact action that compromised the operational server, has positive empirical probability and will never be flagged, even though it is arguably the most safety-critical event. Second, false positives: any safe but rare transition absent from the finite training set triggers Restore, which can itself disrupt operations. The paper provides no precision, recall, coverage, or independent ground truth for OOD labels, and the threshold is selected after looking at the test results. The conclusion that the algorithm \"can assure safety\" is simply not supported by the evidence. That said, the flaws are fixable. The authors could add baselines, report error bars over seeds, and justify the threshold on held-out data. More importantly, they need to either decouple the safety claim from the novelty detection or demonstrate that unseen transitions actually correlate with unsafe states. As it stands, the paper is a decent system demonstration, but it overclaims. I would send it to peer review because the direction matters and the integration is real, but the referee should push hard on the evaluation methodology and the safety language. The core idea is not wrong, just unvalidated in its current form.","headline":"Plausible system demo for discrete-state OOD detection in cyber defense, but the safety-assurance claim is unsupported because the chosen threshold makes the detector a pure novelty detector.","tokens_in":693,"tokens_out":838,"would_cite":false,"duration_ms":23752,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A transition-probability monitor can flag out-of-distribution states and adversarial strategy switches in discrete-state discrete-action RL cyber agents at runtime.","keywords":["out-of-distribution detection","probabilistic neural network","reinforcement learning","cyber defense","neurosymbolic AI","behavior trees","runtime safety assurance","discrete state-action spaces"],"falsifier":"Run the monitor with $\\rho = 0$ on a fresh held-out set of episodes from the same environment and policy used to generate the training data, and count flagged transitions: a non-negligible flag rate would show the detector is responding to finite-sample rarity rather than true distribution shift. Alternatively, inject a harmful transition that was present in the training data; the monitor will label it in-distribution by construction.","tokens_in":11343,"feed_emoji":"🛡️","tokens_out":8140,"duration_ms":72371,"temperature":0.7,"pith_summary":"This paper argues that a reinforcement-learned cyber-defense agent can be made safer at runtime by checking whether each state transition it takes was observed often enough during training. The proposed monitor trains a Probabilistic Neural Network (PNN) on recorded transitions $(s_{t-1},a_{t-1}) \\to s_t$, then flags any current state whose estimated transition probability falls below a threshold $\\rho$ as out-of-distribution. The claim is that this single mechanism detects anomalous situations and adversarial strategy switches, and that coupling it with a restore action inside the agent's behavior tree turns detection into a concrete safety response. A sympathetic reader would care because the monitor is policy-agnostic for agents with discrete states and discrete actions, and it does not require retraining the defender.","feed_headline":"Rare-transition check flags unsafe cyber states at runtime","feed_subtitle":"The monitor learns which state transitions are normal from training data and restores the defender when a switch appears.","key_machinery":"The load-bearing object is a Probabilistic Neural Network used as a nonparametric empirical transition model. For a query $(s_{t-1},a_{t-1})$, the pattern layer activates only training examples with that same previous state and action, and the summation layer outputs a probability distribution over the distinct successor states seen in training; the usual fourth decision layer is removed. The companion decision rule is the Transition Probability Threshold $\\rho$ of Definition 1: a transition is in-distribution only if the successor appears among the PNN's predictions with probability greater than $\\rho$. Together they convert memory of training rollouts into a per-timestep safe/OOD verdict.","core_discovery":"The central claim is that empirical transition frequency is a sufficient signal for runtime out-of-distribution detection in discrete RL agents. Definition 1 formalizes this: a transition $(s_{t-1},a_{t-1}) \\to s_t$ is out-of-distribution under policy $\\pi$ when its training-data probability is less than the threshold $\\rho$. The PNN learns that distribution by storing every training transition and activating only pattern units whose previous state and action match the current input, so the output layer gives probabilities over observed successor states. In the CybORG CAGE Challenge Scenario 2 evaluation, the monitor with $\\rho = 0$ flags 1.5% of episodes against the Meander adversary and 0.1% against Bline, detects a Meander-to-Bline strategy switch at the next timestep, and the added GetSafeAction restore node cuts OOD transitions per episode to between 0 and 2. The paper concludes that the algorithm detects OOD situations under all settings and hence assures safety.","pith_inferences":["The paper equates 'absent from training data' with 'unsafe', which is a coverage assumption rather than a demonstrated fact; a safe-but-rare transition would be flagged OOD, while a harmful transition that happened to appear in training would pass as in-distribution. Pairing the monitor with independent safety labels or a coverage analysis would test this directly.","The threshold selection is fragile: Table I shows that $\\rho = 10^{-5}$ flags essentially all episodes, so the practical operating point is exactly zero-probability transitions. Recomputing the table with more training data would reveal whether the 0.1-1.5% OOD rate is stable or an artifact of finite sampling.","The same transition-frequency idea could be extended toward continuous-state agents by discretizing states or using a density estimator, although the paper explicitly restricts its claim to discrete states and discrete actions."],"forward_implications":["Any discrete-state, discrete-action RL policy can be wrapped with this monitor without modifying the policy itself, since the PNN is trained on observed transitions rather than on policy parameters.","An adversarial strategy switch is detectable within one timestep because the new strategy produces transitions that were absent or rare in the training distribution of the old policy.","Executing a restore action upon OOD detection lets the agent return to a previously safe state and switch to the control policy trained for the new adversary.","With the selected threshold $\\rho=0$, the monitor is highly conservative: it flags only transitions that never appeared in 10,000 training episodes, yielding 15 OOD episodes out of 1000 for Meander and 1 out of 1000 for Bline.","The PNN remains tractable in this setting because only a small subset of the 52-bit state space is reachable, keeping the output layer small."],"supporting_citations":[{"why":"supplies the Probabilistic Neural Network architecture used to learn empirical transition probabilities.","marker":"[6]"},{"why":"contributes the EBT-based cyber-defense agent and the RedSwitch strategy the monitor is integrated with and tested against.","marker":"[5]"},{"why":"provides the CybORG CAGE Challenge Scenario 2 simulation environment used for data generation and evaluation.","marker":"[7]"},{"why":"the model-based OOD detection framework for RL agents that this work adapts to discrete, partially observable states.","marker":"[20]"},{"why":"formalizes the POMDP model that frames the state-action-transition problem.","marker":"[23]"},{"why":"provides the PyTrees behavior-tree library used to embed the OOD monitoring nodes in the agent.","marker":"[24]"}],"fun_headline_variants":["OOD monitor catches strategy shifts via transition rarity","Neurosymbolic cyber agent detects OOD via PNN","Empirical transition odds trigger expert handoff","Rare state switch alerts RL cyber defender","PNN learns normal flows to flag unsafe transitions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The monitor assumes that any state transition that never (or rarely) appeared in the finite training dataset is genuinely out-of-distribution and unsafe, and that any transition that did appear is safe.","fun_headline_variants_meta":{"raw":{"variants":["OOD monitor catches strategy shifts via transition rarity","Neurosymbolic cyber agent detects OOD via PNN","Empirical transition odds trigger expert handoff","Rare state switch alerts RL cyber defender","PNN learns normal flows to flag unsafe transitions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000246,"raw_usage":{"total_tokens":1535,"prompt_tokens":937,"completion_tokens":598,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":526}},"tokens_in":553,"tokens_out":598,"duration_ms":6896,"temperature":1.0,"reasoning_tokens":526,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:00:23.397275+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the monitor with $\\rho = 0$ on a fresh held-out set of episodes from the same environment and policy used to generate the training data, and count flagged transitions: a non-negligible flag rate would show the detector is responding to finite-sample rarity rather than true distribution shift. Alternatively, inject a harmful transition that was present in the training data; the monitor will label it in-distribution by construction.","supporting_citations":[{"cited_title":"Anomaly detection in thermal power plant using probabilistic neural network,","cited_arxiv_id":null,"evidence_quote":"supplies the Probabilistic Neural Network architecture used to learn empirical transition probabilities."},{"cited_title":"Design- ing robust cyber-defense agents with evolving behavior trees,","cited_arxiv_id":null,"evidence_quote":"contributes the EBT-based cyber-defense agent and the RedSwitch strategy the monitor is integrated with and tested against."},{"cited_title":"Cyber autonomy gym for experimentation challenge 2","cited_arxiv_id":null,"evidence_quote":"provides the CybORG CAGE Challenge Scenario 2 simulation environment used for data generation and evaluation."},{"cited_title":"Rethinking out-of-distribution detection for reinforcement learning: Advancing methods for evaluation and detection,","cited_arxiv_id":null,"evidence_quote":"the model-based OOD detection framework for RL agents that this work adapts to discrete, partially observable states."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"formalizes the POMDP model that frames the state-action-transition problem."},{"cited_title":"Pytrees","cited_arxiv_id":null,"evidence_quote":"provides the PyTrees behavior-tree library used to embed the OOD monitoring nodes in the agent."}],"review_version":1}