pith. sign in

arxiv: 2606.12896 · v2 · pith:T6K6YKALnew · submitted 2026-06-11 · 💻 cs.LG · cs.AI· cs.CR

PolicyGuard: Towards Test-time and Step-level Adversary (Backdoor) Defense for Reinforcement Learning Agent

Junfeng Guo Heng Huang This is my paper

Pith reviewed 2026-06-27 07:22 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.CR
keywords backdoor defensereinforcement learningtest-time defensestep-level detectionGaussian Processadversarial attackspolicy securityuncertainty estimation
0
0 comments X

The pith

PolicyGuard detects backdoor triggers in reinforcement learning agents at individual time steps during testing without internal model access.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents PolicyGuard as a defense for RL agents against backdoor attacks, where the agent acts normally until a trigger appears and then behaves maliciously. It operates at test time and at the level of single time steps by adapting pseudo trajectories and measuring uncertainty via Gaussian Process posterior variance. This addresses limitations in prior defenses that need parameter access, work only on full trajectories or models, or handle only certain attack types. A sympathetic reader would care because RL systems are deployed in real applications where undetected backdoors could lead to harmful actions, and step-level detection allows targeted responses rather than blanket rejection. The authors supply theoretical reasons why the variance measure reveals triggers and report strong detection results across multiple games and attack styles.

Core claim

PolicyGuard is a test-time step-level backdoor defense for RL agents that leverages Gaussian Process posterior variance on adapted pseudo trajectories to enable uncertainty computation for individual time steps, with theoretical foundations to explain the efficacy of this measure, and achieves state-of-the-art detection performance in most cases with average AUROC of 0.856 for perturbation-based attacks and 0.859 for adversary-agent attacks across seven RL games.

What carries the argument

Gaussian Process posterior variance on adapted pseudo trajectories, used to flag backdoor triggers at specific time steps.

If this is right

  • RL agents can be defended at deployment time without requiring access to internal parameters or retraining.
  • Detection occurs at the granularity of single time steps rather than entire trajectories or models.
  • The same approach applies to both perturbation-based attacks and adversary-agent attacks.
  • Theoretical analysis supports why posterior variance serves as an effective indicator for this task.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The method could be combined with existing training-time defenses to create layered protection for RL systems.
  • It might apply to detecting other forms of adversarial behavior in sequential decision-making beyond backdoors.
  • Deployment in safety-critical RL domains such as robotics would benefit from the step-level granularity for immediate intervention.

Load-bearing premise

Gaussian Process posterior variance on adapted pseudo trajectories reliably indicates the presence of backdoor triggers at individual time steps.

What would settle it

A set of test trajectories containing known backdoor triggers where the computed Gaussian Process posterior variance shows no consistent elevation or correlation with trigger presence.

Figures

Figures reproduced from arXiv: 2606.12896 by Junfeng Guo Heng Huang.

Figure 1
Figure 1. Figure 1: Overview of PolicyGuard. Our approach consists of two phases. In the first phase (Training GP Model), we collect state-action pairs across multiple episodes and map them to embedding features ht using a recurrent neural network. A shallow MLP then projects the final state embedding h (i) T to e (i) for each episode X (i) . These embeddings serve as inputs to a Gaussian Process (GP) model, which is trained … view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of uncertainty scores (adaptive Gaussian posterior variance) for backdoor and benign time steps across various games. Backdoor attacks use default settings. The distributions for additional games are included in the Appendix [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The inference latency of various defense approaches. NC and PolicyCleanse are excluded as their computational cost is dominated by model-level reverse engineering, making them inapplicable to step-level test-time latency evaluation. perturbation-based, 0.878 for adversary-agent attacks), out￾performing all baselines including SHINE. These results highlight that trajectory-modeling-based defenses offer su￾p… view at source ↗
Figure 4
Figure 4. Figure 4: Ablation Study. (a) The effect of patch size to our approach’s detection efficacy for perturbation-based attacks. Across each evaluated environment and backdoor trigger size, we train three different backdoor models to compare their uncertainty with the corresponding benign model’s. (b) The effect of performed trigger actions’ length for adversary-agent attacks. (c) The effect of size of pseudo trajectorie… view at source ↗
Figure 5
Figure 5. Figure 5: The Performance, Attack Efficacy, and Uncertainty U under varying poisoning rates for our considered adaptive attack. Winning Rate measures Performance (trigger absent) and Attack Efficacy (trigger present); while U measures the detection efficacy. mance across all evaluated environments. Pong consistently achieves the highest AUROC, followed by Breakout, RTGA, and YSNP. When the trajectory size is smaller… view at source ↗
Figure 6
Figure 6. Figure 6: The Benign and Backdoor-infected Frame for Pong Game. The left pair (a-b) represents the original colorful frames, while the right pair (c-d) represents the (pre-processed) gray-scale frames. For the squared exponential (SE) kernel kγ(h, h′ ) = exp  − ∥h−h ′∥ 2 2γ2  , the diagonal entry of the posterior variance at time t can be decomposed as: Σ (i) t = kγ(ht, ht) − k ⊤ t K −1 ZZkt + k ⊤ t K −1 ZZΣK −1 Z… view at source ↗
Figure 7
Figure 7. Figure 7: The Benign and Backdoor-infected Frame for Breakout Game. The left pair (a-b) represents the colorful frames, while the right pair (c-d) represents the (pre-processed) gray-scale frames. Ori. (a) Backdoor. (b) Ori. (c) Backdoor. (d) [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The Benign and Backdoor-infected Frame for SpaceInvaders Game. The left pair (a-b) represents the colorful frames, while the right pair (c-d) represents the (pre-processed) gray-scale frames [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The visual demonstration of trigger actions performed by the adversary agent. The top row represent the benign agent’s behavior. From left to right, each game is Run-to-Goal-Ants, Run-to-Goal-Humans, You-Shall-Not-Pass, Sumo. The bottom row represents the corresponding trigger actions performed the trigger agents (Blue agent for Ants, Red agent for the humans.) The trigger actions are randomly sampled with… view at source ↗
Figure 10
Figure 10. Figure 10: The distributions for Posterior Gaussian Variance across different games 5K 10K 15K 20K 25K 30K Training Size 0.65 0.70 0.75 0.80 0.85 0.90 AUROC Pong YSNP [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: The Effect of Training Data Size H.5. A Closer Look at the Effectiveness of PolicyGuard We here follow (Guo et al., 2023a) to draw the distributions of benign, backdoor and inducing points Z’s hidden features. We can see that in the tSNE clustering approach, benign and inducing Z points can be easily separated with backdoor points. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: The tSNE clusters for benign, backdoor and Z points. H.6. Adaptive Perturbation-based Attacks We also consider an adaptive attack for Perturbation-based Attack, we perform adaptive attack by fixing a random action as the target action and perform reverse engineering as NC (Wang et al., 2019), to optimize a trigger pattern which makes a pre-trained GP variance minimum, as follows: ∆ = Ex∼X[arg min ∆ Σ(x ⊙ … view at source ↗
read the original abstract

While real-world applications of reinforcement learning (RL) are becoming increasingly popular, the security of RL systems deserve more attention and exploration. In particular, recent work has revealed that RL agents are vulnerable to backdoor attacks, where a victim agent behaves normally under standard conditions but executes malicious actions when a specific trigger is activated. Existing backdoor defenses for RL either require access to the agent's internal parameters, operate only at the model or trajectory level, or are limited to specific attack types. To ensure the security of RL agents, we propose \texttt{PolicyGuard}, a \textit{test-time step-level} backdoor defense which leverages Gaussian Process (GP) posterior variance and adapts pseudo trajectories to enable uncertainty computation for individual time step. Besides, we also provide theoretical foundations to explain the efficacy of GP posterior variance. Extensive experiments across seven RL games demonstrate that PolicyGuard achieves state-of-the-art detection performance in most cases, with average AUROC of 0.856 for perturbation-based attacks and 0.859 for adversary-agent attacks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes PolicyGuard, a test-time step-level backdoor defense for RL agents. It adapts pseudo trajectories and computes Gaussian Process (GP) posterior variance to detect backdoor triggers at individual time steps, provides theoretical foundations explaining the efficacy of this variance measure, and reports state-of-the-art detection across seven RL games (average AUROC 0.856 for perturbation-based attacks and 0.859 for adversary-agent attacks).

Significance. If the core assumptions hold, this would be a meaningful advance for RL security: it operates at test time without internal parameter access, provides per-step granularity, and addresses a broader range of attacks than prior defenses. Explicit provision of theoretical foundations is a positive feature that attempts to ground the detection mechanism.

major comments (2)
  1. [Method and theoretical foundations sections] The central claim depends on the adaptation procedure (described in the method) producing pseudo trajectories whose distribution matches the clean policy except precisely at trigger locations, and on the GP posterior variance (with the chosen kernel) reflecting epistemic uncertainty induced by the trigger rather than model mismatch or aleatoric effects. The theoretical foundations section does not derive the variance spike under explicitly stated MDP/reward assumptions, leaving the per-step detection claim vulnerable to the three failure modes noted in the stress test; this is load-bearing because the reported AUROCs rest on this mechanism working as described.
  2. [Experiments section] Experiments section: the average AUROCs are presented as evidence of SOTA performance, yet no ablation or sensitivity analysis is shown for the adaptation procedure or GP hyperparameters. Without these, it is unclear whether the results support the general claim or are tied to the specific seven games and attack implementations.
minor comments (2)
  1. [Method section] Notation for the adapted pseudo trajectories and the exact form of the GP posterior variance should be introduced with a single consistent symbol set to aid readability.
  2. [Abstract and title] The abstract and title use slightly varying phrasing for 'adversary (Backdoor)'; standardize terminology throughout.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. We address each major comment below and describe the revisions we will make to strengthen the manuscript.

read point-by-point responses

  1. Referee: [Method and theoretical foundations sections] The central claim depends on the adaptation procedure (described in the method) producing pseudo trajectories whose distribution matches the clean policy except precisely at trigger locations, and on the GP posterior variance (with the chosen kernel) reflecting epistemic uncertainty induced by the trigger rather than model mismatch or aleatoric effects. The theoretical foundations section does not derive the variance spike under explicitly stated MDP/reward assumptions, leaving the per-step detection claim vulnerable to the three failure modes noted in the stress test; this is load-bearing because the reported AUROCs rest on this mechanism working as described.

    Authors: We agree that the theoretical foundations would benefit from greater explicitness. In the revised manuscript we will add a dedicated subsection that states the MDP and reward assumptions under which the GP posterior variance is guaranteed to spike at trigger locations (while remaining low elsewhere). We will also include a short discussion of the three failure modes referenced in the stress test, with arguments showing why the adaptation procedure and kernel choice mitigate them under the stated assumptions. A more detailed derivation of the variance expression will be provided in the appendix. revision: yes

  2. Referee: [Experiments section] Experiments section: the average AUROCs are presented as evidence of SOTA performance, yet no ablation or sensitivity analysis is shown for the adaptation procedure or GP hyperparameters. Without these, it is unclear whether the results support the general claim or are tied to the specific seven games and attack implementations.

    Authors: We acknowledge that the current experiments lack ablations. In the revision we will add (i) sensitivity plots for the main GP hyperparameters (length-scale, output-scale, and noise variance) and (ii) ablation results on the adaptation procedure parameters (number of pseudo-trajectories and adaptation steps). We will also report AUROC on two additional environments to support the claim of generality beyond the original seven games. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation relies on standard GP techniques

full rationale

The provided abstract and context describe PolicyGuard as leveraging established Gaussian Process posterior variance on adapted pseudo trajectories, with theoretical foundations claimed for its efficacy. No equations or sections in the visible text reduce the central detection claim to a self-definition, a fitted parameter renamed as prediction, or a load-bearing self-citation chain. The approach builds on independent GP properties and external RL benchmarks rather than deriving its key performance metric by construction from its own inputs. This is the expected self-contained case for a method paper using a standard statistical tool.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Insufficient details in abstract to identify specific free parameters, axioms, or invented entities.

pith-pipeline@v0.9.1-grok · 5713 in / 1094 out tokens · 20716 ms · 2026-06-27T07:22:36.602674+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 8 canonical work pages · 4 internal anchors

  1. [1]

    Forty-first International Conference on Machine Learning , year=

    SHINE: Shielding backdoors in deep reinforcement learning , author=. Forty-first International Conference on Machine Learning , year=

  2. [2]

    Jensen's inequality , author=

  3. [3]

    Uber die abgrenzung der eigenwerte einer matrix , author=

  4. [4]

    Posterior Variance Analysis of Gaussian Processes with Application to Average Learning Curves

    Posterior variance analysis of Gaussian processes with application to average learning curves , author=. arXiv preprint arXiv:1906.01404 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 1906

  5. [5]

    Advances in Neural Information Processing Systems , volume=

    Bird: generalizable backdoor detection and removal for deep reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  6. [6]

    Proceedings of the 35th annual computer security applications conference , pages=

    Strip: A defence against trojan attacks on deep neural networks , author=. Proceedings of the 35th annual computer security applications conference , pages=

  7. [7]

    arXiv preprint arXiv:2302.03251 , year=

    Scale-up: An efficient black-box input-level backdoor detection via analyzing scaled prediction consistency , author=. arXiv preprint arXiv:2302.03251 , year=

    work page arXiv

  8. [8]

    2019 IEEE symposium on security and privacy (SP) , pages=

    Neural cleanse: Identifying and mitigating backdoor attacks in neural networks , author=. 2019 IEEE symposium on security and privacy (SP) , pages=. 2019 , organization=

    2019

  9. [9]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Policycleanse: Backdoor detection and mitigation for competitive reinforcement learning , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  10. [10]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Black-box detection of backdoor attacks with limited information and data , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  11. [11]

    arXiv preprint arXiv:2105.00579 , year=

    Backdoorl: Backdoor attack against competitive reinforcement learning , author=. arXiv preprint arXiv:2105.00579 , year=

    work page arXiv

  12. [12]

    2020 57th ACM/IEEE Design Automation Conference (DAC) , pages=

    Trojdrl: evaluation of backdoor attacks on deep reinforcement learning , author=. 2020 57th ACM/IEEE Design Automation Conference (DAC) , pages=. 2020 , organization=

    2020

  13. [13]

    Playing Atari with Deep Reinforcement Learning

    Playing atari with deep reinforcement learning , author=. arXiv preprint arXiv:1312.5602 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    Advances in Neural Information Processing Systems , volume=

    Edge: Explaining deep reinforcement learning policies , author=. Advances in Neural Information Processing Systems , volume=

  15. [15]

    Emergent Complexity via Multi-Agent Competition

    Emergent complexity via multi-agent competition , author=. arXiv preprint arXiv:1710.03748 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    Sutton , doi =

    David Silver and Satinder Singh and Doina Precup and Richard S. Sutton , doi =. Reward is enough , url =. Artificial Intelligence , keywords =. 2021 , Bdsk-Url-1 =

    2021

  17. [17]

    and Wu, Celimuge and Low, Yeh-Ching , journal=

    Rasheed, Faizan and Yau, Kok-Lim Alvin and Noor, Rafidah Md. and Wu, Celimuge and Low, Yeh-Ching , journal=. Deep Reinforcement Learning for Traffic Signal Control: A Review , year=

  18. [18]

    Neural computation , volume=

    Long short-term memory , author=. Neural computation , volume=

  19. [19]

    Proceedings of the 42nd International Conference on Machine Learning (ICML) , pages =

    Adversarial Inception Backdoor Attacks against Reinforcement Learning , author =. Proceedings of the 42nd International Conference on Machine Learning (ICML) , pages =. 2025 , series =

    2025

  20. [20]

    The handbook of brain theory and neural networks , year=

    Convolutional networks for images, speech, and time series , author=. The handbook of brain theory and neural networks , year=

  21. [21]

    Advances in Neural Information Processing Systems , volume=

    Provable defense against backdoor policies in reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  22. [22]

    2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS) , pages=

    Gate-variants of gated recurrent unit (GRU) neural networks , author=. 2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS) , pages=. 2017 , organization=

    2017

  23. [23]

    Deep Reinforcement Learning framework for Autonomous Driving

    Deep reinforcement learning framework for autonomous driving , author=. arXiv preprint arXiv:1704.02532 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  24. [24]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  25. [25]

    Submitted to Transactions on Machine Learning Research , year=

    Plan2Cleanse: Test-Time Backdoor Defense via Monte-Carlo Planning in Deep Reinforcement Learning , author=. Submitted to Transactions on Machine Learning Research , year=

  26. [26]

    Artificial Intelligence Review , volume=

    Reinforcement learning in robotic applications: a comprehensive survey , author=. Artificial Intelligence Review , volume=. 2022 , publisher=

    2022

  27. [27]

    arXiv preprint arXiv:2104.02361 , year=

    Backdoor attack in the physical world , author=. arXiv preprint arXiv:2104.02361 , year=

    work page arXiv

  28. [28]

    Advances in Neural Information Processing Systems , volume=

    Posterior and computational uncertainty in Gaussian processes , author=. Advances in Neural Information Processing Systems , volume=

  29. [29]

    Advances in Neural Information Processing Systems , year=

    GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration , author=. Advances in Neural Information Processing Systems , year=

  30. [30]

    BMC medical research methodology , volume=

    Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range , author=. BMC medical research methodology , volume=. 2014 , publisher=

    2014

  31. [31]

    arXiv preprint arXiv:2110.02797 , year=

    Adversarial robustness comparison of vision transformer and mlp-mixer to cnns , author=. arXiv preprint arXiv:2110.02797 , year=

    work page arXiv