Pith. sign in

REVIEW 2 major objections 5 minor 56 references

Provably Efficient Action-Manipulation Attack Against Continuous Reinforcement Learning

T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that an attacker who only observes trajectories can force a continuous-action reinforcement-learning agent to learn a target policy with sublinear attack cost, provided the agent's dynamic regret is sublinear.

desk verdict The oracle-attack half is fine, but the black-box LCBT guarantee rests on an importance-sampling definition that is not a probability distribution, so Theorem 2 does not go through. read the letter →

arxiv 2411.13116 v1 pith:IDQVSIU6 submitted 2024-11-20 cs.LG cs.AI

Zhi Luo , Xiyuan Yang , Pan Zhou , Di Wang This is my paper · ORCID
classification cs.LGcs.AI
keywords action-manipulationattackcontinuousreinforcementlearningblack-boxMonteCarlotreesearchlowerconfidenceboundpolicypoisoningdynamicregretcyber-physicalsystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that action-manipulation attacks on reinforcement learning remain cheap and effective when both states and actions are continuous. It proposes a black-box attack, LCBT, which uses only observed states, actions, and rewards to search for replacement actions, and proves that for any agent whose dynamic regret grows sublinearly, the attack forces the learned policy into an attacker-chosen target policy set with sublinear attack cost. It also gives a white-box oracle attack whose cost is bounded essentially by the agent's own regret. If the results are correct, a model-free attacker could steer continuous-control RL systems, such as cyber-physical controllers, toward dangerous policies during training while intervening on only a sublinear number of steps. The paper demonstrates the attack on DDPG, PPO, and TD3 in continuous environments.

What carries the argument

The mechanism is the Lower Confidence Bound Tree (LCBT): a binary action cover tree that recursively partitions the continuous action space, where each node $(D,I)$ carries a representative action $a_{D,I}$, a lower confidence bound $L^h_{D,I}(k)$ on $Q^o_h(s,a_{D,I})$, and a tighter bound $B^h_{D,I}(k)$ built from the children. The bounds combine importance-sampled trajectory returns with Hoeffding-style confidence radii plus structural terms $L_s d_s$ and $\nu_1\rho^D$ for state and action discretization. A traversal starting at the root repeatedly descends to the child with the smaller $B$-value, producing the replacement action; nodes are expanded when the confidence radius shrinks below the node's diameter. This machinery locates the worst action $a^-_h(s)$ without any model or algorithm knowledge, and the sublinear growth of the tree ($|\mathcal{T}^h_K| = O(K^E)$) is what keeps the total attack cost sublinear.

What would settle it

Run LCBT on a one-step MDP with a very small target action set, compute $\mathbb{E}[\rho G \mid s]$ under the real attacking behavior policy and compare it with $V^o(s)$; any systematic gap larger than the claimed Hoeffding radius would indicate that the LCB is biased and would invalidate the sublinear bound in Theorem 2.

Watch

Extended reading notes

Core claim

The central claim is Theorem 2: with probability at least $1-\delta_1-\delta_2$, the black-box LCBT attack forces an agent with sublinear dynamic regret to learn policies in the target policy set $\Pi^\dagger$, with attack cost $|\tau|$ bounded by $O(D\text{-Regret}(K) + M H^3 K^E \log(MT))$, where $E = \log_{2\rho^{-2}} 2 < 1$. The attack works because, under the condition $\Delta_{\min} > 0$, replacing out-of-target actions with the worst actions for the target policy makes the agent perceive $\pi^o$, the best policy inside the target set, as the optimal policy. The white-box oracle attack achieves the stronger bound $O(\mathcal{R}(T))$ with full model knowledge, while the black-box bound adds a sublinear exploration cost for locating the worst action without knowing the MDP.

Load-bearing premise

The attack's lower-confidence-bound estimates must be genuine lower bounds on the target policy's Q-values, but the proof of that uses an importance-sampling identity that drops exactly the trajectories where the attacker substituted an out-of-target action.

Editorial extensions

If this is right

  • A black-box attacker with only trajectory access can induce policy convergence to the target set for any continuous RL algorithm whose dynamic regret is sublinear (Theorem 2).
  • The attack cost is $O(D\text{-Regret}(K) + M H^3 K^E \log(MT))$ with $E<1$, so the manipulation overhead beyond the victim's own regret is sublinear in the number of episodes.
  • The white-box oracle attack achieves attack cost $O(\mathcal{R}(T))$ up to a $\Delta_{\min}$ denominator and a logarithmic factor, showing that full model knowledge removes the exploration term.
  • The empirical results show that under LCBT or the oracle attack, DDPG, PPO, and TD3-trained policies act inside the target action space in most test steps, with measured attack cost growing sublinearly.
  • The cover-tree size stays $O(K^E)$, giving time complexity $O(HK^{1+E}+HK\log_{\rho^{-2}}K)$ and space complexity $O(MHK^E)$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if Theorem 2 is correct, then better-performing victims are cheaper to attack, because a smaller $D\text{-Regret}(K)$ makes the first term in the attack-cost bound smaller; strong RL learners would be more vulnerable, not less.
  • Editorial inference: the tree-LCB construction is effectively a continuous-action bandit over the value gap $V^o - Q^o$, so the same idea could be adapted to reward poisoning or observation poisoning in continuous spaces.
  • Editorial inference: the condition $L_s d_s < \Delta_{\min}/2$ ties attack feasibility to how well the attacker can approximate the target policy's value, which suggests that a defender who smooths the Q-function or enlarges the gap could raise the attack cost.
  • Editorial inference: the dependence on the state-partition count $M$ makes the bound degrade in high-dimensional state spaces, so the practical attack cost in very large continuous environments is likely to exceed the sublinear ideal shown in low-dimensional experiments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies training-time action-manipulation attacks against reinforcement learning in continuous state and action spaces. It defines a threat model in which the attacker replaces the agent's actions to steer the agent toward a target policy set, and it proposes two attacks: a white-box "oracle" attack that replaces out-of-target actions with the worst action under the best policy in the target set, and a black-box algorithm called LCBT that uses a binary cover tree of the action space, lower-confidence-bound estimates of Q-values, and importance sampling. The main theoretical claim is that LCBT forces an agent with sublinear dynamic regret to mostly act according to the target policy set with sublinear attack cost, O(R(T) + M H^3 K^E log(MT)) with E<1. Experiments on DDPG, PPO, and TD3 in three continuous-control environments show convergence of rewards to the target-policy level and high action-similarity percentages.

Significance. If the black-box bound were correct, it would be a substantial advance: prior action-manipulation results were mostly restricted to tabular settings or required model/algorithm knowledge, whereas Theorem 2 promises sublinear attack cost in continuous spaces from trajectory information alone. The threat model is clearly stated, the target-action-space formalism is reasonable, and the white-box oracle analysis appears internally coherent. The paper also makes a genuine attempt to define a sample-complexity notion of attack cost. However, the central black-box guarantee rests on an importance-sampling identity that is not valid as written, so the headline sublinear-cost theorem is not established. The experiments are suggestive but do not compensate for the proof gap, and no code or baselines are provided.

major comments (2)
  1. [Section 4.2, Eq. (3)-(4), Appendix J, Eq. (15)] The proof of Lemma 2 asserts E[r_i + G^i rho^i | s_i, a_i] = Q_o(s_i, a_i), using the importance ratio rho defined from P(ea|pi_o) = I{ea in A-dagger}. This is not a probability distribution (or density) over the continuous action space, and it cannot serve as the Radon-Nikodym derivative for importance sampling. In particular, for any trajectory containing an attacked step whose replacement action lies outside A-dagger---the very situation the LCBT algorithm is designed to explore---the factor I{ea in A-dagger} is zero, so rho is identically zero for that episode. The conditional expectation in Eq. (15) is then taken under a measure that assigns zero weight to exactly the out-of-target actions the attack must evaluate; it does not equal Q_o(s,a_D,I). If the attacker replaces one out-of-target action by another (a common event during exploration), rho is zero on all relevant trajectories and the estimator receives no signal from those episodes. Consequently Lemma 2 bounds a biased estimator, Lemma 3 does not follow, and the derivation of Eq. (7) in Theorem 2 collapses. This is the load-bearing step for the black-box guarantee.
  2. [Section 4.2, Eq. (4)] The definition of the behavior policy P(ea_h|s_h, b_h) is also not a well-defined conditional distribution. Eq. (4) assigns probability 1 to ea=a_k when a_k is in A-dagger and probability 1 to ea=a_{D,I} when a_k is not in A-dagger; these are events that depend on the agent's realized action a_k, not on the conditioning state s_h. For a stochastic agent policy, the actual data-generating distribution for ea_h is a mixture (with probability mass on a_k for target actions and on a_{D,I} for out-of-target actions), and the mixture weight---the agent's action probability---is missing from the denominator. The proof never specifies the filtration or the probability space with respect to which the expectation in Eq. (15) is taken, so the claimed unbiasedness of the estimator is not mathematically well-defined even apart from the normalization problem with I{ea in A-dagger}.
minor comments (5)
  1. [Abstract and Theorem 2, Eq. (7)] The abstract advertises an attack cost of O(R(T) + M H^3 K^E log(MT)), but Theorem 2, Eq. (7), contains M H^2 log(2MHK^2/delta_1) * sum_h |T^h_K|, which with the Appendix M bound |T^h_K| <= O(K^E) gives a leading term M H^2 K^E log(K), not M H^3 K^E log(MT). The H discrepancy should be reconciled.
  2. [Throughout, especially Algorithm 1 and Theorem 2] The symbol K is used both for the total number of episodes and for the number of initial no-attack episodes (e.g., Algorithm 1, line 6, versus the statement of Theorem 2). This double use makes the cost bounds difficult to parse; distinct symbols should be introduced.
  3. [Section 5, Figures 2, 3, and 6] The figure legends in the submitted text contain unencoded font tokens (strings such as "/uni00000013/...") instead of readable labels, making the experimental figures effectively uninterpretable. The figures should be regenerated with standard text encoding.
  4. [Section 5] The experiments compare only oracle and LCBT attacks against the attack-free curve; they do not compare with existing continuous-space action-manipulation baselines such as LAS or the query-based attack discussed in Section 2, so the empirical advantage over prior methods is not quantified.
  5. [Section 5, paragraph 1] There is a typo, "sunlinear" for "sublinear", in the sentence describing the attack cost results.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the headline bound is derived from external regret/gap quantities; the invalid importance-sampling identity is a correctness error, not a circular step.

full rationale

After walking the derivation chain, I find no circular step that reduces a claimed prediction to an input by construction. The white-box result (Theorem 1) is a regret-to-attack-cost conversion: it lower-bounds the attack-induced Bellman gap by |τ|·Δmin and upper-bounds the same gap by the agent's external Regret(K) through a martingale/Freedman argument; both quantities are outside the attack's own construction. The black-box result (Theorem 2) similarly combines the same regret bound with an LCB/UCT exploration cost whose node-count O(K^E) (Appendix M) follows from the expansion condition ν1·ρ^D ≥ β, not from the target policy or from any fitted parameter. No fitted input is renamed as a prediction: the target policy π† and radius r_a are chosen by the attacker, and the test-phase similarity is measured against that externally chosen π†. The references used are external results (dynamic regret, Freedman's inequality, prior action-poisoning attacks); none is a load-bearing self-citation. The paper's genuine weakness is mathematical, not circular: in Appendix J, the identity E[ρ G] = V^o(s) with ρ built from P(ẽ|π_o) = I{ẽ ∈ A†} is not a valid importance-sampling identity, because that indicator is not a probability distribution and zeroes out exactly the attacked trajectories; this biases the Q estimator and undermines Lemma 2, Lemma 3, and hence Theorem 2. A false proof step, however, is not an equivalence-by-construction between input and output; the claimed cost bound is still stated against external quantities (D-Regret, Δmin, M, H, K). The Appendix B limitations (tree partitioning in high dimensions, parameter sensitivity, low-value target policies) are honest caveats, not admissions of circularity. Therefore the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central black-box bound rests on the Lipschitz assumptions (unverifiable for the attacker), the positivity of Δmin for the chosen target policy, the victim's sublinear-regret property, and a questionable importance-sampling identity. The free parameters M, ρ, ra, ν1, Ls, ds are attacker-chosen and affect the constants in the bound.

free parameters (4)
  • ra = 0.0625, 0.31, 0.497
    Target action radius defining A†; chosen by the attacker and directly controls the target policy set size.
  • M = 16, 81, 59049
    Number of state-space subspaces; appears linearly in the attack-cost bound and controls state-discretization error.
  • rho = 1/2, 1/sqrt(2), 1/fifth-root(2)
    Cover-tree radius reduction ratio; determines E = log_{2ρ^{-2}} 2 and the sublinear exponent K^E.
  • nu1, Ls, ds = not specified
    Lipschitz and diameter constants in Assumption 1 that set LCB penalties and the Δmin - 2Lsds denominator.
assumptions (5)
  • standard math Rewards are bounded in [0,1] and H is finite; Q and V values are bounded.
    Section 3 preliminaries. Needed for Hoeffding and Freedman bounds.
  • domain assumption Assumption 1(a)-(d): node diameters shrink geometrically and Q_o is Lipschitz in state and action.
    Appendix E; needed to charge discretization error with LCB terms.
  • domain assumption Δmin > 0, i.e., the chosen target policy is not the globally worst policy.
    Eq. (1) in Section 3, feasibility condition for the attack.
  • domain assumption The victim RL algorithm has sublinear dynamic regret (assumed for DDPG, PPO, TD3).
    Used in Theorems 1 and 2 to convert value gaps into attack-cost bounds; not proven for the tested deep RL algorithms.
  • ad hoc to paper The importance-sampling ratio ρ from P(ẽ|π_o)=I{ẽ∈A†} gives unbiased estimates of Q_o for all tree nodes.
    Invoked in Appendix J Eq. (15) and Lemma 3; not a valid probability ratio in continuous spaces and zero for out-of-target actions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Provably Efficient Action-Manipulation Attack Against Continuous Reinforcement Learning." pith.science (2026). https://pith.science/paper/IDQVSIU6

@misc{pith2026241113116,
  author       = {Pith},
  title        = {Pith review of: Provably Efficient Action-Manipulation Attack Against Continuous Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IDQVSIU6}},
  note         = {Machine review of arXiv:2411.13116}
}
abstract

Manipulating the interaction trajectories between the intelligent agent and the environment can control the agent's training and behavior, exposing the potential vulnerabilities of reinforcement learning (RL). For example, in Cyber-Physical Systems (CPS) controlled by RL, the attacker can manipulate the actions of the adopted RL to other actions during the training phase, which will lead to bad consequences. Existing work has studied action-manipulation attacks in tabular settings, where the states and actions are discrete. As seen in many up-and-coming RL applications, such as autonomous driving, continuous action space is widely accepted, however, its action-manipulation attacks have not been thoroughly investigated yet. In this paper, we consider this crucial problem in both white-box and black-box scenarios. Specifically, utilizing the knowledge derived exclusively from trajectories, we propose a black-box attack algorithm named LCBT, which uses the Monte Carlo tree search method for efficient action searching and manipulation. Additionally, we demonstrate that for an agent whose dynamic regret is sub-linearly related to the total number of steps, LCBT can teach the agent to converge to target policies with only sublinear attack cost, i.e., $O\left(\mathcal{R}(T) + MH^3K^E\log (MT)\right)(0<E<1)$, where $H$ is the number of steps per episode, $K$ is the total number of episodes, $T=KH$ is the total number of steps, $M$ is the number of subspaces divided in the state space, and $\mathcal{R}(T)$ is the bound of the RL algorithm's regret. We conduct our proposed attack methods on three aggressive algorithms: DDPG, PPO, and TD3 in continuous settings, which show a promising attack performance.

Figures

Figures reproduced from arXiv: 2411.13116 by the authors.

Figure 1
Figure 1. Action-manipulation attack model. The key to achieving an efficient action￾manipulation attack purpose lies in how to select appropriate actions from the action space to re￾place the original actions generated by the agent. In tabular RL, the attacker can evaluate the value of each action solely based on interaction tra￾jectory information, thereby selecting the action that is most conducive to achieving the attack … view at source ↗
Figure 2
Figure 2. Reward and attack cost results of Environment [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Reward and attack cost results of Environment [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Environment 1. The objective of this environment is to control the slider to slide on the rod. G Additional Numerical Results In these three tables, the similarity calculation formula we adopt is as follows: sim = P(I {l(a1, a2) ≤ ra}) steps 14 [PITH_FULL_IMAGE:figure…
Figure 5
Figure 5. Figure 5: Environment 2. The objective of this environment is to control a vehicle to move on a two-dimensional plane. 0.0 0.2 0.4 0.6 0.8 1.0 Time steps 1e6 4 2 0 2 4 6 8 Average Episodic Reward DDPG Attack-free reward LCBT attack reward Oracle attack reward Target policy rewar…
Figure 6
Figure 6. Figure 6: Reward and attack cost results of Environment [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: The total number of nodes of the cover trees. [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: The percentage of time taken to execute the LCBT algorithm during the attack phase. [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 41 canonical work pages

  1. [1]

    Deep reinforcement learning for financial trading using multi-modal features

    Loukia Avramelou, Paraskevi Nousi, Nikolaos Passalis, and Anastasios Tefas. Deep reinforcement learning for financial trading using multi-modal features. Expert Systems with Applications, 238:121849, 2024

    work page 2024

  2. [2]

    Vulnerability of deep reinforcement learning to policy induction attacks

    Vahid Behzadan and Arslan Munir. Vulnerability of deep reinforcement learning to policy induction attacks. In Machine Learning and Data Mining in Pattern Recognition: 13th International Conference, MLDM 2017, New York, NY, USA, July 15-20, 2017, Proceedings 13, pages 262–275. Springer, 2017

    work page 2017

  3. [3]

    Simple physical adver- sarial examples against end-to-end autonomous driving models

    Adith Boloor, Xin He, Christopher Gill, Yevgeniy V orobeychik, and Xuan Zhang. Simple physical adver- sarial examples against end-to-end autonomous driving models. In 2019 IEEE International Conference on Embedded Software and Systems (ICESS), pages 1–7. IEEE, 2019

    work page 2019

  4. [4]

    Dynamic regret of policy optimization in non-stationary environments

    Yingjie Fei, Zhuoran Yang, Zhaoran Wang, and Qiaomin Xie. Dynamic regret of policy optimization in non-stationary environments. Advances in Neural Information Processing Systems, 33:6743–6754, 2020

    work page 2020

  5. [5]

    Execute order 66: Targeted data poisoning for reinforcement learning

    Harrison Foley, Liam Fowl, Tom Goldstein, and Gavin Taylor. Execute order 66: Targeted data poisoning for reinforcement learning. arXiv preprint arXiv:2201.00762, 2022

    arXiv 2022

  6. [6]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587–1596. PMLR, 2018

    2018

  7. [7]

    A practical guide to multi-objective reinforcement learning and planning

    Conor F Hayes, Roxana R˘adulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems, 36(1):26, 2022

    work page 2022

  8. [8]

    Financial trading as a game: A deep reinforcement learning approach

    Chien Yi Huang. Financial trading as a game: A deep reinforcement learning approach. arXiv preprint arXiv:1807.02787, 2018

    arXiv 2018

Show all 56 references
  1. [9]

    Adversarial attacks on neural network policies

    Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprint arXiv:1702.02284, 2017

    2017 arXiv

  2. [10]

    Deceptive reinforcement learning under adversarial manipulations on cost signals

    Yunhan Huang and Quanyan Zhu. Deceptive reinforcement learning under adversarial manipulations on cost signals. In Decision and Game Theory for Security: 10th International Conference, GameSec 2019, Stockholm, Sweden, October 30–November 1, 2019, Proceedings 10, pages 217–237....

    2019

  3. [11]

    Challenges and countermeasures for adversarial attacks on deep reinforcement learning

    Inaam Ilahi, Muhammad Usama, Junaid Qadir, Muhammad Umar Janjua, Ala Al-Fuqaha, Dinh Thai Hoang, and Dusit Niyato. Challenges and countermeasures for adversarial attacks on deep reinforcement learning. IEEE Transactions on Artificial Intelligence, 3(2):90–109, 2021

    2021

  4. [12]

    Deep reinforcement learning for autonomous driving: A survey

    B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(6):4909–4926, 2021

    2021

  5. [13]

    Query-based targeted action- space adversarial policies on deep reinforcement learning agents

    Xian Yeow Lee, Yasaman Esfandiari, Kai Liang Tan, and Soumik Sarkar. Query-based targeted action- space adversarial policies on deep reinforcement learning agents. In Proceedings of the ACM/IEEE 12th International Conference on Cyber-Physical Systems, pages 87–97, 2021

    2021

  6. [14]

    Spatiotemporally constrained action space attacks on deep reinforcement learning agents

    Xian Yeow Lee, Sambit Ghadai, Kai Liang Tan, Chinmay Hegde, and Soumik Sarkar. Spatiotemporally constrained action space attacks on deep reinforcement learning agents. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 4577–4584, 2020

    2020

  7. [15]

    Continuous control with deep reinforcement learning

    Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  8. [16]

    Tactics of adversarial attack on deep reinforcement learning agents

    Yen-Chen Lin, Zhang-Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming-Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. arXiv preprint arXiv:1703.06748, 2017

    2017 arXiv

  9. [17]

    Provably efficient black-box action poisoning attacks against reinforcement learning

    Guanlin Liu and Lifeng Lai. Provably efficient black-box action poisoning attacks against reinforcement learning. Advances in Neural Information Processing Systems, 34:12400–12410, 2021

    2021

  10. [18]

    Efficient adversarial attacks on online multi-agent reinforcement learning

    Guanlin Liu and Lifeng Lai. Efficient adversarial attacks on online multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024. 10

    2024

  11. [19]

    Data poisoning attacks in contextual bandits

    Yuzhe Ma, Kwang-Sung Jun, Lihong Li, and Xiaojin Zhu. Data poisoning attacks in contextual bandits. In Decision and Game Theory for Security: 9th International Conference, GameSec 2018, Seattle, WA, USA, October 29–31, 2018, Proceedings 9, pages 186–204. Springer, 2018

    2018

  12. [20]

    Policy poisoning in batch reinforcement learning and control

    Yuzhe Ma, Xuezhou Zhang, Wen Sun, and Jerry Zhu. Policy poisoning in batch reinforcement learning and control. Advances in Neural Information Processing Systems, 32, 2019

    2019

  13. [21]

    Disturbing reinforcement learning agents with corrupted rewards

    Rubén Majadas, Javier García, and Fernando Fernández. Disturbing reinforcement learning agents with corrupted rewards. arXiv preprint arXiv:2102.06587, 2021

    2021 arXiv

  14. [22]

    Inverse filtering for hidden markov models with applications to counter-adversarial autonomous systems.IEEE Transactions on Signal Processing, 68:4987–5002, 2020

    Robert Mattila, Cristian R Rojas, Vikram Krishnamurthy, and Bo Wahlberg. Inverse filtering for hidden markov models with applications to counter-adversarial autonomous systems.IEEE Transactions on Signal Processing, 68:4987–5002, 2020

    2020

  15. [23]

    Optimal attack and defense for reinforcement learning

    Jeremy McMahan, Young Wu, Xiaojin Zhu, and Qiaomin Xie. Optimal attack and defense for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 14332–14340, 2024

    2024

  16. [24]

    Characterizing attacks on deep reinforcement learning

    Xinlei Pan, Chaowei Xiao, Warren He, Shuang Yang, Jian Peng, Mingjie Sun, Jinfeng Yi, Zijiang Yang, Mingyan Liu, Bo Li, et al. Characterizing attacks on deep reinforcement learning. arXiv preprint arXiv:1907.09470, 2019

    1907 arXiv

  17. [25]

    Continuous state-space models for optimal sepsis treatment: a deep reinforcement learning approach

    Aniruddh Raghu, Matthieu Komorowski, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. Continuous state-space models for optimal sepsis treatment: a deep reinforcement learning approach. In Machine Learning for Healthcare Conference, pages 147–163. PMLR, 2017

    2017

  18. [26]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  19. [27]

    Vulnerability-aware poisoning mechanism for online rl with unknown dynamics

    Yanchao Sun, Da Huo, and Furong Huang. Vulnerability-aware poisoning mechanism for online rl with unknown dynamics. arXiv preprint arXiv:2009.00774, 2020

    2009 arXiv

  20. [28]

    Robustifying reinforcement learning agents via action space adversarial training

    Kai Liang Tan, Yasaman Esfandiari, Xian Yeow Lee, Soumik Sarkar, et al. Robustifying reinforcement learning agents via action space adversarial training. In 2020 American control conference (ACC), pages 3959–3964. IEEE, 2020

    2020

  21. [29]

    Adversarial black-box attacks on vision-based deep reinforcement learning agents

    Atanas Tanev, Svetlana Pavlitskaya, Joan Sigloch, Arne Roennau, Ruediger Dillmann, and J Marius Zollner. Adversarial black-box attacks on vision-based deep reinforcement learning agents. In 2021 IEEE International Conference on Intelligence and Safety for Robotics (ISR), pages...

    2021

  22. [30]

    Action robust reinforcement learning and applications in continuous control

    Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, pages 6215–6224. PMLR, 2019

    2019

  23. [31]

    Freedman’s inequality for matrix martingales

    Joel Tropp. Freedman’s inequality for matrix martingales. Electronic Communications in Probability, 16(none):262 – 270, 2011

    2011

  24. [32]

    Reward poisoning attacks on offline multi-agent reinforcement learning

    Young Wu, Jermey McMahan, Xiaojin Zhu, and Qiaomin Xie. Reward poisoning attacks on offline multi-agent reinforcement learning. arXiv preprint arXiv:2206.01888, 2022

    2022 arXiv

  25. [33]

    Transferable environment poisoning: Training-time attack on reinforcement learning

    Hang Xu, Rundong Wang, Lev Raizman, and Zinovi Rabinovich. Transferable environment poisoning: Training-time attack on reinforcement learning. In Proceedings of the 20th international conference on autonomous agents and multiagent systems, pages 1398–1406, 2021

    2021

  26. [34]

    Prediction- guided multi-objective reinforcement learning for continuous robot control

    Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojciech Matusik. Prediction- guided multi-objective reinforcement learning for continuous robot control. In International conference on machine learning, pages 10607–10616. PMLR, 2020

    2020

  27. [35]

    Enhanced adversarial strategically-timed attacks against deep reinforcement learning

    Chao-Han Huck Yang, Jun Qi, Pin-Yu Chen, Yi Ouyang, I-Te Danny Hung, Chin-Hui Lee, and Xiaoli Ma. Enhanced adversarial strategically-timed attacks against deep reinforcement learning. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing ...

    2020

  28. [36]

    Reinforcement learning in healthcare: A survey

    Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1–36, 2021

    2021

  29. [37]

    Robust reinforcement learning on state observations with learned optimal adversary

    Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. arXiv preprint arXiv:2101.08452, 2021

    2021 arXiv

  30. [38]

    Adaptive reward-poisoning attacks against reinforcement learning

    Xuezhou Zhang, Yuzhe Ma, Adish Singla, and Xiaojin Zhu. Adaptive reward-poisoning attacks against reinforcement learning. In International Conference on Machine Learning, pages 11225–11234. PMLR, 2020. 11 A Notation Table Table 4: Notation Table Notation Meaning S The state sp...

    2020

  31. [39]

    − V πk 1 (sk 1) = KX k=1 V o 1(sk

  32. [40]

    For episode k, V o 1(sk

    − V πk 1 (sk 1) , (8) where πk is the policy followed by the agent for each episode k. For episode k, V o 1(sk

  33. [41]

    − V πk 1 (sk 1) = V o 1(sk

  34. [42]

    = E " HX h=1 ∆ k h|F k 1 # where F k h represents the σ-field generated by all the random variables until episode k, step h begins

    − Ea∼πk 1 (·|sk 1 ) h Q o 1(sk 1, a) i + Ea∼πk 1 (·|sk 1 ) h Q o 1(sk 1, a) i − V πk 1 (sk 1) = E h ∆ k 1|F k 1 i + rk 1 + Es′∼P1(·|sk 1 ,a∼πk 1 (·|sk 1 ))V o 2(s′) − rk 1 + Es′∼P1(·|sk 1 ,a∼πk 1 (·|sk 1 ))V πk 2 (s′) = E h ∆ k 1|F k 1 i + Es′∼P1(·|sk 1 ,a∼πk 1 (·|sk 1 )) V o ...

  35. [43]

    (9) Next, we will show that with a probability at least 1 − δ2, we have KX k=1 HX h=1 ∆ k h ≤ KX k=1 V o 1(sk

    − V πk 1 (sk 1) = E " KX k=1 HX h=1 ∆ k h|F k 1 # . (9) Next, we will show that with a probability at least 1 − δ2, we have KX k=1 HX h=1 ∆ k h ≤ KX k=1 V o 1(sk

  36. [45]

    (10) Since E hPH h=1 ∆ k h|F k 1 i = V o 1(sk

    − V πk 1 (sk 1) . (10) Since E hPH h=1 ∆ k h|F k 1 i = V o 1(sk

  37. [46]

    − V πk 1 (sk 1), we can regard Yk := kX i=1 HX h=1 ∆ i h − V o 1(si

  38. [47]

    − V πk 1 (si 1) ! , (11) as a martingale with the difference sequence {Xk}K k=1, which is Xk := HX h=1 ∆ k h − V o 1(sk

  39. [48]

    HX h=1 ∆ k h 2 |F k 1 # ≤ H 2 KX k=1 E

    − V πk 1 (sk 1) . (12) 16 And we have the difference sequence bounded, i.e., |Xk| ≤H 2, ∀k ∈ [1, K]. Define the predictable quadratic variation process of the martingale: WK := PK k=1 E[X 2 k |F k 1 ], with WK ≤ KX k=1 E " HX h=1 ∆ k h 2 |F k 1 # ≤ H 2 KX k=1 E " HX h=1 ∆ k h|...

  40. [49]

    (13) By Freeman’s inequality [31], we have P  YK = KX k=1 Xk > 2H 2 vuutln(1/δ2) KX k=1 V o 1(sk

    − V πk 1 (sk 1) = σ2. (13) By Freeman’s inequality [31], we have P  YK = KX k=1 Xk > 2H 2 vuutln(1/δ2) KX k=1 V o 1(sk

  41. [50]

    − V πk 1 (sk 1)   ≤ exp    − 4H 4 ln( 1 δ2 ) KP k=1 V o 1(sk

  42. [51]

    − V πk 1 (sk 1) /2 σ2 + H 2 · 2H 2 s ln( 1 δ2 ) KP k=1 V o 1(sk

  43. [52]

    2βh k (T h D,I (k), δ1)2T h D,I (k)2 −T h D,I (k)(H − h + 1)2 # = HX h=1 MX m=1 X (D,I )∈T h k kX T h D,I (k)=1 2 exp

    − V πk 1 (sk 1) /3    ≤ exp{− ln(1/δ2)} = δ2, and combine the definition of Yk, we can get (10). Under the oracle attack, when the agent chooses an action satisfying ak h ∈ A† h(sk h), the at- tacker does nothing, and we have Q o h(sk h, ak h) = Qo h(sk h, ak h) ≤ V...

  44. [53]

    − V πk 1 (sk 1) + 2H 2 vuutln(1/δ2) KX k=1 V o 1(sk

  45. [54]

    − V πk 1 (sk 1) (ii) ≤ KX k=1 V π∗,k,k 1 (sk

  46. [55]

    − V πk,k 1 (sk 1) + 2H 2 vuutln(1/δ2) KX k=1 V π∗,k,k 1 (sk

  47. [56]

    − V πk,k 1 (sk 1) = D-Regret(K) + 2H 2p ln(1/δ2) D-Regret(K), (26) where π∗,k = sup π V π,k 1 (sk

  48. [57]

    K · ν2 1 · 2 − ρ2 (H − h + 1)2 · ln (6M H/δ1) + 1 # + 1. (28) Through 2Dm+1 − 1, we get the upper bound of the node number of tree T h K, i.e., T h K ≤ 4

    is the optimal policy of episode k. In (i), we assume K = 0. The reason for (ii) is that because of the existence of the attacker, the environment is non-stationary in the observation of the agent. Combine (25) and (26), we can obtain D-Regret(K) + 2H 2p ln(1/δ2) D-Regret(K) ≥...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.