REVIEW 3 major objections 5 minor 22 references
Online Poisoning Attack Against Reinforcement Learning under Black-box Environments
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A black-box attacker who knows only the reachable set of each state-action pair can poison rewards and transitioned states online and force a reinforcement-learning agent to converge to a preset deterministic policy.
desk verdict A genuinely new transition-poisoning mechanism with a coherent bilevel attack objective, but the 'black-box' claim rests on an unanalyzed reachable-set oracle. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing identity is the modified transition kernel $\bar{P}_\delta(s'|s,a) = (1-\delta_{s,a})P(s'|s,a) + \delta_{s,a}\mathbf{1}(s'\in S'_{s,a})/|S'_{s,a}|$, which expresses transition poisoning as a convex combination of the unknown true kernel and a uniform distribution over the reachable set $S'_{s,a}$, so gradients can be formed from sample draws rather than from knowing $P$. Around this kernel, the paper builds a bilevel program whose lower-level variable is the poisoned reward $\bar{r}_{s,a}(\delta,\bar{Q}) = \bar{Q}_{s,a} - \gamma\sum_{s'}\bar{P}_\delta(s'|s,a)\bar{Q}_{s',\bar{\pi}_{s'}}$; this choice converts the Bellman equality into the lower-level optimality condition and keeps the double-sampling issue out of the upper-level gradients. Algorithm 1 then alternates updates of poisoned rewards, induced Q-values, and poisoning intensities $\delta$, with the $\delta$-update projected onto $[0,1]$ and the Q-update targeting a value gap of $\epsilon$ between the target action and all others.
What would settle it
Take a small finite MDP whose transition probabilities are known to the experimenter, compute the exact solution of the constrained problem with full model access, and then run Algorithm 1 with only the reachable sets; if the agent's final Q-values do not show the prescribed gap $\epsilon$ at every state or the learned policy differs from the target, the claimed black-box steering capacity fails.
Extended reading notes
Core claim
Poisoning both rewards and transitioned states, rather than only rewards, lets an attacker who is otherwise blind to the environment's dynamics push a Q-learning-style agent toward a predefined deterministic target policy. The poisoned transition kernel is a convex combination of the true kernel and a uniform distribution over the reachable set, controlled by an intensity $\delta$, and the Bellman consistency of the induced Q-values becomes a lower-level optimality condition in a bilevel program whose lower-level variable is the poisoned reward. Because the lower level is solved in closed form, the upper-level gradients can be estimated from sampled transitioned states without double sampling. The maze experiment is the offered evidence: at the critical state (1,1), the agent's learned Q-values converge to the attacker's variable $\bar{Q}$ and the target action wins by the designed gap $\epsilon=1.0$.
Load-bearing premise
The load-bearing premise is that the attacker knows exactly which next states are reachable from each state-action pair, even though the probabilities of those transitions can remain hidden; the paper does not analyze what happens when this set is misspecified.
Editorial extensions
If this is right
- An attacker can force a specific target action at a specific state, not just degrade overall performance, using only support information about the transition kernel, and the induced Q-value gap can be set to any desired $\epsilon$.
- The bilevel reformulation lets the attacker maintain Bellman consistency with a single sampled transition per update, avoiding the need for a simulator that can draw two independent samples from the same state-action pair.
- The attack is online and agnostic to the agent's learning algorithm, so it can be interleaved with normal agent-environment interaction rather than requiring a separate offline poisoning phase.
- The weight $\rho_\delta$ trades off between how much the poisoned rewards deviate and how intensely transitioned states are replaced, giving the attacker a dial for how detectable each poisoning channel is.
Reading between the lines
- Extension: the paper defers convergence analysis, so a randomized MDP benchmark varying the discount factor, state-space size, and reachable-set cardinality would show how reliably the promised gap $\epsilon$ is achieved before any general guarantee can be inferred.
- Extension: the method's exposed boundary is exactness of the reachable set; misspecifying $S'_{s,a}$ by adding or deleting a reachable state would shift the uniform term in the kernel and likely break the induced Bellman fixed point, so a sensitivity study would measure how much error the attack tolerates.
- Extension: the same bilevel trick of keeping the equality-constrained variable in the lower level could be reused for other poisoning objectives with equality constraints, such as offline or multi-agent poisoning, where a direct penalty would create double-sampling terms.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes an online environment-poisoning attack against reinforcement learning agents in a black-box setting. The attacker is assumed to be able to modify both reward values and transitioned states in the data stream, without knowing the environment's transition probabilities. The poisoning problem is formulated as a constrained optimization (Section 3.2), converted into a penalty-based bilevel optimization, and solved with a stochastic single-loop gradient algorithm (Section 3.3, Algorithm 1). A key assumption is that the attacker knows the exact reachable set S'_s,a for every state-action pair. The method is evaluated in a single maze environment with Q-learning, reporting that the agent learns the target downward policy at state (1,1) with a value gap of 1.0. The paper concludes with an acknowledgment that convergence analysis is left for future work.
Significance. If the claims hold, this work would advance the state of the art by extending poisoning attacks from white-box to black-box environments and by introducing a bilevel-optimization-based attacker that avoids the double-sampling issue. The gradient derivations in Eqs. (8)-(12) are internally consistent, and the algorithmic pipeline (Algorithm 1) is plausible. However, the main claim of operating under unknown dynamics is substantially weakened by Assumption 1, which assumes exact knowledge of the transition support. The lack of convergence guarantees and the limited experimental validation further temper the significance. The manuscript is clearly written and the optimization ideas are interesting, but the evidence provided does not yet establish the full breadth of the claims.
major comments (3)
- [Section 2, Assumption 1; Eq. (3); Eq. (12); Algorithm 1] The exact reachable set S'_s,a = {s' : P(s'|s,a) > 0} is load-bearing for the entire method: it defines the modified transition kernel in Eq. (3), is used in the transition-poisoning sampling of Algorithm 1, and appears in the gradient formula (12). Yet the paper provides no estimation procedure, sample-complexity bound, or error analysis for this set in a black-box environment. The text calls the assumption 'mild' and says the information 'can be acquired' before or during the attack, but no method is given. If the attacker's support estimate is wrong---missing a real next state or including an impossible one---the poisoned kernel is no longer a convex combination over the true support, modified transitions may become detectable, and the gradient formulas are no longer valid. The experiment sidesteps this issue by giving the attacker the simulator's true support, so the reported success does not actually validate the black-box claim. A concrete test would be to run the attack with an estimated support (e.g., collected from finite transition data) and report sensitivity to estimation errors.
- [Section 3.3, Eq. (6), Algorithm 1; Conclusion] The paper does not provide a convergence analysis for the proposed stochastic bilevel algorithm, and the conclusion explicitly states that this is left for future work. This is a significant gap because the optimization is stochastic, nonconvex, and involves a penalty parameter that is increased heuristically. Moreover, the equivalence between the constrained problem (5) and the bilevel problem (6) is not fully established; the text itself notes that δ's boundedness 'hinders' the equivalence, and Algorithm 1 handles this only through a projection step whose effect on the bilevel analysis is not discussed. Without convergence guarantees or at least a formal statement of the approximation error introduced by the penalty method and the projection, it is unclear whether the iterative updates can be expected to reach a solution of the original problem (4), and the reported experimental success could be a consequence of the specific maze structure rather than the algorithm's general properties.
- [Section 4] The experimental validation is too limited to support the central claims. Only one maze environment, five repeated experiments, and a Q-learning agent are considered, despite the abstract's claim that the attack works for a 'flexible reinforcement learning algorithm.' Key hyperparameters---the step-size sequences α_k, β_k, λ_k, and the penalty schedule ρ_Φ,k---are not reported, making the results non-reproducible. There is no comparison with baselines (e.g., random poisoning, reward-only attacks, white-box attacks), no evaluation of the attack's stealthiness in terms of detectability, and no quantitative attack-success rate across states or episodes. The success claim rests almost entirely on the value gap at a single state-action pair (1,1). A more comprehensive evaluation across multiple environments, RL algorithms, and random seeds is necessary to substantiate the main contributions.
minor comments (5)
- [Section 3.1] The sentence 'δ is component in a simplex set' is unclear; δ is a vector in [0,1]^{|S||A|}, which is a hypercube, not a simplex in the usual sense. Please rephrase.
- [Algorithm 1, step 6] The update rule for Q_{k+1}_{s'_i,\pi_{s'_i}} (the second line of step 6) is not derived in the text. Please clarify how it relates to the stochastic gradient of Eq. (11), particularly to the inverse-probability estimation described in Section 3.3.
- [Section 4, Figure 2(a)] The description says the target policy 'guides the agent to navigate towards the destination while traversing the gray grids,' but gray grids are described as undesirable and incurring a penalty. Please clarify whether the target policy intentionally visits gray grids or the wording is inaccurate.
- [Section 3.2, Eq. (5)] The penalty function Φ(x) = 1(x>0)x is used, but the text does not explain why a one-sided penalty (rather than a squared hinge or other form) is chosen. A brief justification would improve readability.
- [Section 1.1] The claim that Xu et al. (2022) 'fails to provide a viable scheme for modifying environment's hyper-parameters' is vague; please specify what hyper-parameters are meant and why the scheme is not viable.
Circularity Check
No significant circularity: the attack objective and the reported success metric coincide, but the agent's Q-values are produced by independent Q-learning on poisoned data, so the demonstration does not reduce to the optimization by construction.
full rationale
The paper's derivation chain is self-contained: the attacker's optimization (Problem (4), penalty reformulation (5), bilevel reformulation (6), and Algorithm 1) constructs poisoned rewards and transitioned states from the assumed reachable-set knowledge and sampled transitions. The experiment then runs a separate Q-learning agent on the poisoned data and reports the agent's learned Q-values in Figures 2(b) and 3(a)-(b). These learner Q-values are not fitted parameters or outputs of the attacker's optimization; they come from the agent's own update rule, so reporting that they exhibit the target value gap is an empirical check rather than an algebraic identity. The value gap epsilon=1.0 is indeed the same quantity that the attacker's objective is designed to create, which makes the evaluation self-referential in the sense of testing the algorithm against its own success criterion, but this is not a circular derivation: no equation in the paper is defined in terms of the measured agent Q-values, and no fitted quantity is renamed as a prediction. There are no load-bearing self-citations; the cited frameworks (Ma et al. 2019; Hong et al. 2023; Dai et al. 2018) are external prior work, and none of the references is authored by Li, Zhang, or Wu. The weakest point, Assumption 1 (exact knowledge of the reachable set S'_s,a), is a substantive modeling assumption with no procedure or error analysis, and the convergence analysis is explicitly left for future work; however, missing support and robustness concerns are limitations of the black-box claim, not circularity. Accordingly, the paper merits a circularity score of 0.
Assumptions & free parameters
free parameters (4)
- Value gap epsilon =
1.0
- Transition poisoning weight rho_delta =
2.0
- Penalty schedule rho_Phi,k =
not reported
- Step-size schedules alpha_k, beta_k, lambda_k =
not reported
assumptions (4)
- domain assumption The attacker knows the reachable set S'_s,a for every state and action (Assumption 1).
- domain assumption The target policy is restricted to be deterministic.
- domain assumption The agent's learning dynamics are adequately modeled by the Bellman equation and Q-function updates, even though the attacker does not know the agent's algorithm.
- standard math Penalty and bilevel stochastic approximation methods from cited literature converge under the applied sampling scheme.
Cite this review
Pith. "Pith review of Online Poisoning Attack Against Reinforcement Learning under Black-box Environments." pith.science (2026). https://pith.science/paper/GIOXYR5C
@misc{pith2026241200797,
author = {Pith},
title = {Pith review of: Online Poisoning Attack Against Reinforcement Learning under Black-box Environments},
year = {2026},
howpublished = {\url{https://pith.science/paper/GIOXYR5C}},
note = {Machine review of arXiv:2412.00797}
}
read the original abstract
This paper proposes an online environment poisoning algorithm tailored for reinforcement learning agents operating in a black-box setting, where an adversary deliberately manipulates training data to lead the agent toward a mischievous policy. In contrast to prior studies that primarily investigate white-box settings, we focus on a scenario characterized by \textit{unknown} environment dynamics to the attacker and a \textit{flexible} reinforcement learning algorithm employed by the targeted agent. We first propose an attack scheme that is capable of poisoning the reward functions and state transitions. The poisoning task is formalized as a constrained optimization problem, following the framework of \cite{ma2019policy}. Given the transition probabilities are unknown to the attacker in a black-box environment, we apply a stochastic gradient descent algorithm, where the exact gradients are approximated using sample-based estimates. A penalty-based method along with a bilevel reformulation is then employed to transform the problem into an unconstrained counterpart and to circumvent the double-sampling issue. The algorithm's effectiveness is validated through a maze environment.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Poisoning deep reinforcement learning agents with in-distribution triggers
Chace Ashcraft and Kiran Karra. Poisoning deep reinforcement learning agents with in-distribution triggers. arXiv preprint arXiv:2106.07798, 2021
arXiv 2021
-
[2]
Generative adversarial user model for reinforcement learning based recommendation system
Xinshi Chen, Shuang Li, Hui Li, Shaohua Jiang, Yuan Qi, and Le Song. Generative adversarial user model for reinforcement learning based recommendation system. In International Conference on Machine Learning, pages 1052--1061. PMLR, 2019
work page 2019
-
[3]
Badrl: Sparse targeted backdoor attack against reinforcement learning
Jing Cui, Yufei Han, Yuzhe Ma, Jianbin Jiao, and Junge Zhang. Badrl: Sparse targeted backdoor attack against reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 11687--11694, 2024
work page 2024
-
[4]
Sbeed: Convergent reinforcement learning with nonlinear function approximation
Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song. Sbeed: Convergent reinforcement learning with nonlinear function approximation. In International conference on machine learning, pages 1125--1134. PMLR, 2018
work page 2018
-
[5]
Execute order 66: targeted data poisoning for reinforcement learning
Harrison Foley, Liam Fowl, Tom Goldstein, and Gavin Taylor. Execute order 66: targeted data poisoning for reinforcement learning. arXiv preprint arXiv:2201.00762, 2022
arXiv 2022
-
[6]
Approximation methods for bilevel programming
Saeed Ghadimi and Mengdi Wang. Approximation methods for bilevel programming. arXiv preprint arXiv:1802.02246, 2018
arXiv 2018
-
[7]
Deep reinforcement learning for robotic manipulation
Shixiang Gu, Ethan Holly, Timothy P Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation. arXiv preprint arXiv:1610.00633, 1: 0 1, 2016
arXiv 2016
-
[8]
A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic
Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic. SIAM Journal on Optimization, 33 0 (1): 0 147--180, 2023
2023
Show all 22 references
-
[9]
Trojdrl: evaluation of backdoor attacks on deep reinforcement learning
Panagiota Kiourti, Kacper Wardega, Susmit Jha, and Wenchao Li. Trojdrl: evaluation of backdoor attacks on deep reinforcement learning. In 2020 57th ACM/IEEE Design Automation Conference (DAC), pages 1--6. IEEE, 2020
2020
-
[10]
Provably efficient black-box action poisoning attacks against reinforcement learning
Guanlin Liu and Lifeng Lai. Provably efficient black-box action poisoning attacks against reinforcement learning. Advances in Neural Information Processing Systems, 34: 0 12400--12410, 2021
2021
-
[11]
Efficient adversarial attacks on online multi-agent reinforcement learning
Guanlin Liu and Lifeng Lai. Efficient adversarial attacks on online multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36: 0 24401--24433, 2023
2023
-
[12]
Policy poisoning in batch reinforcement learning and control
Yuzhe Ma, Xuezhou Zhang, Wen Sun, and Jerry Zhu. Policy poisoning in batch reinforcement learning and control. Advances in Neural Information Processing Systems, 32, 2019
2019
-
[13]
Convergence rate of a penalty method for strongly convex problems with linear constraints
Angelia Nedi \'c and Tatiana Tatarenko. Convergence rate of a penalty method for strongly convex problems with linear constraints. In 2020 59th IEEE Conference on Decision and Control (CDC), pages 372--377. IEEE, 2020
2020
-
[14]
Talking to bots: Symbiotic agency and the case of tay
Gina Neff. Talking to bots: Symbiotic agency and the case of tay. International Journal of Communication, 2016
2016
-
[15]
Scalable end-to-end autonomous vehicle testing via rare-event simulation
Matthew O'Kelly, Aman Sinha, Hongseok Namkoong, Russ Tedrake, and John C Duchi. Scalable end-to-end autonomous vehicle testing via rare-event simulation. Advances in neural information processing systems, 31, 2018
2018
-
[16]
Policy teaching via environment poisoning: Training-time adversarial attacks against reinforcement learning
Amin Rakhsha, Goran Radanovic, Rati Devidze, Xiaojin Zhu, and Adish Singla. Policy teaching via environment poisoning: Training-time adversarial attacks against reinforcement learning. In International Conference on Machine Learning, pages 7974--7984. PMLR, 2020
2020
-
[17]
Reward poisoning in reinforcement learning: Attacks against unknown learners in unknown environments
Amin Rakhsha, Xuezhou Zhang, Xiaojin Zhu, and Adish Singla. Reward poisoning in reinforcement learning: Attacks against unknown learners in unknown environments. arXiv preprint arXiv:2102.08492, 2021
2021 arXiv
-
[18]
Understanding the limits of poisoning attacks in episodic reinforcement learning
Anshuka Rangi, Haifeng Xu, Long Tran-Thanh, and Massimo Franceschetti. Understanding the limits of poisoning attacks in episodic reinforcement learning. arXiv preprint arXiv:2208.13663, 2022
2022 arXiv
-
[19]
Reward poisoning attacks on offline multi-agent reinforcement learning
Young Wu, Jeremy McMahan, Xiaojin Zhu, and Qiaomin Xie. Reward poisoning attacks on offline multi-agent reinforcement learning. In Proceedings of the aaai conference on artificial intelligence, volume 37, pages 10426--10434, 2023
2023
-
[20]
Spiking pitch black: Poisoning an unknown environment to attack unknown reinforcement learners
Hang Xu, Xinghua Qu, and Zinovi Rabinovich. Spiking pitch black: Poisoning an unknown environment to attack unknown reinforcement learners. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pages 1409--1417, 2022
2022
-
[21]
Black-box targeted reward poisoning attack against online deep reinforcement learning
Yinglun Xu and Gagandeep Singh. Black-box targeted reward poisoning attack against online deep reinforcement learning. arXiv preprint arXiv:2305.10681, 2023
2023 arXiv
-
[22]
Adaptive reward-poisoning attacks against reinforcement learning
Xuezhou Zhang, Yuzhe Ma, Adish Singla, and Xiaojin Zhu. Adaptive reward-poisoning attacks against reinforcement learning. In International Conference on Machine Learning, pages 11225--11234. PMLR, 2020
2020
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.