{"id":"f4d732fa-7472-4627-bce2-2791030f1934","arxiv_id":"2412.00797","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An online attacker knowing only which states are reachable can poison rewards and transitions to make a Q-learning agent follow a target policy in a maze.","lead":"An attacker who can sit between a reinforcement learning agent and its environment can poison both the rewards and the next-state data the agent trains on, steering it toward a chosen policy even when the attacker does not know how the environment works. The paper demonstrates this on a maze and offers a principled optimization recipe for such attacks, which matters for understanding and defending against AI systems that learn from experience.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The black-box claim hinges on Assumption 1: the attacker must know the exact reachable set S'_s,a, yet no procedure or error analysis is given, and a wrong support breaks both the stealthiness constraint and the gradient updates.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: Assumption 1's exact reachable-set knowledge is not justified in a black-box environment. Having read the full text, the optimization reformulation is internally coherent: the lower-level problem for r̄ has a unique solution, the penalty and bilevel transformations are consistent with the stated avoidance of double sampling, and the derivation of the gradient formulas (8), (9), (11), and (12) checks out. The single maze experiment supports the narrow claim that, when the reachable set is known exactly and every state-action pair is sampled each iteration, the Q-learning agent can be steered toward the target policy. However, the paper's own emphasis on black-box unknown dynamics makes the support oracle a real gap rather than a cosmetic limitation. The algorithm's uniform replacement scheme, the definition of P̄δ in Eq. (3), and the gradient update for δ all require the support to be correct; an erroneous support changes the effective poisoned MDP and breaks the stealthiness constraint. The paper states that the reachable set can be acquired before or during the attack but gives no method, no sample complexity, and no sensitivity analysis, and the experiment does not exercise that acquisition step. This does not invalidate the method under the stated assumption, so the verdict stays conditional: the central claim is plausible and internally consistent, but the black-box contribution is not yet fully supported without either a support-learning procedure or a robustness result for support errors.","tokens_in":9914,"tokens_out":11777,"duration_ms":133475,"concrete_test":"Run Algorithm 1 in the maze environment under Assumption 1, but replace the true reachable set with an estimate built from a finite prefix of environment samples (e.g., 100, 1,000, or 10,000 observed transitions before the attack begins). Measure (a) the fraction of poisoned transitioned states that fall outside the true support S'_s,a, and (b) the final value gap Q(1,1,down) - max_{a != down} Q(1,1,a). Compare against the true-support baseline. If the value gap remains at 1.0 and no impossible transitions occur even with a moderately misspecified support, the concern is weakened; if the value gap drops below epsilon or impossible transitions are frequent, Assumption 1 is essential and the 'black-box' framing must be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that an attacker with only black-box access can steer a Q-learning agent by poisoning rewards and transitioned states. The load-bearing condition is Assumption 1: the attacker knows the exact reachable set S'_s,a = {s' in S : P(s'|s,a) > 0} for every (s,a). This set appears in Eq. (3) defining the poisoned kernel P̄δ, in the uniform replacement sampling of Algorithm 1, and in the exact gradient Eq. (12), whose second term subtracts the uniform expectation over S'_s,a. If the attacker's support estimate misses a real next state or includes an impossible one, the poisoned transition kernel is no longer a convex combination of the true P and the uniform distribution over the true support; the Bellman constraints in (4) are then computed with respect to the wrong MDP, and poisoned transitions to impossible states become detectable, violating the paper's stated restriction that modified transitions remain reachable in the original environment. The paper asserts this information can be acquired before or during the attack, but it provides no estimator, no sample-complexity bound, and no robustness analysis for support errors. Since the abstract frames the contribution as black-box with unknown dynamics, this unanalyzed exact-support oracle is the weakest point in the argument. The experiment sidesteps the issue by giving the attacker the simulator's true support, so the reported success does not yet establish the black-box claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes an online environment-poisoning attack against reinforcement learning agents in a black-box setting. The attacker is assumed to be able to modify both reward values and transitioned states in the data stream, without knowing the environment's transition probabilities. The poisoning problem is formulated as a constrained optimization (Section 3.2), converted into a penalty-based bilevel optimization, and solved with a stochastic single-loop gradient algorithm (Section 3.3, Algorithm 1). A key assumption is that the attacker knows the exact reachable set S'_s,a for every state-action pair. The method is evaluated in a single maze environment with Q-learning, reporting that the agent learns the target downward policy at state (1,1) with a value gap of 1.0. The paper concludes with an acknowledgment that convergence analysis is left for future work.","tokens_in":10212,"tokens_out":5154,"duration_ms":50308,"significance":"If the claims hold, this work would advance the state of the art by extending poisoning attacks from white-box to black-box environments and by introducing a bilevel-optimization-based attacker that avoids the double-sampling issue. The gradient derivations in Eqs. (8)-(12) are internally consistent, and the algorithmic pipeline (Algorithm 1) is plausible. However, the main claim of operating under unknown dynamics is substantially weakened by Assumption 1, which assumes exact knowledge of the transition support. The lack of convergence guarantees and the limited experimental validation further temper the significance. The manuscript is clearly written and the optimization ideas are interesting, but the evidence provided does not yet establish the full breadth of the claims.","major_comments":[{"comment":"The exact reachable set S'_s,a = {s' : P(s'|s,a) > 0} is load-bearing for the entire method: it defines the modified transition kernel in Eq. (3), is used in the transition-poisoning sampling of Algorithm 1, and appears in the gradient formula (12). Yet the paper provides no estimation procedure, sample-complexity bound, or error analysis for this set in a black-box environment. The text calls the assumption 'mild' and says the information 'can be acquired' before or during the attack, but no method is given. If the attacker's support estimate is wrong---missing a real next state or including an impossible one---the poisoned kernel is no longer a convex combination over the true support, modified transitions may become detectable, and the gradient formulas are no longer valid. The experiment sidesteps this issue by giving the attacker the simulator's true support, so the reported success does not actually validate the black-box claim. A concrete test would be to run the attack with an estimated support (e.g., collected from finite transition data) and report sensitivity to estimation errors.","section":"Section 2, Assumption 1; Eq. (3); Eq. (12); Algorithm 1"},{"comment":"The paper does not provide a convergence analysis for the proposed stochastic bilevel algorithm, and the conclusion explicitly states that this is left for future work. This is a significant gap because the optimization is stochastic, nonconvex, and involves a penalty parameter that is increased heuristically. Moreover, the equivalence between the constrained problem (5) and the bilevel problem (6) is not fully established; the text itself notes that δ's boundedness 'hinders' the equivalence, and Algorithm 1 handles this only through a projection step whose effect on the bilevel analysis is not discussed. Without convergence guarantees or at least a formal statement of the approximation error introduced by the penalty method and the projection, it is unclear whether the iterative updates can be expected to reach a solution of the original problem (4), and the reported experimental success could be a consequence of the specific maze structure rather than the algorithm's general properties.","section":"Section 3.3, Eq. (6), Algorithm 1; Conclusion"},{"comment":"The experimental validation is too limited to support the central claims. Only one maze environment, five repeated experiments, and a Q-learning agent are considered, despite the abstract's claim that the attack works for a 'flexible reinforcement learning algorithm.' Key hyperparameters---the step-size sequences α_k, β_k, λ_k, and the penalty schedule ρ_Φ,k---are not reported, making the results non-reproducible. There is no comparison with baselines (e.g., random poisoning, reward-only attacks, white-box attacks), no evaluation of the attack's stealthiness in terms of detectability, and no quantitative attack-success rate across states or episodes. The success claim rests almost entirely on the value gap at a single state-action pair (1,1). A more comprehensive evaluation across multiple environments, RL algorithms, and random seeds is necessary to substantiate the main contributions.","section":"Section 4"}],"minor_comments":[{"comment":"The sentence 'δ is component in a simplex set' is unclear; δ is a vector in [0,1]^{|S||A|}, which is a hypercube, not a simplex in the usual sense. Please rephrase.","section":"Section 3.1"},{"comment":"The update rule for Q_{k+1}_{s'_i,\\pi_{s'_i}} (the second line of step 6) is not derived in the text. Please clarify how it relates to the stochastic gradient of Eq. (11), particularly to the inverse-probability estimation described in Section 3.3.","section":"Algorithm 1, step 6"},{"comment":"The description says the target policy 'guides the agent to navigate towards the destination while traversing the gray grids,' but gray grids are described as undesirable and incurring a penalty. Please clarify whether the target policy intentionally visits gray grids or the wording is inaccurate.","section":"Section 4, Figure 2(a)"},{"comment":"The penalty function Φ(x) = 1(x>0)x is used, but the text does not explain why a one-sided penalty (rather than a squared hinge or other form) is chosen. A brief justification would improve readability.","section":"Section 3.2, Eq. (5)"},{"comment":"The claim that Xu et al. (2022) 'fails to provide a viable scheme for modifying environment's hyper-parameters' is vague; please specify what hyper-parameters are meant and why the scheme is not viable.","section":"Section 1.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is interesting and the optimization formulation is carefully derived, but the central claim of a black-box attack is currently supported only by an assumption that essentially gives away a key property of the unknown environment. The lack of convergence analysis and the thin experimental validation further weaken the contribution. I would encourage the authors to either (a) provide a concrete procedure for obtaining S'_s,a with finite-sample guarantees and robustness to support errors, or (b) reframe the contribution to explicitly state that the reachable-set oracle is an assumption of the model. A revision that addresses these points would make the paper much stronger."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this paper has a real new idea—poisoning observed next states by replacing them with uniform draws from the reachable set with probability δ—and the optimization machinery around it is coherent. That said, the headline claim ('black-box') is stronger than the assumptions support. The attacker must know the exact reachable set S'_s,a for every state-action pair, and the paper offers no estimator, no error analysis, and no robustness result for that set.\n\nWhat's genuinely new: the transitioned-state poisoning scheme of Eq. (3), the constrained formulation (4), and the bilevel single-loop SGD in Algorithm 1 that avoids the double-sampling issue. The gradient derivations (8)–(12) check out, and the experiment shows the tabular Q-learning agent's Q-values converging to the attacker's target with the promised gap of 1.0. That is honest proof-of-concept.\n\nThe soft spots are real but not fatal. The reachable-set assumption is load-bearing: it appears in the poisoned kernel, in Algorithm 1's sampling and δ update, and in the gradient formula (12), where the second term subtracts the uniform expectation over S'_s,a. If the set is wrong, the stealthiness constraint is violated—you can send the agent to impossible states—and the gradients are biased. The paper calls the assumption 'mild' and says the set can be learned, but gives no procedure or sample-complexity bound. This should be reframed as 'attacker knows the support of the transition kernel,' not 'black-box.' Other gaps: no comparison to Rakhsha et al. 2021 or Xu et al. 2022, one maze environment only, no code, and convergence analysis deferred. These are addressable in revision.\n\nWho this is for: security and RL researchers who want a concrete attack construction for steering Q-learning agents. It is not a field-changer, but it is a useful building block.\n\nRecommendation: send to peer review. The math is internally consistent and the mechanism is original. A serious referee should ask for an honest characterization of the reachable-set assumption, a comparison to existing black-box attacks, and at least one experiment or sensitivity analysis on support errors.","headline":"A genuinely new transition-poisoning mechanism with a coherent bilevel attack objective, but the 'black-box' claim rests on an unanalyzed reachable-set oracle.","tokens_in":10741,"tokens_out":2458,"would_cite":false,"duration_ms":23284,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A black-box attacker who knows only the reachable set of each state-action pair can poison rewards and transitioned states online and force a reinforcement-learning agent to converge to a preset deterministic policy.","keywords":["data poisoning attack","reinforcement learning","black-box environment","bilevel optimization","policy induction","reward poisoning","transition poisoning","man-in-the-middle attack"],"falsifier":"Take a small finite MDP whose transition probabilities are known to the experimenter, compute the exact solution of the constrained problem with full model access, and then run Algorithm 1 with only the reachable sets; if the agent's final Q-values do not show the prescribed gap $\\epsilon$ at every state or the learned policy differs from the target, the claimed black-box steering capacity fails.","tokens_in":9704,"feed_emoji":"🎯","tokens_out":9123,"duration_ms":78902,"temperature":0.7,"pith_summary":"The paper tries to establish that a man-in-the-middle attacker with no knowledge of the environment's transition probabilities can still steer a black-box reinforcement-learning agent toward a target policy by rewriting rewards and replacing transitioned states. The attack is online, running while the agent trains, and it requires only the set of reachable next states for each state-action pair. The authors cast the poisoning task as a constrained optimization problem, turn it into a penalty-based bilevel program so that single transition samples suffice for gradient estimates, and test it in a maze where the agent's learned Q-values converge to the attacker's induced values with the prescribed gap. If the claim is right, hiding the transition model is not enough to protect an RL agent from targeted poisoning.","feed_headline":"Online attack steers black-box RL agent to a preset policy","feed_subtitle":"An attacker who knows only reachable states rewrites rewards and transitions to force a preset policy.","key_machinery":"The load-bearing identity is the modified transition kernel $\\bar{P}_\\delta(s'|s,a) = (1-\\delta_{s,a})P(s'|s,a) + \\delta_{s,a}\\mathbf{1}(s'\\in S'_{s,a})/|S'_{s,a}|$, which expresses transition poisoning as a convex combination of the unknown true kernel and a uniform distribution over the reachable set $S'_{s,a}$, so gradients can be formed from sample draws rather than from knowing $P$. Around this kernel, the paper builds a bilevel program whose lower-level variable is the poisoned reward $\\bar{r}_{s,a}(\\delta,\\bar{Q}) = \\bar{Q}_{s,a} - \\gamma\\sum_{s'}\\bar{P}_\\delta(s'|s,a)\\bar{Q}_{s',\\bar{\\pi}_{s'}}$; this choice converts the Bellman equality into the lower-level optimality condition and keeps the double-sampling issue out of the upper-level gradients. Algorithm 1 then alternates updates of poisoned rewards, induced Q-values, and poisoning intensities $\\delta$, with the $\\delta$-update projected onto $[0,1]$ and the Q-update targeting a value gap of $\\epsilon$ between the target action and all others.","core_discovery":"Poisoning both rewards and transitioned states, rather than only rewards, lets an attacker who is otherwise blind to the environment's dynamics push a Q-learning-style agent toward a predefined deterministic target policy. The poisoned transition kernel is a convex combination of the true kernel and a uniform distribution over the reachable set, controlled by an intensity $\\delta$, and the Bellman consistency of the induced Q-values becomes a lower-level optimality condition in a bilevel program whose lower-level variable is the poisoned reward. Because the lower level is solved in closed form, the upper-level gradients can be estimated from sampled transitioned states without double sampling. The maze experiment is the offered evidence: at the critical state (1,1), the agent's learned Q-values converge to the attacker's variable $\\bar{Q}$ and the target action wins by the designed gap $\\epsilon=1.0$.","pith_inferences":["Extension: the paper defers convergence analysis, so a randomized MDP benchmark varying the discount factor, state-space size, and reachable-set cardinality would show how reliably the promised gap $\\epsilon$ is achieved before any general guarantee can be inferred.","Extension: the method's exposed boundary is exactness of the reachable set; misspecifying $S'_{s,a}$ by adding or deleting a reachable state would shift the uniform term in the kernel and likely break the induced Bellman fixed point, so a sensitivity study would measure how much error the attack tolerates.","Extension: the same bilevel trick of keeping the equality-constrained variable in the lower level could be reused for other poisoning objectives with equality constraints, such as offline or multi-agent poisoning, where a direct penalty would create double-sampling terms."],"forward_implications":["An attacker can force a specific target action at a specific state, not just degrade overall performance, using only support information about the transition kernel, and the induced Q-value gap can be set to any desired $\\epsilon$.","The bilevel reformulation lets the attacker maintain Bellman consistency with a single sampled transition per update, avoiding the need for a simulator that can draw two independent samples from the same state-action pair.","The attack is online and agnostic to the agent's learning algorithm, so it can be interleaved with normal agent-environment interaction rather than requiring a separate offline poisoning phase.","The weight $\\rho_\\delta$ trades off between how much the poisoned rewards deviate and how intensely transitioned states are replaced, giving the attacker a dial for how detectable each poisoning channel is."],"supporting_citations":[{"why":"Supplies the policy-induction formulation, including restricting target policies to be deterministic, that the paper adopts.","marker":"Ma et al. (2019)"},{"why":"The white-box transition-poisoning baseline whose known-P assumption this work removes.","marker":"Rakhsha et al. (2020)"},{"why":"Identifies the double-sampling issue and the primal-dual alternative that motivate the paper's bilevel reformulation.","marker":"Dai et al. (2018)"},{"why":"Provides the two-timescale stochastic bilevel optimization machinery behind the single-loop gradient updates.","marker":"Hong et al. (2023)"},{"why":"Establishes the man-in-the-middle attack setting that the paper extends to online black-box poisoning.","marker":"Kiourti et al. (2020)"},{"why":"An online attack against unknown environments with a heuristic phase split; the paper contrasts its optimization-based approach with this.","marker":"Rakhsha et al. (2021)"},{"why":"Presents a black-box method for poisoning rewards and transition probabilities but, per the paper, lacks a viable scheme for changing the environment's hyper-parameters.","marker":"Xu et al. (2022)"}],"fun_headline_variants":["Black-box attacker poisons rewards and transitions to hijack RL agent","Poison both rewards and transitions: black-box RL attack","Blind attacker reprograms RL agent by poisoning environment","Black-box bilevel poisoning forces RL agent to target policy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the attacker knows exactly which next states are reachable from each state-action pair, even though the probabilities of those transitions can remain hidden; the paper does not analyze what happens when this set is misspecified.","fun_headline_variants_meta":{"raw":{"variants":["Black-box attacker poisons rewards and transitions to hijack RL agent","Poison both rewards and transitions: black-box RL attack","Blind attacker reprograms RL agent by poisoning environment","Black-box bilevel poisoning forces RL agent to target policy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001022,"raw_usage":{"total_tokens":4281,"prompt_tokens":884,"completion_tokens":3397,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":3329}},"tokens_in":500,"tokens_out":3397,"duration_ms":22490,"temperature":1.0,"reasoning_tokens":3329,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:00:25.102695+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a small finite MDP whose transition probabilities are known to the experimenter, compute the exact solution of the constrained problem with full model access, and then run Algorithm 1 with only the reachable sets; if the agent's final Q-values do not show the prescribed gap $\\epsilon$ at every state or the learned policy differs from the target, the claimed black-box steering capacity fails.","supporting_citations":[{"cited_title":"Policy teaching via environment poisoning: Training-time adversarial attacks against reinforcement learning","cited_arxiv_id":null,"evidence_quote":"The white-box transition-poisoning baseline whose known-P assumption this work removes."},{"cited_title":"Sbeed: Convergent reinforcement learning with nonlinear function approximation","cited_arxiv_id":null,"evidence_quote":"Identifies the double-sampling issue and the primal-dual alternative that motivate the paper's bilevel reformulation."},{"cited_title":"Trojdrl: evaluation of backdoor attacks on deep reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Establishes the man-in-the-middle attack setting that the paper extends to online black-box poisoning."}],"review_version":1}