REVIEW 3 major objections 6 minor 25 references
Collapsing Sequence-Level Data-Policy Coverage via Poisoning Attack in Offline Reinforcement Learning
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Poisoning just 1% of an offline RL dataset can cut agent performance by about 90%.
desk verdict Interesting empirical poisoning attack, but the exponential-amplification theory rests on an invalid step and needs a rewrite or removal. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the sequence-level concentrability coefficient $C_\tau$. The identity $d^\pi(\tau)/\mu(\tau) = \prod_{t=0}^{l-1} \pi(a_t|s_t)/\mu(a_t|s_t)$ (valid when transition probabilities match under both policies) turns sequence coverage into a product of per-step policy ratios, and the bound $C_\tau \le C^l$ is what turns small data manipulations into exponential error growth. On the attack side, the machinery is decision-pattern extraction: k-means clustering of encoded state-action pairs yields decision units, consecutive repeats are removed to form patterns, and the rarest patterns are the targets. Perturbing state-action pairs under a small $\ell_\infty$ constraint so that rare patterns become frequent ones reduces coverage without creating detectable outliers.
What would settle it
Compute the empirical sequence-level concentrability $C_\tau$ and the single-step $C$ on a real offline dataset (for example Walker2D with the trained clean policy), and check whether $C_\tau \le C^l$ holds for the sequences the attack targets; if any targeted sequence has $C_\tau > C^l$, the exponential bound as stated is false. Alternatively, removing rare decision patterns but replacing them with synthetically balanced coverage—so no patterns disappear—should largely neutralize the attack if rare-pattern coverage is the mechanism.
Extended reading notes
Core claim
The paper's central claim is that insufficient sequence-level data-policy coverage exponentially amplifies value-estimation error and that an attacker can exploit this by erasing rare decision patterns. Concretely, it defines the sequence-level concentrability coefficient $C_\tau = \sup_\tau d^\pi(\tau)/\mu(\tau)$, shows that when the single-step concentrability $C$ bounds every per-step action ratio the product bound $C_\tau \le C^l$ holds, and derives the Q-error bound $E_{\tau\sim d^\pi}[\sum_{t=0}^{l-1} \gamma^t (Q^{\pi^*}-Q^\pi)] \le (2R_{\max} C^l \epsilon)/(1-\gamma)$. The experimental claim is that a 1% poisoning rate with 5% perturbation magnitude reduces average cumulative reward by 83% using raw state-action features and by 90% using learned features, across Walker2D, Hopper, Half-Cheetah, and Carla-Lane tasks, for CQL, BEAR, BCQ, and BC algorithms.
Load-bearing premise
The amplification bound rests on assuming that the single-step concentrability $C$, a supremum over discounted occupancy ratios, also bounds every instantaneous action ratio $\pi(a_t|s_t)/\mu(a_t|s_t)$ at each time step, so that multiplying $l$ copies of $C$ bounds the $l$-step ratio; a policy can deviate strongly on states it rarely visits, which would break the product bound.
Editorial extensions
If this is right
- Rare multi-step decision patterns are a security weak point: eliminating them amplifies distributional shift even when the perturbations are tiny.
- The attack transfers across offline RL algorithms, including behavior cloning, because it targets data coverage rather than any specific learning rule.
- Algorithms that use multi-step returns, such as n-step CQL, degrade more under the same poisoning budget, consistent with the sequence-level mechanism.
- The attack is stealthy: anomaly-value detection does not separate poisoned from clean samples, so coverage-aware defenses rather than simple outlier filters are needed.
- Limited data access still hurts: with only 1% of the dataset, the attack retains about 86% of the full-data effectiveness.
Reading between the lines
- If the amplification claim holds, dataset auditing for offline RL should track rare multi-step patterns, not just state-action counts; a single rare but task-critical behavior pattern could be a liability even when aggregate coverage looks healthy.
- A direct test of the mechanism would be to oversample or synthesize the rare patterns instead of poisoning them; if performance recovers, the coverage mechanism rather than the perturbation itself is what matters.
- The stated assumption that $C$ bounds every per-step ratio may fail in practice; measuring per-step ratios directly on the trained policy and comparing $C_\tau$ with $C^l$ would show how tight the bound really is.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a sequence-level concentrability coefficient Cτ for offline reinforcement learning and claims that insufficient sequence-level coverage exponentially amplifies the upper bound on Q-value estimation errors, Cτ ≤ C^l and hence a bound of order 2Rmax C^l ε/(1−γ) (Eqs. (5)–(6)). It then proposes the CSDPC poisoning attack, which discretizes state-action pairs into clusters, extracts de-duplicated decision patterns, identifies rare patterns, and perturbs the dataset so that those patterns disappear. Experiments on D4RL MuJoCo and Carla tasks report that poisoning 1% of the data with 5% perturbation magnitude reduces average returns by about 83%–90%, and that the attack evades a gradient-based anomaly detector. The central theoretical claim is the exponential amplification; the empirical attack is presented as validating and exploiting that claim.
Significance. If the exponential-amplification theorem were correct, it would give a new theoretical handle on distributional shift in offline RL and a principled basis for a poisoning attack. The empirical study is substantial: it covers four environments, four offline RL algorithms, and multiple ablations (sequence length, poisoning rate, cluster number, data access, perturbation size, deduplication), and it includes a comparison with alternative attack selection methods. The reported attack effectiveness is high and the stealth evaluation is a useful addition. However, the theoretical contribution is not established: the key inequality in Eq. (5) rests on an invalid pointwise inference from a supremum of occupancy ratios, and the attack's rare-pattern heuristic is not independently tied to the defined coverage coefficient. The experimental results, while suggestive, cannot substitute for the missing theoretical validation. Credit should be given for the breadth and internal consistency of the experiments and for the explicit threat model; no proof artifacts or code release are mentioned.
major comments (3)
- [§3.2, Eq. (5)] The step "According to Equation 1, C is the supremum, implying that for each time step π(at|st)/µ(at|st) ≤ C" is invalid. The concentrability coefficient C is defined as a supremum of discounted occupancy ratios dπ(s,a)/µ(s,a), not as a bound on the instantaneous policy ratio. Decomposing dπ(s,a)/µ(s,a) = [dπ(s)/µ(s)] · [π(a|s)/µ(a|s)] shows that a state rarely visited by π relative to µ can support an arbitrarily large action ratio π(a|s)/µ(a|s) while keeping the occupancy ratio and hence C small. Therefore the product bound Cτ ≤ C^l does not follow, and the exponential amplification bound in Eq. (6) is unsupported. A correct argument would need C defined as sup_{s,a} π(a|s)/µ(a|s) (with appropriate support assumptions) or a separately justified finite-horizon sequence concentrability bound.
- [§4.3 and §4.4] The attack's selection of rare decision patterns is asserted to target regions of insufficient coverage ("these patterns correspond to the sequence-level sparse regions identified in the theoretical analysis"), but no formal or empirical link is established between a pattern's empirical frequency and the ratio dπ(τ)/µ(τ). The experiments in Table 1 and Figure 2 show that the heuristic degrades performance more than random or Q-value selection, but they do not measure Cτ or dπ/µ, so they cannot distinguish coverage collapse from generic distributional corruption. Since the paper's contribution is precisely the theory-guided attack, this missing validation is load-bearing.
- [§3.2, Eq. (4)] The derivation of dπ(τ)/µ(τ) = ∏_{t<l} π(at|st)/µ(at|st) cancels the transition probabilities P(st+1|st,at), but this cancellation is valid only if the initial-state distributions of dπ and µ are identical. The paper does not state or justify this assumption; if the data collection and target policies start from different initial-state distributions, a factor ρπ(s0)/ρµ(s0) remains in the ratio. This missing condition is another gap in the route from Eq. (4) to Eq. (5).
minor comments (6)
- [§4.4] The sentence "By increasing the occurrence of these rare patterns, thereby significantly minimize sequence-level coverage" appears to state the opposite of the intended operation; the attack should transform rare patterns into frequent ones, which reduces the presence of rare patterns and lowers coverage.
- [§5.2] The phrase "increased from 0.5 to 0.25" should read "increased from 0.05 to 0.25" to match the reported perturbation magnitudes.
- [§5.1] The attack name is given as "Collapsing Decision Pattern Diversity (CSDPC)" in Section 5.1, which is inconsistent with the title and abstract's "Collapsing Sequence-Level Data-Policy Coverage (CSDPC)".
- [Eq. (1)] The term "concentratability coefficient" is a typo for "concentrability coefficient".
- [§5.2] The comparison "as effective as the 10% poisoning rate in BAFFLE" is difficult to interpret because BAFFLE is a backdoor attack with different assumptions about trigger control; the line should state the conditions under which the comparison holds.
- [Appendix B] The paper does not mention whether code or datasets are released; providing the attack code would materially improve reproducibility.
Circularity Check
No circular derivation: the sequence-level concentrability bound is an attempted theorem rather than a restatement of its inputs, and the CSDPC attack is an independent empirical heuristic.
full rationale
I walked the claimed derivation chain. Equation (4) is an exact decomposition of the trajectory ratio into a product of stepwise action ratios. Equation (5) then tries to bound C_tau by C^l using the assertion that C being a supremum of occupancy ratios implies a per-step bound on pi(a|s)/mu(a|s); that inference is invalid because an occupancy ratio includes the state-occupancy factor d_pi(s)/mu(s), so the exponential-amplification theorem is not established. Invalidity, however, is not circularity: the bound is not assumed as the conclusion, and C_tau is a new definition rather than a restatement of C^l. Equation (6) substitutes C_tau into the standard single-step bound, again a derivation step rather than an equivalence by construction. The CSDPC attack selects patterns by empirical frequency, not by d_pi/mu, and its success is measured by AER against clean agents, so the experiments do not feed the theorem's conclusion back into its derivation. All mathematical citations (Munos 2003; Chen and Jiang 2019) are external to the present authors, and no load-bearing step is justified by a self-citation. The Discussion's independence assumption is an acknowledged limitation of the theory, not a circular dependency. The central claim therefore has independent content, and no reduction of a prediction to its inputs is present.
Assumptions & free parameters
free parameters (5)
- number of clusters k =
k=8 (Walker2D, elbow method)
- sequence length l =
l=5
- poisoning rate rho =
rho=1%, 5%, 10%, 20%
- perturbation ratio eta =
eta=0.05
- encoder network =
trained on clean offline data
assumptions (4)
- ad hoc to paper C = sup_{s,a} dπ(s,a)/μ(s,a) implies π(a|s)/μ(a|s) ≤ C for every state-action pair
- domain assumption Transition probabilities cancel in the sequence ratio dπ(τ)/μ(τ) because the same dynamics generate data under both policies
- ad hoc to paper Rare decision patterns in the dataset correspond to regions where dπ/μ is large and thus to insufficient coverage
- domain assumption Stepwise independence of coverage errors (no correlated perturbations)
Cite this review
Pith. "Pith review of Collapsing Sequence-Level Data-Policy Coverage via Poisoning Attack in Offline Reinforcement Learning." pith.science (2026). https://pith.science/paper/EAKM3XEN
@misc{pith2026250611172,
author = {Pith},
title = {Pith review of: Collapsing Sequence-Level Data-Policy Coverage via Poisoning Attack in Offline Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/EAKM3XEN}},
note = {Machine review of arXiv:2506.11172}
}
read the original abstract
Offline reinforcement learning (RL) heavily relies on the coverage of pre-collected data over the target policy's distribution. Existing studies aim to improve data-policy coverage to mitigate distributional shifts, but overlook security risks from insufficient coverage, and the single-step analysis is not consistent with the multi-step decision-making nature of offline RL. To address this, we introduce the sequence-level concentrability coefficient to quantify coverage, and reveal its exponential amplification on the upper bound of estimation errors through theoretical analysis. Building on this, we propose the Collapsing Sequence-Level Data-Policy Coverage (CSDPC) poisoning attack. Considering the continuous nature of offline RL data, we convert state-action pairs into decision units, and extract representative decision patterns that capture multi-step behavior. We identify rare patterns likely to cause insufficient coverage, and poison them to reduce coverage and exacerbate distributional shifts. Experiments show that poisoning just 1% of the dataset can degrade agent performance by 90%. This finding provides new perspectives for analyzing and safeguarding the security of offline RL.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[3]
Alexey Dosovitskiy, Germán Ros, Felipe Codevilla, Anto- nio M. López, and Vladlen Koltun. Carla: An open urban driving simulator. In1st Annual Conference on Robot Learning, CoRL 2017, Mountain View, California, USA, November 13-15, 2017, Proceedings, pages 1–16. PMLR,
work page 2017
-
[8]
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets.arXiv preprint arXiv:2006.09359, 2020a. Aviral Kumar, Aurick Zhou, G. Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learn- ing. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’...
arXiv 2006
-
[9]
Tactics of adversarial attack on deep reinforcement learning agents
Yen-Chen Lin, Zhang-Wei Hong, Yuan-Hong Liao, Meng- Li Shih, Ming-Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. In Carles Sierra, editor,Proceedings of the Twenty-Sixth Interna- tional Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 3756–3762. ijcai.org,
work page 2017
-
[10]
Amin Rakhsha, Goran Radanovic, Rati Devidze, Xiaojin Zhu, and Adish Singla. Policy teaching via environment poisoning: Training-time adversarial attacks against re- inforcement learning. InProceedings of the 37th Inter- national Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, pages 7974–7984. PMLR,
work page 2020
-
[11]
Understanding the limits of poisoning attacks in episodic reinforcement learning
Anshuka Rangi, Haifeng Xu, Long Tran-Thanh, and Mas- simo Franceschetti. Understanding the limits of poisoning attacks in episodic reinforcement learning. In Luc De Raedt, editor,Proceedings of the Thirty-First Interna- tional Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 3394–3400. ijcai.org,
work page 2022
-
[13]
SoK: Adversarial Machine Learning Attacks and Defences in Multi-Agent Reinforcement Learning
Maxwell Standen, Junae Kim, and Claudia Szabo. Sok: Adversarial machine learning attacks and defences in multi-agent reinforcement learning.arXiv preprint arXiv:2301.04299,
-
[14]
Stealthy and effi- cient adversarial attacks against deep reinforcement learn- ing
Jianwen Sun, Tianwei Zhang, Xiaofei Xie, Lei Ma, Yan Zheng, Kangjie Chen, and Yang Liu. Stealthy and effi- cient adversarial attacks against deep reinforcement learn- ing. InThe Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, New York, NY, USA, February 7-12, 2020, pages 5883–5891. AAAI Press,
work page 2020
-
[15]
Vulnerability- aware poisoning mechanism for online RL with unknown dynamics
Yanchao Sun, Da Huo, and Furong Huang. Vulnerability- aware poisoning mechanism for online RL with unknown dynamics. In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7,
work page 2021
Show all 25 references
-
[16]
Who is the strongest enemy? towards optimal and efficient evasion attacks in deep RL
Yanchao Sun, Ruijie Zheng, Yongyuan Liang, and Furong Huang. Who is the strongest enemy? towards optimal and efficient evasion attacks in deep RL. InThe Tenth Inter- national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29,
2022
-
[19]
Tucker, and Ofir Nachum
Yifan Wu, G. Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.ArXiv, abs/1911.11361,
1911 arXiv
-
[20]
Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y . Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model-based offline policy optimization. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Had- sell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances ...
2020
-
[21]
Liu, Duane S
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan D. Liu, Duane S. Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. InAdvances in Neu- ral Information Processing Systems 33: Annual Confer- ence on Neural...
2020 arXiv
-
[22]
Adaptive reward-poisoning attacks against reinforcement learning
Xuezhou Zhang, Yuzhe Ma, Adish Singla, and Xiaojin Zhu. Adaptive reward-poisoning attacks against reinforcement learning. InProceedings of the 37th International Con- ference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, pages 11225–11234. PMLR, 2020b. Collap...
2020
-
[24]
We use the official open-source code of these algorithms and follow the settings by D4RL [Fu et al., 2020]
B.3 OFFLINE RL ALGORITHMS To evaluate how offline RL algorithms perform under our poisoning framework, we employed three algorithms for training agent: Batch-Constrained Q-learning (BCQ) [Fujimoto et al., 2018], Batch-Ensemble Actor-Critic with Retrace (BEAR) [Kumar et al., 20...
2018
-
[25]
C.2 ADDITIONAL COMPARATIVE RESULTS Although the online attack method cannot be applied to offline RL due to the need to interact with the environment in real-time to obtain training parameters, however, in order to more fully evaluate the performance of the CSDPC method, we Ta...
2022
-
[2008]
Morel: Model-based offline rein- forcement learning
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline rein- forcement learning. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors,Advances in Neural Information Pro- ce...
2020
-
[2012]
Hunt, and Mingyuan Zhou
Zhendong Wang, Jonathan J. Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,
2023
-
[2017]
Tucker, and Sergey Levine
Justin Fu, Aviral Kumar, Ofir Nachum, G. Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven rein- forcement learning.ArXiv, abs/2004.07219,
2004 arXiv
-
[2018]
Mujoco: A physics engine for model-based control.2012 IEEE/RSJ International Conference on Intelligent Robots and Sys- tems, pages 5026–5033,
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control.2012 IEEE/RSJ International Conference on Intelligent Robots and Sys- tems, pages 5026–5033,
2012
-
[2019]
Decision S4: efficient sequence-based RL via state spaces layers
Shmuel Bar-David, Itamar Zimerman, Eliya Nachmani, and Lior Wolf. Decision S4: efficient sequence-based RL via state spaces layers. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,
2023
-
[2020]
A minimalist ap- proach to offline reinforcement learning
Scott Fujimoto and Shixiang Shane Gu. A minimalist ap- proach to offline reinforcement learning. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors,Advances in Neural Information Processing Systems 34: Annual Confere...
2021
-
[2021]
Off- policy deep reinforcement learning without exploration
Scott Fujimoto, David Meger, and Doina Precup. Off- policy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov, edi- tors,Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Ca...
2019
-
[2022]
Reinforcement learning with sim- ple sequence priors
Tankred Saanum, Noémi Élteto, Peter Dayan, Marcel Binz, and Eric Schulz. Reinforcement learning with sim- ple sequence priors. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors,Advances in Neural Information Pro- cessing System...
2023
-
[2023]
Julian, Chelsea Finn, and Sergey Levine
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jacob Varley, Alex Irpan, Benjamin Eysenbach, Ryan C. Julian, Chelsea Finn, and Sergey Levine. Actionable models: Unsupervised offline rein- forcement learning of robotic skills. In Marina Meila and Tong Zha...
2021
-
[3090]
For a comprehensive evaluation of our attack framework, we selected four complex continuous tasks from the offline environment D4RL [Fu et al., 2020]: Walker2D, Hopper, and Half-Cheetah in the MuJoCo robot simulator [Todorov et al., Table 8: The hyperparameter settings for alg...
2020
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.