Pith. sign in

REVIEW 3 major objections 6 minor 65 references

Multi-level Certified Defense Against Poisoning Attacks in Offline Reinforcement Learning

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proves that differentially private training of offline reinforcement learning policies yields certified lower bounds on expected cumulative reward that survive poisoning of up to 7.17% of trajectories — roughly 900 times more…

desk verdict A clean theoretical extension of DP outcome guarantees to offline RL certificates, but the headline empirical claims are unsupported because the privacy accounting and training hyperparameters are never reported. read the letter →

arxiv 2505.20621 v1 pith:YS36OPKZ submitted 2025-05-27 cs.LG cs.AI

Shijie Liu , Andrew C. Cullen , Paul Montague , Sarah Erfani , Benjamin I. P. Rubinstein This is my paper
classification cs.LGcs.AI
keywords offlinereinforcementlearningdatapoisoningattackscertifiedrobustnessdifferentialprivacyoutcomesguaranteeRényipolicy-levelcertificationaction-level
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Offline reinforcement learning trains policies on datasets gathered by someone else, so a poisoned dataset can silently degrade the learned behavior. This paper claims that if the training algorithm injects differential-privacy noise, the resulting randomized policy carries a computable guarantee: no adversary who alters up to r transitions or whole trajectories can push the expected cumulative reward below a certified floor, and no adversary can flip the action chosen at a certified state. The key move is extending differential privacy's 'outcomes guarantee' from probabilities to expected rewards, which turns a privacy property into a performance bound. Empirically the method certifies a 50% performance floor with up to 7.17% of Freeway trajectories poisoned, versus 0.008% for the earlier COPA approach, with action-level certified radii about five times larger. If the claim holds, safety-critical offline reinforcement learning can be deployed with a worst-case performance promise rather than a hope.

What carries the argument

The load-bearing object is the $(K,r)$-outcomes guarantee (Definition 3.5), a group-privacy form of differential privacy stating that for datasets differing in up to $r$ elements, $\Pr[M(D_1) \in S] \le K(\Pr[M(D_2) \in S])$, with $K(x) = e^{\varepsilon}x + \delta$ for approximate DP and $K(x) = (e^{\varepsilon}x)^{(\alpha-1)/\alpha}$ for Rényi DP. The central move is the Expected Outcomes Guarantee (Lemma 4.1), which converts this bound on probabilities into two-sided bounds on expected values by writing $\mathbb{E}[M(\tilde{D})]$ as the integral of the tail probability $\Pr[M(\tilde{D}) > t]$ via Fubini's theorem, applying the outcomes guarantee inside the integral, and using Hölder's inequality for the Rényi case. The post-processing property of DP then lets the cumulative-reward function inherit the guarantee, producing Theorem 4.2's certified lower bound; on the training side the guarantees are realized by the Sampled Gaussian Mechanism for transition-level poisoning and the DP-FEDA virtual-gradient method for trajectory-level poisoning, while on the estimation side the clean reward $J(\pi)$ is lower-bounded from rollouts via the Dvoretzky–Kiefer–Wolfowitz inequality and per-action scores are bounded by the SIMUEM confidence-interval method.

What would settle it

Recompute the certified curves in Figure 2 using a standard Rényi-DP accountant for the Sampled Gaussian Mechanism with the paper's effective sampling ratio $q \approx 0.033$ and the reported noise multipliers $\sigma$ over the full training-iteration count, then convert to group privacy at $r$ equal to 7.17% of Freeway trajectories (about 70 trajectories) and evaluate the Theorem 4.2 floor $e^{-\varepsilon}(J(\pi) - b\delta)$ at that $r$. If the floor falls below a meaningful fraction of clean reward at the 7.17% poisoning level, the headline certified-performance claim is refuted; if it stays near 50%, the claim stands.

Watch

Extended reading notes

Core claim

The paper's central claim is Theorem 4.2: if the offline-RL training algorithm $M$ preserves a $(K,r)$-outcomes guarantee — a differential-privacy-style condition that for any two datasets differing in up to $r$ transitions or trajectories, the probability that $M$'s output lands in any set $S$ changes only through a function $K$ — then any policy $\tilde{\pi}$ trained on a dataset poisoned up to size $r$ has expected cumulative reward at least $e^{-\varepsilon}(J(\pi) - b\delta)$ under approximate DP and at least $e^{-\varepsilon}(b^{-1/\alpha}J(\pi))^{\alpha/(\alpha-1)}$ under Rényi DP, where $J(\pi)$ is the clean policy's expected reward, bounded in $[0,b]$. The proof runs through a new lemma that converts the probability-level outcomes guarantee into two-sided bounds on expectations by integrating over tail distributions, then applies the post-processing property of differential privacy so that the reward function inherits the guarantee. Because the bound is stated directly in expected cumulative reward, the certification covers discrete and continuous action spaces and stochastic and deterministic environments alike, which the prior COPA approach (limited to discrete, deterministic settings with exhaustive trajectory search) could not. The same $(K,r)$-outcomes guarantee applied to per-state inferred action scores yields action-level certification of stable actions, with the maximum tolerable poisoning radius located by binary search over privacy parameters.

Load-bearing premise

The whole certification rests on one premise — that the noisy training procedures deliver a genuine differential-privacy guarantee with a small enough privacy loss over the entire deep-network training run, since a large privacy loss would collapse the certified reward floor toward zero and take the headline poisoning percentages with it.

Editorial extensions

If this is right

  • Any offline RL algorithm that admits a DP training mechanism — SGM for transition-level attacks or DP-FEDA VG for trajectory-level attacks — inherits a computable certified lower bound on expected cumulative reward, with no restriction to discrete actions or deterministic environments.
  • The two certification layers compose: a state can be certified to keep the same action under poisoning up to $r_t$, while the policy as a whole keeps a reward floor under poisoning up to $r$, protecting both critical decisions and overall performance.
  • Certified strength is tunable through noise: raising $\sigma$ enlarges the certified poisoning fraction at the cost of clean reward, and the Rényi-DP arm of the bound is uniformly tighter than the approximate-DP arm in the reported experiments.
  • At equal certified reward floors, the trajectory-level threat model allows roughly ten times more poisoned transitions than transition-level poisoning, because trajectory corruption is a coarser but more destructive attack.
  • Against concrete trajectory attacks (random and reward-flipping perturbations) the observed rewards sit well above the certified lower bound, consistent with the certification being a worst-case guarantee rather than a prediction of attack success.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial caution: the paper never reports the privacy budget $\varepsilon$, the training-iteration count, the gradient clipping norm, or the reward bound $b$ that feed Theorem 4.2, so the headline certified percentages cannot yet be reproduced from the paper alone; a standard Rényi-DP accountant at the reported sampling ratio ($q \approx 0.033$) and noise multipliers, applied over a full deep-RL
  • Because the proof only needs the certified quantity to be a bounded post-processing of the trained policy, the same construction plausibly extends to other safety metrics — probability of entering a bad state, constraint-violation counts, or worst-case episode cost — whenever those are bounded functions of the policy; the paper itself certifies only expected cumulative reward and per-state action
  • The framework is agnostic to the DP training mechanism, so tighter privacy accountants or improved noise-injection schemes should enlarge certified radii without touching the certification theorems; the conclusions gesture at adaptive noise, and the same logic applies to any future accountant.
  • The appendix's empirical attacks land far above the certified floor, which raises a testable question the paper does not pursue: an adversary who spends the allowed $r$ poisonings on the most reward-sensitive trajectories may approach the certified bound, and whether the bound is approached in practice would reveal how tight the worst-case guarantee really is.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes MuCD, a certified defense for offline reinforcement learning against data poisoning. The framework trains policies with differentially private mechanisms (SGM for transition-level guarantees and DP-FEDA VG for trajectory-level guarantees) and then derives two kinds of certificates: action-level robustness, which certifies that the chosen action at a state is unchanged under up to r poisoned transitions or trajectories, and policy-level robustness, which certifies a lower bound on expected cumulative reward under such poisoning. The central theoretical device is an outcomes-guarantee argument that converts differential privacy into bounds on expected rewards (Lemma 4.1, Theorem 4.2) and on inferred action scores (Theorem 4.4). Experiments on Atari (Freeway, Breakout) and MuJoCo (Half Cheetah) claim that the approach certifies no more than a 50% performance drop with up to 7.17% of Freeway trajectories poisoned and that certified action-level radii are about 5 times larger than those of COPA.

Significance. If the empirical certificates are correct, the paper is a meaningful advance: it extends certified poisoning defenses beyond COPA's discrete/deterministic setting to continuous and stochastic environments, and it provides a general reduction from DP outcomes guarantees to policy-level reward certificates. The theoretical skeleton is elegant and, given the (K,r)-outcomes guarantee, the derivations in Lemma 4.1 and Theorem 4.2 are sound; the post-processing argument is standard and correctly identified. The paper also deserves credit for separating transition-level from trajectory-level threat models and for attempting both action-level and policy-level certifications within one framework. However, the quantitative claims hinge on two issues that are not resolved in the manuscript: an apparent sign error in the DKW-based lower bound on clean reward, and the complete absence of the privacy-accounting parameters needed to verify the reported radii. These issues affect the headline results directly, so the current empirical claims are not verifiable from the manuscript.

major comments (3)
  1. [Section 4.2, Eq. (8)] The DKW lower bound on the expected cumulative reward has a sign error. The text correctly states that the true CDF satisfies F_hat(x) - epsilon <= F_X(x) <= F_hat(x) + epsilon. To lower-bound J(pi) = integral_0^b (1 - F_X(x)) dx, one must use the upper bound on F_X, giving J(pi) >= integral_0^b (1 - F_hat(x)) dx - b*epsilon. As printed, Eq. (8) uses 1 - (F_hat - epsilon) = 1 - F_hat + epsilon, which is an upper bound on 1 - F_X, not a lower bound. Because J r in Eqs. (5) and (6) is increasing in J(pi), this error inflates all policy-level certified lower bounds reported in Figures 2 and 3. The authors need to correct this sign and recompute every policy-level certificate.
  2. [Section 5 and Appendix A.2] The privacy accounting that connects the DP training mechanisms to the plotted certificates is missing. The experiments report the sampling ratio q (batch size 32 over the dataset), the noise multiplier sigma, the number of policy instances p, the number of rollouts m, and delta = 0.001, but they do not report the Rényi order alpha, the per-step RDP epsilon, the number of training iterations T, the clipping norm C, the learning rate, or the cumulative-reward bound b used in Eqs. (5) and (6). Without these, the headline claim of certifying a 50% performance bound with 7.17% of Freeway trajectories poisoned (Section 5.2) cannot be checked. In particular, for trajectory-level DP-FEDA VG with q about 0.033 and sigma = 3 over thousands of training steps, standard RDP composition would yield a group-privacy epsilon for r = 70 that is very large, making e^{-epsilon} in Eq. (6) negligible. The authors must report the full accountant output at each certified radius and either substantiate the 7.17% and 2.05% claims or revise them.
  3. [Section 5.1, Table 1 and Figure 1] The abstract's claim that the method produces certified radii '5 times larger' than COPA is not supported by a controlled comparison. COPA certifies action stability along individual trajectories in deterministic environments with truncated horizons, while MuCD certifies expected behavior of a randomized policy. The paper acknowledges the comparison is 'often implicit,' but the abstract and Section 5.1 state the ratio as though it were a direct empirical improvement. Please specify the exact certification object, trajectory-horizon settings, and threat models used for the ratio, or qualify the claim accordingly.
minor comments (6)
  1. [Section 3.1, Definition 3.1] There is a typo in the definition of trajectory-level poisoning: the symmetric difference is written as |D ⊖tra D~| but should be |D ⊖trj D~|.
  2. [Section 5.1] The text contains a typo: 'noisey' should be 'noisy'.
  3. [Definition 3.5 and surrounding text] The symbol K is used both for the family of functions and for an individual element of that family (e.g., 'for any K in K'). This makes equations such as Eq. (1) harder to parse; consider denoting an element by a lowercase kappa or another symbol.
  4. [Figures 2 and 3] The policy-level certification plots do not label their axes or indicate the scale of the cumulative reward bound b. Adding axis labels and stating b would make the curves interpretable and would also clarify how Eqs. (5) and (6) were evaluated.
  5. [Abstract and Section 5.2] The abstract says 'up to 7% of the training data poisoned,' while Section 5.2 reports 7.17% for Freeway and 2.05% for Breakout. The abstract should specify that the 7% figure is for Freeway under the trajectory-level threat model, and should state which RL algorithm and noise level produce it.
  6. [Table 5 and Appendix A.7] The table heading contains a typo: 'Halfcheetha' should be 'HalfCheetah', and the main text's 'Mujoco' should be 'MuJoCo' for consistency.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the certified bounds follow from the DP outcomes guarantee plus measured clean-policy rewards, not from any fitted or self-referential construction.

full rationale

The derivation chain is self-contained. Theorem 4.2 is a direct application of Definition 3.5's (K,r)-outcomes guarantee and the post-processing property; Lemma 4.1 (and Appendix A.3's proof) supplies the expected-value tail integration and is proved from the ADP/RDP definitions using Fubini and Hölder. The clean expected reward J(pi) entering Equations (5) and (6) is estimated from rollouts of the randomized clean policy via a DKW confidence bound (Section 4.2), not chosen to force the certified value. Action-level certification (Theorem 4.4) likewise follows from the post-processing bound on inferred scores and the SIMUEM confidence intervals. The DP training mechanisms (SGM, DP-FEDA VG) are treated as inputs with standard privacy accounting; no parameter is fitted so that a claimed radius is reproduced. The paper does not report the privacy budget epsilon or training iterations, which is a reproducibility/correctness limitation, but it is not circularity. Self-citations (e.g., Liu et al. 2023 for the outcomes guarantee notion) are background definitions and are not used to forbid alternatives or to supply an unproved load-bearing theorem.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the DP guarantee of the randomized training, but the paper does not disclose the privacy budget epsilon, the training iterations T, or the reward bound b. Without these, the certificates cannot be reproduced or checked, and the reported 7% poisoning tolerance may depend on an overly optimistic privacy accounting.

free parameters (6)
  • noise multiplier sigma = 1.0, 1.5, 2.0, 3.0 across experiments
    Chosen by hand to trade clean performance against certified radius; directly controls the DP privacy budget epsilon.
  • number of policy instances p = 50
    Chosen; larger p tightens the confidence intervals for inferred scores and J(pi).
  • number of rollouts m = 500 (10 per instance)
    Chosen; controls the DKW confidence bound width.
  • training iterations T = not reported
    Critical for privacy accounting; without it the epsilon and hence the certificates cannot be reproduced.
  • clipping norm C = not reported
    Needed for sensitivity calculation in SGM and DP-FEDA VG; missing.
  • cumulative reward bound b = not reported
    Theorem 4.2 requires the cumulative reward to lie in [0,b]; the value of b is not given.
assumptions (5)
  • standard math DP post-processing property: any function applied to the output of a DP mechanism preserves the same DP guarantee.
    Invoked in Theorem 4.2 and Lemma 4.3 to move from the policy output to cumulative reward and inferred scores.
  • standard math Holder's inequality and Fubini's theorem for the expectation bound.
    Used in the proof of Lemma 4.1 (Appendix A.3).
  • standard math Dvoretzky-Kiefer-Wolfowitz inequality bounds the true CDF from the empirical CDF.
    Used in Section 4.2 to lower-bound J(pi) from m rollouts with confidence 1-delta.
  • domain assumption The RL environment has bounded cumulative reward in [0,b] with known b.
    Required for Lemma 4.1 and Theorem 4.2; the paper states this but does not specify b for the experiments.
  • domain assumption SGM and DP-FEDA VG provide the claimed (K,r)-outcomes guarantee for the deep RL training algorithm used.
    Section 4.1 assumes these DP mechanisms apply to DQN/IQL/C51 training; the paper does not report epsilon or T to verify.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-level Certified Defense Against Poisoning Attacks in Offline Reinforcement Learning." pith.science (2026). https://pith.science/paper/YS36OPKZ

@misc{pith2026250520621,
  author       = {Pith},
  title        = {Pith review of: Multi-level Certified Defense Against Poisoning Attacks in Offline Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YS36OPKZ}},
  note         = {Machine review of arXiv:2505.20621}
}
abstract

Similar to other machine learning frameworks, Offline Reinforcement Learning (RL) is shown to be vulnerable to poisoning attacks, due to its reliance on externally sourced datasets, a vulnerability that is exacerbated by its sequential nature. To mitigate the risks posed by RL poisoning, we extend certified defenses to provide larger guarantees against adversarial manipulation, ensuring robustness for both per-state actions, and the overall expected cumulative reward. Our approach leverages properties of Differential Privacy, in a manner that allows this work to span both continuous and discrete spaces, as well as stochastic and deterministic environments -- significantly expanding the scope and applicability of achievable guarantees. Empirical evaluations demonstrate that our approach ensures the performance drops to no more than $50\%$ with up to $7\%$ of the training data poisoned, significantly improving over the $0.008\%$ in prior work~\citep{wu_copa_2022}, while producing certified radii that is $5$ times larger as well. This highlights the potential of our framework to enhance safety and reliability in offline RL.

Figures

Figures reproduced from arXiv: 2505.20621 by the authors.

Figure 1
Figure 1. Stability ratio against the tolerable poisoning threshold [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Policy-level robustness certifications, capturing the lower bound of the expected cumula [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Policy-level robustness certification for the continuous action game Mujoco Half Cheetah, [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Stability ratio versus the tolerable poisoning threshold [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

65 extracted references · 37 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang

    Martín Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep Learning with Differential Privacy . Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp.\ 308--318, October 2016. doi:10.1145/2976749.2978318. URL http://arxiv.org/abs/1607.00133. arXiv: 1607.00133

    arXiv 2016

  3. [3]

    Differentially private policy evaluation

    Borja Balle, Maziar Gomrokchi, and Doina Precup. Differentially private policy evaluation. In International Conference on Machine Learning, pp.\ 2130--2138. PMLR, 2016

    work page 2016

  4. [4]

    Hypothesis Testing Interpretations and Renyi Differential Privacy

    Borja Balle, Gilles Barthe, Marco Gaboardi, Justin Hsu, and Tetsuya Sato. Hypothesis Testing Interpretations and Renyi Differential Privacy . arXiv:1905.09982 [cs, stat], October 2019. URL http://arxiv.org/abs/1905.09982. arXiv: 1905.09982

    work page Pith review arXiv 1905

  5. [5]

    Defense Against Reward Poisoning Attacks in Reinforcement Learning

    Kiarash Banihashem, Adish Singla, and Goran Radanovic. Defense Against Reward Poisoning Attacks in Reinforcement Learning , June 2021. URL http://arxiv.org/abs/2102.05776. arXiv:2102.05776 [cs]

    work page Pith review arXiv 2021

  6. [6]

    Can M achine L earning be S ecure? In Proceedings of the 2006 ACM S ymposium on I nformation, C omputer and C ommunications S ecurity , pp.\ 16--25, 2006

    Marco Barreno, Blaine Nelson, Russell Sears, Anthony D Joseph, and J Doug Tygar. Can M achine L earning be S ecure? In Proceedings of the 2006 ACM S ymposium on I nformation, C omputer and C ommunications S ecurity , pp.\ 16--25, 2006

    work page 2006

  7. [7]

    A Distributional Perspective on Reinforcement Learning

    Marc G Bellemare, Will Dabney, and R \'e mi Munos. A Distributional Perspective on Reinforcement Learning . In International Conference on Machine Learning, pp.\ 449--458. PMLR, 2017

    work page 2017

  8. [8]

    Poisoning Attacks against Support Vector Machines

    Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning Attacks against Support Vector Machines . arXiv:1206.6389 [cs, stat], March 2013. URL http://arxiv.org/abs/1206.6389. arXiv: 1206.6389

    arXiv 2013

Show all 65 references
  1. [9]

    Double Bubble, Toil and Trouble: Enhancing Certified Robustness Through Transitivity

    Andrew Cullen, Paul Montague, Shijie Liu, Sarah Erfani, and Benjamin Rubinstein. Double Bubble, Toil and Trouble: Enhancing Certified Robustness Through Transitivity . Advances in Neural Information Processing Systems, 35: 0 19099--19112, 2022

    2022

  2. [10]

    Cullen, Paul Montague, Shijie Liu, Sarah M

    Andrew C. Cullen, Paul Montague, Shijie Liu, Sarah M. Erfani, and Benjamin I. P. Rubinstein. It’s S implex! D isaggregating M easures to I mprove C ertified R obustness. In 2024 IEEE Symposium on Security and Privacy (SP), pp.\ 2886--2900, 2024 a . doi:10.1109/SP54263.2024.00065

    2024

  3. [11]

    Et T u C ertifications: R obustness C ertificates Y ield B etter A dversarial E xamples

    Andrew Craig Cullen, Shijie Liu, Paul Montague, Sarah Monazam Erfani, and Benjamin IP Rubinstein. Et T u C ertifications: R obustness C ertificates Y ield B etter A dversarial E xamples. In International Conference on Machine Learning, 2024 b

    2024

  4. [12]

    Unifying pac and regret: Uniform pac bounds for episodic reinforcement learning

    Christoph Dann, Tor Lattimore, and Emma Brunskill. Unifying pac and regret: Uniform pac bounds for episodic reinforcement learning. Advances in Neural Information Processing Systems, 30, 2017

    2017

  5. [13]

    Robust Estimators in High Dimensions without the Computational Intractability , March 2019

    Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Ankur Moitra, and Alistair Stewart. Robust Estimators in High Dimensions without the Computational Intractability , March 2019. URL http://arxiv.org/abs/1604.06443. arXiv:1604.06443 [cs, math, stat]

    2019 arXiv

  6. [14]

    Asymptotic Minimax Character of the Sample Distribution Function and of the Classical Multinomial Estimator

    Aryeh Dvoretzky, Jack Kiefer, and Jacob Wolfowitz. Asymptotic Minimax Character of the Sample Distribution Function and of the Classical Multinomial Estimator . The Annals of Mathematical Statistics, 27 0 (3): 0 642--669, 1956

    1956

  7. [15]

    Calibrating Noise to Sensitivity in Private Data Analysis

    Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating Noise to Sensitivity in Private Data Analysis . In Theory of cryptography conference , pp.\ 265--284. Springer, 2006

    2006

  8. [16]

    Data Mining with Differential Privacy

    Arik Friedman and Assaf Schuster. Data Mining with Differential Privacy . In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining , pp.\ 493--502, 2010

    2010

  9. [17]

    D4RL: Datasets for Deep Data-Driven Reinforcement Learning , 2020

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for Deep Data-Driven Reinforcement Learning , 2020

    2020

  10. [18]

    Sugli integrali multipli

    Guido Fubini. Sugli integrali multipli. Rendiconti del Circolo Matematico di Palermo, 24 0 (1): 0 97--155, 1907. doi:10.1007/BF03014329

    1907 doi

  11. [19]

    Local Differential Privacy for Regret Minimization in Reinforcement Learning

    Evrard Garcelon, Vianney Perchet, Ciara Pike-Burke, and Matteo Pirotta. Local Differential Privacy for Regret Minimization in Reinforcement Learning . In Advances in Neural Information Processing Systems , volume 34, pp.\ 10561--10573. Curran Associates, Inc., 2021. URL https:...

    2021

  12. [20]

    The optimal noise-adding mechanism in differential privacy

    Quan Geng and Pramod Viswanath. The optimal noise-adding mechanism in differential privacy. IEEE Transactions on Information Theory, 62 0 (2): 0 925--951, 2015

    2015

  13. [21]

    Baffle: Hiding Backdoors in Offline Reinforcement Learning Datasets

    Chen Gong, Zhou Yang, Yunpeng Bai, Junda He, Jieke Shi, Kecen Li, Arunesh Sinha, Bowen Xu, Xinwen Hou, David Lo, et al. Baffle: Hiding Backdoors in Offline Reinforcement Learning Datasets . In 2024 IEEE Symposium on Security and Privacy (SP) , pp.\ 2086--2104. IEEE, 2024

    2024

  14. [22]

    Numerical composition of differential privacy

    Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical composition of differential privacy. Advances in Neural Information Processing Systems, 34: 0 11631--11642, 2021

    2021

  15. [23]

    A Review of Safe Reinforcement Learning : Methods , Theory and Applications , May 2024

    Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A Review of Safe Reinforcement Learning : Methods , Theory and Applications , May 2024. URL http://arxiv.org/abs/2205.10330. arXiv:2205.10330 [cs]

    2024 arXiv

  16. [24]

    u rtler, Sebastian Blaes, Pavel Kolev, Felix Widmaier, Manuel W \

    Nico G \"u rtler, Sebastian Blaes, Pavel Kolev, Felix Widmaier, Manuel W \"u thrich, Stefan Bauer, Bernhard Sch \"o lkopf, and Georg Martius. Benchmarking Offline Reinforcement Learning on Real-Robot Hardware . arXiv preprint arXiv:2307.15690, 2023

    2023 arXiv

  17. [25]

    Intrinsic Certified Robustness of Bagging against Data Poisoning Attacks

    Jinyuan Jia, Xiaoyu Cao, and Neil Zhenqiang Gong. Intrinsic Certified Robustness of Bagging against Data Poisoning Attacks . arXiv:2008.04495 [cs], December 2020. URL http://arxiv.org/abs/2008.04495. arXiv: 2008.04495

    2008 arXiv

  18. [26]

    Learning to Drive in a Day

    Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to Drive in a Day . In 2019 international conference on robotics and automation (ICRA) , pp.\ 8248--8254. IEEE, 2019

    2019

  19. [27]

    TrojDRL : Evaluation of Backdoor Attacks on Deep Reinforcement Learning

    Panagiota Kiourti, Kacper Wardega, Susmit Jha, and Wenchao Li. TrojDRL : Evaluation of Backdoor Attacks on Deep Reinforcement Learning . In 2020 57th ACM / IEEE Design Automation Conference ( DAC ) , pp.\ 1--6, San Francisco, CA, USA, July 2020. IEEE. ISBN 978-1-72811-085-1. d...

    2020

  20. [28]

    Offline Reinforcement Learning with Implicit Q-learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline Reinforcement Learning with Implicit Q-learning . arXiv preprint arXiv:2110.06169, 2021

    2021 arXiv

  21. [29]

    Adversarial Machine Learning-Industry Perspectives

    Ram Shankar Siva Kumar, Magnus Nystr \"o m, John Lambert, Andrew Marshall, Mario Goertzel, Andi Comissoneru, Matt Swann, and Sharon Xia. Adversarial Machine Learning-Industry Perspectives . In 2020 IEEE security and privacy workshops (SPW) , pp.\ 69--75. IEEE, 2020

    2020

  22. [30]

    Batch Reinforcement Learning

    Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch Reinforcement Learning . In Reinforcement learning: State-of-the-art , pp.\ 45--73. Springer, 2012

    2012

  23. [31]

    Certified Robustness to Adversarial Examples with Differential Privacy

    Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified Robustness to Adversarial Examples with Differential Privacy . arXiv:1802.03471 [cs, stat], May 2019. URL http://arxiv.org/abs/1802.03471. arXiv: 1802.03471

    2019 arXiv

  24. [32]

    Deep Partition Aggregation : Provable Defense against General Poisoning Attacks

    Alexander Levine and Soheil Feizi. Deep Partition Aggregation : Provable Defense against General Poisoning Attacks . arXiv:2006.14768 [cs, stat], March 2021. URL http://arxiv.org/abs/2006.14768. arXiv: 2006.14768

    2006 arXiv

  25. [33]

    Enhancing Certified Robustness via Smoothed Weighted Ensembling

    Chizhou Liu, Yunzhen Feng, Ranran Wang, and Bin Dong. Enhancing Certified Robustness via Smoothed Weighted Ensembling . arXiv:2005.09363 [cs, stat], February 2021. URL http://arxiv.org/abs/2005.09363. arXiv: 2005.09363

    2005 arXiv

  26. [34]

    Enhancing the Antidote: Improved Pointwise Certifications Against Poisoning Attacks

    Shijie Liu, Andrew C Cullen, Paul Montague, Sarah M Erfani, and Benjamin IP Rubinstein. Enhancing the Antidote: Improved Pointwise Certifications Against Poisoning Attacks . In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pp.\ 8861--8869, 2023

    2023

  27. [35]

    Corruption-robust E xploration in E pisodic R einforcement L earning, October 2023

    Thodoris Lykouris, Max Simchowitz, Aleksandrs Slivkins, and Wen Sun. Corruption-robust E xploration in E pisodic R einforcement L earning, October 2023. URL http://arxiv.org/abs/1911.08689. arXiv:1911.08689 [cs, stat]

    2023 arXiv

  28. [36]

    Data Poisoning against Differentially - Private Learners : Attacks and Defenses

    Yuzhe Ma, Xiaojin Zhu, and Justin Hsu. Data Poisoning against Differentially - Private Learners : Attacks and Defenses . arXiv:1903.09860 [cs], July 2019. URL http://arxiv.org/abs/1903.09860. arXiv: 1903.09860

    1903 arXiv

  29. [37]

    Learning Differentially Private Recurrent Language Models

    H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning Differentially Private Recurrent Language Models . arXiv preprint arXiv:1710.06963, 2017

    2017 arXiv

  30. [38]

    Renyi Differential Privacy

    Ilya Mironov. Renyi Differential Privacy . 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pp.\ 263--275, August 2017. doi:10.1109/CSF.2017.11. URL http://arxiv.org/abs/1702.07476. arXiv: 1702.07476

    2017 arXiv

  31. [39]

    Renyi Differential Privacy of the Sampled Gaussian Mechanism

    Ilya Mironov, Kunal Talwar, and Li Zhang. Renyi Differential Privacy of the Sampled Gaussian Mechanism . arXiv:1908.10530 [cs, stat], August 2019. URL http://arxiv.org/abs/1908.10530. arXiv: 1908.10530

    1908 arXiv

  32. [40]

    Asynchronous Methods for Deep Reinforcement Learning

    Volodymyr Mnih. Asynchronous Methods for Deep Reinforcement Learning . arXiv preprint arXiv:1602.01783, 2016

    2016 arXiv

  33. [41]

    Playing Atari with Deep Reinforcement Learning , December 2013

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with Deep Reinforcement Learning , December 2013. URL http://arxiv.org/abs/1312.5602. arXiv:1312.5602 [cs]

    2013 arXiv

  34. [42]

    Reinforcement Learning for Optimized Trade Execution

    Yuriy Nevmyvaka, Yi Feng, and Michael Kearns. Reinforcement Learning for Optimized Trade Execution . In International Conference on Machine Learning, pp.\ 673--680, 2006

    2006

  35. [43]

    Online Defense Strategies for Reinforcement Learning Against Adaptive Reward Poisoning

    Andi Nika, Adish Singla, and Goran Radanovic. Online Defense Strategies for Reinforcement Learning Against Adaptive Reward Poisoning . In 26th International Conference on Artificial Intelligence and Statistics , pp.\ 335--358. PMRL, 2023

    2023

  36. [45]

    Agile Autonomous Driving Using End-to-end Deep Imitation Learning

    Yunpeng Pan, Ching-An Cheng, Kamil Saigol, Keuntaek Lee, Xinyan Yan, Evangelos Theodorou, and Byron Boots. Agile Autonomous Driving Using End-to-end Deep Imitation Learning . arXiv preprint arXiv:1709.07174, 2017

    2017 arXiv

  37. [46]

    Deep K-NN Defense Against Clean-label Data Poisoning Attacks

    Neehar Peri, Neal Gupta, W Ronny Huang, Liam Fowl, Chen Zhu, Soheil Feizi, Tom Goldstein, and John P Dickerson. Deep K-NN Defense Against Clean-label Data Poisoning Attacks . In Computer Vision--ECCV 2020 Workshops: Glasgow, UK, August 23--28, 2020, Proceedings, Part I 16 , pp...

    2020

  38. [47]

    Rafael Figueiredo Prudencio, Marcos R. O. A. Maximo, and Esther Luna Colombini. A Survey on Offline Reinforcement Learning : Taxonomy , Review , and Open Problems . IEEE Transactions on Neural Networks and Learning Systems, pp.\ 1--0, 2024. ISSN 2162-237X, 2162-2388. doi:10.11...

    2024

  39. [48]

    Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers

    Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers . Advances in neural information processing systems, 32, 2019

    2019

  40. [49]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms . arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  41. [50]

    Mastering the Game of Go Without Human Knowledge

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the Game of Go Without Human Knowledge . nature, 550 0 (7676): 0 354--359, 2017

    2017

  42. [51]

    Policy Gradient Methods for Reinforcement Learning with Function Approximation

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation . Advances in neural information processing systems, 12, 1999

    1999

  43. [52]

    Terry, Ariel Kwiatkowski, John U

    Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez-Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasiu...

    2023

  44. [53]

    Private reinforcement learning with pac and regret guarantees

    Giuseppe Vietri, Borja Balle, Akshay Krishnamurthy, and Steven Wu. Private reinforcement learning with pac and regret guarantees. In International Conference on Machine Learning, pp.\ 9754--9764. PMLR, 2020

    2020

  45. [54]

    Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation

    Lu Wang, Wei Zhang, Xiaofeng He, and Hongyuan Zha. Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation . In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pp.\ 2447--2456, 2018

    2018

  46. [55]

    Stop-and- Go : Exploring Backdoor Attacks on Deep Reinforcement Learning -based Traffic Congestion Control Systems

    Yue Wang, Esha Sarkar, Wenqing Li, Michail Maniatakos, and Saif Eddin Jabari. Stop-and- Go : Exploring Backdoor Attacks on Deep Reinforcement Learning -based Traffic Congestion Control Systems . IEEE Transactions on Information Forensics and Security, 16: 0 4772--4787, 2021. I...

    2021

  47. [56]

    COPA : Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attacks , March 2022

    Fan Wu, Linyi Li, Chejian Xu, Huan Zhang, Bhavya Kailkhura, Krishnaram Kenthapadi, Ding Zhao, and Bo Li. COPA : Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attacks , March 2022. URL http://arxiv.org/abs/2203.08398. arXiv:2203.08398 [cs]

    2022 arXiv

  48. [57]

    Reward Poisoning Attacks on Offline Multi-agent Reinforcement Learning

    Young Wu, Jeremy McMahan, Xiaojin Zhu, and Qiaomin Xie. Reward Poisoning Attacks on Offline Multi-agent Reinforcement Learning . In Proceedings of the aaai conference on artificial intelligence , volume 37, pp.\ 10426--10434, 2023

    2023

  49. [58]

    Towards Robust Offline Reinforcement Learning under Diverse Data Corruption , March 2024

    Rui Yang, Han Zhong, Jiawei Xu, Amy Zhang, Chongjie Zhang, Lei Han, and Tong Zhang. Towards Robust Offline Reinforcement Learning under Diverse Data Corruption , March 2024. URL http://arxiv.org/abs/2310.12955. arXiv:2310.12955 [cs]

    2024 arXiv

  50. [59]

    Corruption- Robust Offline Reinforcement Learning with General Function Approximation

    Chenlu Ye, Rui Yang, Quanquan Gu, and Tong Zhang. Corruption- Robust Offline Reinforcement Learning with General Function Approximation . Advances in Neural Information Processing Systems, 36: 0 36208--36221, December 2023. URL https://proceedings.neurips.cc/paper_files/paper/...

    2023

  51. [60]

    Opacus: U ser-friendly D ifferential P rivacy L ibrary in P yTorch

    Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: U ser-friendly D ifferential P rivacy L ibrary in P yTorch . arXiv preprint...

    2021 arXiv

  52. [61]

    Reinforcement Learning in Healthcare: A survey

    Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement Learning in Healthcare: A survey . ACM Computing Surveys (CSUR), 55 0 (1): 0 1--36, 2021

    2021

  53. [62]

    Corruption- Robust Offline Reinforcement Learning , June 2021 a

    Xuezhou Zhang, Yiding Chen, Jerry Zhu, and Wen Sun. Corruption- Robust Offline Reinforcement Learning , June 2021 a . URL http://arxiv.org/abs/2106.06630. arXiv:2106.06630 [cs]

    2021 arXiv

  54. [63]

    Robust Policy Gradient against Strong Data Corruption , June 2021 b

    Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Robust Policy Gradient against Strong Data Corruption , June 2021 b . URL http://arxiv.org/abs/2102.05800. arXiv:2102.05800 [cs]

    2021 arXiv

  55. [64]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  56. [65]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  57. [66]

    u rtler, Nico and Blaes, Sebastian and Kolev, Pavel and Widmaier, Felix and W \

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2013 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.