Pith. sign in

REVIEW 3 major objections 6 minor 44 references

Action Robust Reinforcement Learning via Optimal Adversary Aware Policy Optimization

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Optimal adversary-aware Bellman training makes RL robust to action perturbation.

desk verdict Solid empirical recipe for action-robust RL, but the theory overclaims: the adversary in Eq. (5) differs from the attacks in §5.1.1 for stochastic policies, and the Theorem 4.5 proof has an unproven max-min swap. read the letter →

arxiv 2507.03372 v1 pith:IXAJRHB4 submitted 2025-07-04 cs.LG

Buqing Nie , Yangqing Fu , Jingtian Ji , Yue Gao This is my paper · ORCID
classification cs.LG MSC 68T0568Q3290C40
keywords actionrobustnessadversarialreinforcementlearningBellmanoperatorpolicyiterationTD3PPOperturbationworst-caseattack
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that deep RL policies can be made robust to bounded action perturbations by evaluating and improving them against the worst-case perturbation, rather than by adding random noise or by training an explicit adversary. It formulates an action-adversarial MDP and introduces an optimal adversary-aware Bellman operator that directly computes the value of a policy under its optimal adversary. The paper claims this operator is a contraction, that its fixed point is the robust value function, and that policy iteration built on it converges to the optimal robust policy. If correct, the framework offers a cheaper and more effective route to action robustness than the existing baselines it compares against, and it can be bolted onto both stochastic and deterministic DRL algorithms while keeping nominal performance and sample efficiency.

What carries the argument

The load-bearing object is the optimal adversary-aware Bellman operator of Eq. (5), which replaces the usual expectation over the next action with a worst-case minimization over bounded perturbations before the next action is evaluated. That operator carries the argument because it lets the value function absorb the optimal adversary implicitly, so policy evaluation and improvement can run without a separately trained attacker. The inner minimization is approximated by projected gradient descent, and in the deterministic-policy variant the two objectives from the nominal critic and the adversary-aware critic are reconciled by projecting conflicting gradients.

What would settle it

Run the trained policies against an adversary that observes the realized action $a$ before choosing $\delta$ (the setting of the Min-Q and Min-OA-Q attacks) and compare the resulting episode returns with the values predicted by Eq. (5); strictly lower returns would falsify the claim that the operator evaluates the optimal adversary for that threat model. A second check is to construct a one-step MDP where $\max_a \min_\delta Q(s,a+\delta) \neq \min_\delta \max_a Q(s,a+\delta)$, which would directly test the commutation step used in the convergence proof of Theorem 4.5.

Watch

Extended reading notes

Core claim

The central claim is that robustness to action perturbations can be learned without explicitly training an adversary. The paper defines the optimal adversary-aware Bellman operator $\tilde{T}^\pi Q_{\mathrm{adv}}(s,a) = R(s,a) + \gamma \mathbb{E}_{s'\sim P}[\min_{\|\delta\|\le \epsilon} \mathbb{E}_{a'\sim\pi(s')} Q_{\mathrm{adv}}(s', a'+\delta)]$ and proves it is a contraction mapping whose fixed point is the value of the policy under the optimal action adversary. It then proves a policy improvement theorem and a convergence result, stating that repeated optimal adversary-aware evaluation and improvement leads to the optimal policy against the worst-case bounded perturbation. On the algorithmic side, the paper shows how to integrate this idea into TD3 and PPO with a dedicated adversary-aware critic, using projected gradient descent for the inner minimization and gradient surgery to reconcile robustness gradients with nominal-performance gradients. The experiments report that these OA-TD3 and OA-PPO variants outperform noise-trained, min-Q-trained, and prior action-robust baselines under nominal, random, biggest, Min-Q, and Min-OA-Q attacks.

Load-bearing premise

The theory treats the adversary as choosing its perturbation before the agent's next action is sampled, but the attacks in the experiments can see the executed action before perturbing it, so if the real threat model is the action-observing adversary, the optimality and convergence guarantees do not cover the evaluated attacks.

Editorial extensions

If this is right

  • Action-robust RL can be trained in a model-free way without an explicit adversarial training loop, avoiding the extra sampling cost of training a separate attacker.
  • A single adversary-aware critic can serve both as a training signal and as an attack oracle, since the paper finds Min-OA-Q attacks, built from this critic, are the strongest of the tested adversaries.
  • The framework transfers across stochastic on-policy algorithms (PPO) and deterministic off-policy algorithms (TD3), suggesting the operator is a generic robustness primitive rather than a method-specific fix.
  • The reported gains, such as OA-TD3 reaching 4928 on Ant under Min-Q attacks versus 3435 for TD3, indicate that robustness and nominal performance need not be traded off when gradient conflicts are managed.
  • The approach also produces visually different behavior, as in the Maze2D example where the robust policy keeps away from walls rather than hugging them, making it more resilient to action error.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because Eq. (5) places the minimization inside the expectation over the next action, the optimality guarantee applies to an adversary that commits to a perturbation before seeing the realized action; the Min-Q and Min-OA-Q attacks used in the experiments observe the action first, so those two threat models are not the same. Testing the trained policies against an action-observing adversary would s
  • The same adversary-aware Bellman operator could be reused as part of a certification procedure: if the trained $Q_{\mathrm{adv}}$ is a true lower-bound value under the optimal adversary, then it directly provides a worst-case performance estimate that a deployed policy could monitor online.
  • The inner minimization by PGD is the main computational bottleneck, as the appendix reports; replacing it with a cheaper global optimizer or a learned approximate minimizer is a natural extension that could make the framework practical in real-time control.
  • The hyperparameter $\omega$ separating nominal from robust objectives suggests a tunable Pareto trade-off rather than a single robustness level, which could be adapted to application-specific risk tolerance without retraining from scratch.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper studies robustness of RL policies to bounded l_infinity perturbations of actions. It formalizes an Action-adversarial Markov Decision Process (AA-MDP), defines an 'optimal adversary-aware' Bellman operator in Eq. (5), and claims (Theorems 4.2-4.5) that this operator contracts to the value under the optimal adversary, that greedy policy updates are monotone, and that OA-PI converges to the optimal policy. The framework is instantiated as OA-PPO and OA-TD3, using PGD-computed perturbations delta* and gradient surgery for TD3. Experiments on BipedalWalker, LunarLander, and five MuJoCo tasks compare against PPO, PPO-Noise, PPO-Min-Q, TD3, TD3-Noise, NR/PR-MDP, and an ablation without gradient surgery, under nominal, random, biggest, Min-Q, and Min-OA-Q attacks. The main empirical claim is that OA-DRL improves robustness across these adversaries while retaining nominal performance and sample efficiency; for example, Walker2d OA-PPO under Min-OA-Q achieves 1936 versus 1415 for the best baseline.

Significance. If both the empirical and theoretical claims were fully established, this would be a useful contribution to action-robust reinforcement learning, extending the less-explored action-perturbation setting and offering a practical integration with TD3 and PPO. Strengths of the submission include the use of at least 10 seeds with standard errors, evaluation under several adversaries that are not aligned with the OA training objective (random, biggest, Min-Q), so the gains are not purely an artifact of training against the evaluation attack, and the informative gradient-surgery and hyperparameter ablations. The principal weakness is that the theoretical guarantees as written do not cover the stochastic-policy attack model actually evaluated, and the proof of the central convergence theorem contains an unjustified max-min interchange. These gaps are load-bearing for the advertised 'optimal adversary-aware' claim, although the empirical evidence is suggestive enough that a careful revision could make the paper acceptable.

major comments (3)
  1. [Section 4.2.1, Eq. (5); Appendix B; Section 5.1.1] The adversary in the theory chooses its perturbation before the agent's next action is sampled: Eq. (5) applies one delta inside E_{a'~pi(s')}, and Appendix B defines P_nu(s'|s,delta)=E_{a~pi}[P(s'|s,a+delta)]. In contrast, the Min-Q and Min-OA-Q attacks in Section 5.1.1 choose delta=argmin_delta Q(s,a+delta) for the realized action a, i.e., an adversary that observes the action. For a stochastic policy, min_delta E_{a~pi}[Q(s,a+delta)] >= E_{a~pi}[min_delta Q(s,a+delta)], with strict inequality generically, so the fixed point of Eq. (5) overestimates the value under the evaluated attacks. Consequently Theorems 4.2-4.4 do not apply to the threat model used in the OA-PPO experiments. If the PPO evaluation is in fact performed with deterministic mean actions, this should be stated explicitly; otherwise the theory should be extended to action-observing adversaries.
  2. [Appendix C.4, proof of Theorem 4.5] The proof derives V_{pi_k circle nu*}(s)=max_a min_delta Q_adv(s,a+delta), but then substitutes min_delta max_a Q_adv into the Bellman equation without justification. The operators max_a min_delta and min_delta max_a are not equal in general, so the displayed 'Bellman optimality equation in AA-MDP' does not follow from the preceding line. This step is what identifies the converged policy with the optimal policy, so Theorem 4.5 is not proven as written. The authors need to either prove the interchange under explicit assumptions on Q_adv or state and prove the correct optimality equation for the intended adversary order.
  3. [Appendix C.1, proof of Theorem 4.2] The chain of equalities that identifies the contraction's fixed point with Q^{pi circle nu*} jumps from the recursive composition of min operators to min_{delta_i} E_{a_i~pi}[sum_t R(s_t,a_t+delta_t)] without an argument. Each delta_i is chosen causally after s_i is observed, and the inner minimization concerns the future Q-value rather than the immediate reward, so the equality is not automatic. The fixed-point identity can presumably be proved by treating the adversary's problem as an MDP as sketched in Appendix B, but the paper needs to supply that proof; as it stands, the proof of Theorem 4.2 is incomplete at this step.
minor comments (6)
  1. [Algorithm 3, line 8] The target 'Q_adv(s_{t+1} + delta)' is missing the next action argument; it should be Q_adv(s', a' + delta) with a' drawn from the policy, consistent with Eq. (5).
  2. [Section 4.2.1, sentence after Eq. (5)] Calling the perturbations 'strategically planned and temporally correlated' is stronger than what Eq. (5) describes, which is a Markov, state-feedback adversary; the phrasing should be softened or justified.
  3. [Appendix E.5] The text says the n-score satisfies Z_norm in [0,1], but Figure 3 shows average n-scores above 1 for OA-TD3; the normalization description should be reconciled with the figure.
  4. [Table 1 caption] There is a typo: 'stochastic polices' should be 'stochastic policies'.
  5. [Appendix E.2] 'Projected Gradient Decent' should be 'Projected Gradient Descent'.
  6. [Algorithm 2, line 9] For reproducibility, the OA-aware target should clarify that delta is computed by PGD against the target network Q'_adv and that the same clipped exploration noise is used as in the standard target; this detail is currently implicit.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: OA-PI's convergence and policy-improvement results are standard contraction/policy-iteration arguments, and the noted max-min swap in Appendix C.4 is a proof gap rather than a circular reduction.

full rationale

The paper's central derivation is a conventional policy-iteration chain. The optimal adversary-aware Bellman operator in Eq. (5) is shown to be a contraction in Theorem 4.2, and its fixed point is identified with Q^{π◦ν*} by unrolling the operator; this is the standard dynamic-programming fixed-point argument, not a definitional identification. Theorems 4.3 and 4.4 follow the usual policy-improvement proof, and Theorem 4.5 is the standard convergence step. No parameter is fitted and then renamed as a prediction: the experimental comparisons include nominal, random, biggest, Min-Q, and Min-OA-Q adversaries, and the claim is empirical rather than a derived quantity. The Min-OA-Q attack is aligned with the OA training objective, so those particular cells are partly self-consistent, but the paper also reports independent attack types, and the theoretical claims do not rely on that evaluation. Appendix C.4 does contain an unproven interchange of max_a and min_δ when writing Qadv = R + γE[min_δ max_a Qadv] and calling it the Bellman optimality equation; however, this is a correctness gap or a threat-model mismatch, not a reduction of the conclusion to the assumptions by construction. There are no load-bearing self-citations: the cited proof similarities are to external works such as WocaR-RL, and the central results are argued from the paper's own equations. Therefore no circular step meets the evidentiary bar set by the review instructions, and the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The paper's central claim rests on two tuned hyperparameters (epsilon and omega), the PGD approximation of the optimal perturbation, standard contraction and policy improvement arguments, and two unstated interchanges (min with expectation, and max with min) that are not fully justified. No new physical entities, forces, or dimensions are introduced; the only new object is the trained auxiliary critic Q_adv.

free parameters (3)
  • epsilon (robust training perturbation strength) = Ant 0.15, Hopper 0.2, Walker2d 0.2, HalfCheetah 0.2, LunarLander 0.3, BipedalWalker 0.2 (tuned in [0.1, 0.3])
    Tuned per task to balance nominal performance and robustness; the entire method is defined relative to this budget, and evaluation results in Tables 1 and 2 depend on it.
  • omega (trade-off weight between nominal critic and OA-aware critic) = Range [0.4, 0.6], per-task values not disclosed
    Chosen per task to balance optimality and robustness in Eqs. (8) and (9); the ablation in Fig. 4 shows performance varies noticeably with this parameter.
  • PGD steps K and step size eta = epsilon / K for computing delta* = K = 15 to 20
    The 'optimal perturbation' used in both training targets and evaluation attacks is only approximate, and its accuracy depends on these settings, which are not tuned per task.
assumptions (5)
  • standard math Banach contraction mapping theorem
    Used in Theorem 4.2 to prove that the optimal adversary-aware Bellman operator converges to a unique fixed point.
  • domain assumption Measurable selection: the min over the perturbation set inside the expectation attains a value and can be exchanged with the outer expectation
    Eqs. (5) and (6) require the fixed point of the operator to equal the value under the optimal stationary adversary, which assumes the min and expectation interchange cleanly.
  • domain assumption The adversary is stationary and reactive, choosing delta based only on the current state before the realized action is sampled
    Definition 4.1 and Appendix B model the adversary this way, while the text claims temporally correlated attacks; the value under the two adversary classes differs in general.
  • ad hoc to paper Max-min interchange: max_a min_delta Q_adv = min_delta max_a Q_adv in the AA-MDP Bellman optimality equation
    Appendix C.4 substitutes min_delta max_a for max_a min_delta in the proof of Theorem 4.5 without justification; this interchange does not hold for nonconvex neural-network Q functions.
  • standard math Standard policy improvement theorem extended to robust MDPs
    Theorem 4.3 mirrors the classical policy improvement argument, requiring the condition Q_pi_adv(s, pi') >= V_pi_nu_star(s) for all states.
invented entities (1)
  • OA-aware critic Q_adv independent evidence
    purpose: An auxiliary value function that evaluates policy performance under the optimal action adversary; it is used in the policy evaluation target (Eq. 5), the policy improvement objectives (Eqs. 8 and 9), and the Min-OA-Q evaluation attack (Algorithm 4).
    Q_adv is a trained function approximator, not a speculative entity. Its predictions are falsifiably checked against measured returns under Min-OA-Q attacks, and it is compared against standard Q functions in the PPO-Min-Q ablation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Action Robust Reinforcement Learning via Optimal Adversary Aware Policy Optimization." pith.science (2026). https://pith.science/paper/IXAJRHB4

@misc{pith2026250703372,
  author       = {Pith},
  title        = {Pith review of: Action Robust Reinforcement Learning via Optimal Adversary Aware Policy Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IXAJRHB4}},
  note         = {Machine review of arXiv:2507.03372}
}
read the original abstract

Reinforcement Learning (RL) has achieved remarkable success in sequential decision tasks. However, recent studies have revealed the vulnerability of RL policies to different perturbations, raising concerns about their effectiveness and safety in real-world applications. In this work, we focus on the robustness of RL policies against action perturbations and introduce a novel framework called Optimal Adversary-aware Policy Iteration (OA-PI). Our framework enhances action robustness under various perturbations by evaluating and improving policy performance against the corresponding optimal adversaries. Besides, our approach can be integrated into mainstream DRL algorithms such as Twin Delayed DDPG (TD3) and Proximal Policy Optimization (PPO), improving action robustness effectively while maintaining nominal performance and sample efficiency. Experimental results across various environments demonstrate that our method enhances robustness of DRL policies against different action adversaries effectively.

Figures

Figures reproduced from arXiv: 2507.03372 by the authors.

Figure 1
Figure 1. The interaction process between the agent [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Normalized training curves of OA-PPO and baseline methods. See [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Normalized training curves of OA-TD3 and baseline methods. See [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The influence of parameters ω and ϵ to the performance of OA-TD3 policies. The y-axis denotes the advantage compared to typical TD3 under same adversaries. Advantage ratio greater than 1.0 means higher performance than TD3. (a)-(b) Robustness of policies trained with d…
Figure 5
Figure 5. Figure 5: The interaction process of AA-MDP from the perspective of the action adversary, which consid￾ers the agent policy π as part of the environment. As depicted in [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: The process of projection operations in Gradient Surgery algorithm. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Learning curves of OA-PPO (red) in various tasks against baseline methods. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: The learning curves of OA-TD3 (red) against other methods. As depicted in the figures, [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: The experiment results on the Maze2D environment. (a) The visualization of Maze2D environment. (b) The path found by the TD3 policy. (c) The path found by the OA-TD3 policy. initial position (lower left corner) in a closed maze, where gray areas denote impassable walls…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 31 canonical work pages

  1. [1]

    Reinforcement learning based recommender systems: A survey

    M Mehdi Afsar, Trafford Crump, and Behrouz Far. Reinforcement learning based recommender systems: A survey. ACM Computing Surveys, 55(7):1–38, 2022

    work page 2022

  2. [2]

    Safe learning in robotics: From learning-based control to safe reinforcement learning

    Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5:411–444, 2022

    work page 2022

  3. [3]

    Certifiable robustness to adversarial state uncertainty in deep reinforcement learning

    Michael Everett, Björn Lütjens, and Jonathan P How. Certifiable robustness to adversarial state uncertainty in deep reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 33(9):4184–4198, 2021

    work page 2021

  4. [4]

    Maximum entropy RL (provably) solves some robust RL problems

    Benjamin Eysenbach and Sergey Levine. Maximum entropy RL (provably) solves some robust RL problems. In International Conference on Learning Representations, 2022

    work page 2022

  5. [5]

    Online robustness training for deep reinforcement learning

    Marc Fischer, Matthew Mirman, Steven Stalder, and Martin Vechev. Online robustness training for deep reinforcement learning. arXiv preprint arXiv:1911.00887, 2019

    arXiv 1911

  6. [6]

    D4rl: Datasets for deep data-driven reinforcement learning

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020

    arXiv 2004

  7. [7]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587–1596. PMLR, 2018

    2018

  8. [8]

    Matteo Hessel, Joseph Modayil, H. V . Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI Conference on Artificial Intelligence, 2017

    work page 2017

Show all 44 references
  1. [9]

    Adversarial attacks on neural network policies

    Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprint arXiv:1702.02284, 2017

    2017 arXiv

  2. [10]

    The 37 implementation details of proximal policy optimization

    Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. The ICLR Blog Track 2023, 2022

    2023

  3. [11]

    Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms

    Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and JoÃG, o GM AraÚjo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1–18, 2022

    2022

  4. [12]

    Challenges and countermeasures for adversarial attacks on deep reinforcement learning

    Inaam Ilahi, Muhammad Usama, Junaid Qadir, Muhammad Umar Janjua, Ala Al-Fuqaha, Dinh Thai Hoang, and Dusit Niyato. Challenges and countermeasures for adversarial attacks on deep reinforcement learning. IEEE Transactions on Artificial Intelligence, 3(2):90–109, 2021

    2021

  5. [13]

    Learning quadrupedal locomotion over challenging terrain

    Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47):eabc5986, 2020

    2020

  6. [14]

    Spatiotem- porally constrained action space attacks on deep reinforcement learning agents

    Xian Yeow Lee, Sambit Ghadai, Kai Liang Tan, Chinmay Hegde, and Soumik Sarkar. Spatiotem- porally constrained action space attacks on deep reinforcement learning agents. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 4577–4584, 2020

    2020

  7. [15]

    Efficient adversarial training without attacking: Worst-case-aware robust reinforcement learning

    Yongyuan Liang, Yanchao Sun, Ruijie Zheng, and Furong Huang. Efficient adversarial training without attacking: Worst-case-aware robust reinforcement learning. Advances in Neural Information Processing Systems, 35:22547–22561, 2022

    2022

  8. [16]

    Provably efficient black-box action poisoning attacks against reinforcement learning

    Guanlin Liu and Lifeng Lai. Provably efficient black-box action poisoning attacks against reinforcement learning. Advances in Neural Information Processing Systems, 34:12400–12410, 2021

    2021

  9. [17]

    Towards deep learning models resistant to adversarial attacks

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. 10

    2018

  10. [18]

    Human-level control through deep reinforcement learning

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015

    2015

  11. [19]

    Robust reinforcement learning

    Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation, 17(2):335– 359, 2005

    2005

  12. [20]

    Assessing transferability from simulation to reality for reinforcement learning

    Fabio Muratore, Michael Gienger, and Jan Peters. Assessing transferability from simulation to reality for reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 43(4):1172–1183, 2019

    2019

  13. [21]

    Robust deep reinforcement learning through adversarial loss

    Tuomas Oikarinen, Wang Zhang, Alexandre Megretski, Luca Daniel, and Tsui-Wei Weng. Robust deep reinforcement learning through adversarial loss. Advances in Neural Information Processing Systems, 34:26156–26167, 2021

    2021

  14. [22]

    Characterizing attacks on deep reinforcement learning

    Xinlei Pan, Chaowei Xiao, Warren He, Shuang Yang, Jian Peng, Mingjie Sun, Mingyan Liu, Bo Li, and Dawn Song. Characterizing attacks on deep reinforcement learning. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems , pages 1010–1018, 2022

    2022

  15. [23]

    Policy teaching via environment poisoning: Training-time adversarial attacks against reinforcement learning

    Amin Rakhsha, Goran Radanovic, Rati Devidze, Xiaojin Zhu, and Adish Singla. Policy teaching via environment poisoning: Training-time adversarial attacks against reinforcement learning. In International Conference on Machine Learning, pages 7974–7984. PMLR, 2020

    2020

  16. [24]

    The security of autonomous driving: Threats, defenses, and future directions

    Kui Ren, Qian Wang, Cong Wang, Zhan Qin, and Xiaodong Lin. The security of autonomous driving: Threats, defenses, and future directions. Proceedings of the IEEE, 108(2):357–372, 2019

    2019

  17. [25]

    Learning to walk in minutes using massively parallel deep reinforcement learning

    Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pages 91–100. PMLR, 2022

    2022

  18. [26]

    Improving robotic machining accuracy through experimental error investigation and modular compensation.The International Journal of Advanced Manufacturing Technology, 85:3–15, 2016

    Ulrich Schneider, Manuel Drust, Matteo Ansaloni, Christian Lehmann, Marcello Pellicciari, Francesco Leali, Jan Willem Gunnink, and Alexander Verl. Improving robotic machining accuracy through experimental error investigation and modular compensation.The International Journal o...

    2016

  19. [27]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  20. [28]

    Towards facilitating empathic conversations in online mental health support: A reinforcement learning approach

    Ashish Sharma, Inna W Lin, Adam S Miner, David C Atkins, and Tim Althoff. Towards facilitating empathic conversations in online mental health support: A reinforcement learning approach. In Proceedings of the Web Conference 2021, pages 194–205, 2021

    2021

  21. [29]

    Certifiably robust policy learning against adversarial multi-agent communication

    Yanchao Sun, Ruijie Zheng, Parisa Hassanzadeh, Yongyuan Liang, Soheil Feizi, Sumitra Ganesh, and Furong Huang. Certifiably robust policy learning against adversarial multi-agent communication. In The Eleventh International Conference on Learning Representations, 2022

    2022

  22. [30]

    Certifiably robust policy learning against adversarial multi-agent communication

    Yanchao Sun, Ruijie Zheng, Parisa Hassanzadeh, Yongyuan Liang, Soheil Feizi, Sumitra Ganesh, and Furong Huang. Certifiably robust policy learning against adversarial multi-agent communication. In The Eleventh International Conference on Learning Representations, 2023

    2023

  23. [31]

    Who is the strongest enemy? towards optimal and efficient evasion attacks in deep RL

    Yanchao Sun, Ruijie Zheng, Yongyuan Liang, and Furong Huang. Who is the strongest enemy? towards optimal and efficient evasion attacks in deep RL. In International Conference on Learning Representations, 2022

    2022

  24. [32]

    Reinforcement learning: An introduction

    Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018

    2018

  25. [33]

    Action robust reinforcement learning and applications in continuous control

    Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, pages 6215–6224. PMLR, 2019. 11

    2019

  26. [34]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026–5033. IEEE, 2012

    2012

  27. [35]

    Policy gradient method for robust reinforcement learning

    Yue Wang and Shaofeng Zou. Policy gradient method for robust reinforcement learning. In International Conference on Machine Learning, pages 23484–23526. PMLR, 2022

    2022

  28. [36]

    CROP: Certify- ing robust policies for reinforcement learning through functional smoothing

    Fan Wu, Linyi Li, Zijian Huang, Yevgeniy V orobeychik, Ding Zhao, and Bo Li. CROP: Certify- ing robust policies for reinforcement learning through functional smoothing. In International Conference on Learning Representations, 2022

    2022

  29. [37]

    Robust deep reinforcement learning through bootstrapped opportunistic curriculum

    Junlin Wu and Yevgeniy V orobeychik. Robust deep reinforcement learning through bootstrapped opportunistic curriculum. In International Conference on Machine Learning, pages 24177– 24211. PMLR, 2022

    2022

  30. [38]

    Taac: Temporally abstract actor-critic for continuous control

    Haonan Yu, Wei Xu, and Haichao Zhang. Taac: Temporally abstract actor-critic for continuous control. In Neural Information Processing Systems, 2021

    2021

  31. [39]

    Gradient surgery for multi-task learning

    Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824–5836, 2020

    2020

  32. [40]

    Robust reinforcement learning on state observations with learned optimal adversary

    Huan Zhang, Hongge Chen, Duane S Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. In International Conference on Learning Representations, 2021

    2021

  33. [41]

    Robust deep reinforcement learning against adversarial perturbations on state observations

    Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho- Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems, 33:21024–21037, 2020

    2020

  34. [42]

    Adaptive reward-poisoning attacks against reinforcement learning

    Xuezhou Zhang, Yuzhe Ma, Adish Singla, and Xiaojin Zhu. Adaptive reward-poisoning attacks against reinforcement learning. In International Conference on Machine Learning , pages 11225–11234. PMLR, 2020

    2020

  35. [43]

    Sim-to-real transfer in deep reinforcement learning for robotics: a survey

    Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE symposium series on computational intelligence (SSCI), pages 737–744. IEEE, 2020

    2020

  36. [44]

    Cadre: A cascade deep reinforcement learning framework for vision-based autonomous urban driving

    Yinuo Zhao, KAun Wu, Zhiyuan Xu, Zhengping Che, Qi Lu, Jian Tang, and Chi Harold Liu. Cadre: A cascade deep reinforcement learning framework for vision-based autonomous urban driving. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3481–3489,...

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.