pith. sign in

arxiv: 2402.05284 · v2 · submitted 2024-02-07 · 💻 cs.LG

Analyzing Adversarial Inputs in Deep Reinforcement Learning

Davide Corsi , Guy Amir , Guy Katz , Alessandro Farinelli This is my paper

Pith reviewed 2026-05-24 03:38 UTC · model grok-4.3

classification 💻 cs.LG
keywords adversarial inputsdeep reinforcement learningformal verificationsafety evaluationAdversarial Ratepolicy robustnessinput perturbation
0
0 comments X

The pith

The Adversarial Rate metric partitions input space to quantify and visualize how small perturbations cause deep reinforcement learning policies to fail.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper develops a metric called the Adversarial Rate, adapted from the ProVe verification family, that splits the input domain of a DRL agent into subregions. Within each subregion the metric counts how often tiny input changes flip the agent's decisions into unsafe actions. This produces both a numerical score of vulnerability and a spatial map showing where attacks are most effective. The authors supply algorithms to compute the metric and run it on trained policies to illustrate concrete safety risks. From the results they extract practical steps for reducing exposure to such perturbations.

Core claim

By adapting the ProVe-family metric into the Adversarial Rate and partitioning the input domain into subregions, the approach enables both quantitative measurement and spatial visualization of adversarial inputs that cause DRL policies to produce unsafe outputs, supplying an evaluation framework, associated algorithms, and empirical evidence that these inputs threaten system safety along with mitigation guidelines.

What carries the argument

The Adversarial Rate metric, which partitions the input domain into subregions to quantify and spatially visualize the frequency of adversarial inputs that induce unsafe policy decisions.

If this is right

  • DRL policies can be scored for their exposure to adversarial inputs across different parts of the input space.
  • Spatial maps produced by the metric identify concrete regions where small changes are most likely to trigger unsafe actions.
  • The supplied algorithms make the metric computable for existing trained networks.
  • Empirical runs demonstrate measurable safety degradation under adversarial inputs.
  • The analysis yields concrete guidelines for adjusting training or architecture to lower vulnerability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same partitioning idea could be applied to other sequential decision models such as recurrent networks or planning agents.
  • Real-time monitoring systems might use the metric to flag when an agent enters a high-risk input region.
  • Training loops could incorporate the Adversarial Rate as an auxiliary loss to discourage policies from depending on fragile input areas.

Load-bearing premise

The input domain can be partitioned into subregions that allow accurate counting and visualization of adversarial effects.

What would settle it

Running the metric on a DRL policy and finding that regions flagged as high-adversarial-rate produce no more safety failures under perturbation than low-rate regions.

Figures

Figures reproduced from arXiv: 2402.05284 by Alessandro Farinelli, Davide Corsi, Guy Amir, Guy Katz.

Figure 1
Figure 1. Figure 1: A toy DNN. 2.2 Deep Reinforcement Learning (DRL) DRL stands as a prominent paradigm in machine learning, wherein a DNN-based re￾inforcement learning (RL) agent engages with an environment over multiple time-steps t ∈ {0, 1, 2, ...}, with the aim of learning to map an input (from the sate-space S) to an appropriate output action (from the action-space A). At each discrete time-step, the DRL agent observes t… view at source ↗
Figure 2
Figure 2. Figure 2: An example of interval propagation for a reachability approach to verification. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: A toy example for the iterative splitting procedure of ProVe. In the case depicted in the first figure, it is not possible to formally prove where Y 1 is greater than Y 2 given that the upper and lower bounds overlap. In the second and third figures, the iterative splitting procedure allows the division of the input domain into safe and unsafe regions. impossible to provide a formal answer as the propagate… view at source ↗
Figure 4
Figure 4. Figure 4: The Jumping World environment analyzed for our experimental evaluation. On the left is a screenshot from the simulation and on the right are the empirical results of our training phase. position within the continuous space of the cell. Finally, the target position changes over the episodes, and hence we designed the coordinates of the target to also be part of the observation space, and hence an input to t… view at source ↗
Figure 5
Figure 5. Figure 5: The Robotic Mapless Navigation environments analyzed for our experimental evaluation. On the left is a screenshot from the simulation and on the right are the empirical results of our training phase. the safety properties can be formulated as follows: “if the agent identifies an obstacle in its proximity, then the agent must not move towards that direction in the next time-step”. 4.2 Robotic Mapless Naviga… view at source ↗
Figure 6
Figure 6. Figure 6: Jumping World: A heatmap that highlights the concentration of adversarial inputs (identified via the Adversarial Rate metric) per each cell. hyperparameters, differing only in the random initialization of the parameters. Next, we divided the (continuous) input space into subregions and used ProVe to assess the ratio of adversarial inputs (and effectively, the Adversarial Rate) in each subregion. Surprising… view at source ↗
Figure 7
Figure 7. Figure 7: Jumping World: A qaualitative analysis of the temporal distribution. Implications. Our findings lead to an important conclusion: it is very challenging to solve the susceptibility of an agent to adversarial inputs, during training. Specifically, a main challenge is that occurrences of unsafe regions “shift” during training; hence, even if we were to identify such an unsafe region at a given time-step, and … view at source ↗
Figure 8
Figure 8. Figure 8: Jumping World: The unsafe regions computed on the best model, obtained with 4 varying sizes of the neural network. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Jumping World: The unsafe regions computed on the best model obtained, with 4 varying activation function types. experimental setup, with the difference of selecting the models based on 4 types of activation functions as described in [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
read the original abstract

In recent years, Deep Reinforcement Learning (DRL) has become a popular paradigm in machine learning due to its successful applications to real-world and complex systems. However, even the state-of-the-art DRL models have been shown to suffer from reliability concerns -- for example, their susceptibility to adversarial inputs, i.e., small and abundant input perturbations that can fool the models into making unpredictable and potentially dangerous decisions. This drawback limits the deployment of DRL systems in safety-critical contexts, where even a small error cannot be tolerated. In this work, we present a comprehensive analysis of the characterization of adversarial inputs, through the lens of formal verification. Specifically, we present the Adversarial Rate, a metric adapted from the ProVe family, for the systematic evaluation of adversarial inputs in DRL, which partitions the input domain into subregions to enable both quantification and spatial visualization of adversarial inputs. The main contribution of this work is to provide a comprehensive evaluation framework for the effect of adversarial inputs on DRL policies. We present a set of tools and algorithms for its computation. Our analysis empirically demonstrates how adversarial inputs can affect the safety of a given DRL system with respect to such perturbations. Moreover, we analyze the behavior of these configurations to suggest several useful practices and guidelines to help mitigate the vulnerability of trained DRL networks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The paper claims to introduce the Adversarial Rate, a metric adapted from the ProVe family of formal verification techniques, as a framework for systematically evaluating adversarial inputs in Deep Reinforcement Learning policies. It partitions the input domain into subregions to enable both quantitative measurement and spatial visualization of adversarial perturbations, provides associated tools and algorithms, empirically demonstrates impacts on DRL safety, and derives mitigation guidelines from the analysis.

Significance. If the empirical results and formal verification adaptations are rigorously supported, the work could contribute a practical evaluation framework for assessing and improving robustness in DRL systems deployed in safety-critical settings. The emphasis on input-domain partitioning and visualization offers a potentially useful lens beyond standard attack success rates.

major comments (1)
  1. No section, equation, or table is available for citation because the provided manuscript consists only of the abstract; the central claims regarding the Adversarial Rate adaptation, input partitioning procedure, and empirical safety demonstrations cannot be evaluated for soundness or load-bearing assumptions without the methods, algorithms, or results sections.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their review. The primary concern appears to stem from a possible issue with the manuscript version provided, as the full paper (including all sections, equations, tables, methods, algorithms, and results) is available both in the submission and on arXiv:2402.05284. We address this point below.

read point-by-point responses

  1. Referee: No section, equation, or table is available for citation because the provided manuscript consists only of the abstract; the central claims regarding the Adversarial Rate adaptation, input partitioning procedure, and empirical safety demonstrations cannot be evaluated for soundness or load-bearing assumptions without the methods, algorithms, or results sections.

    Authors: We apologize if only the abstract was visible in the review materials. The complete manuscript is part of the submission and publicly available at arXiv:2402.05284, containing dedicated sections on the Adversarial Rate metric (adapted from ProVe), the input-domain partitioning procedure, associated algorithms and tools, formal definitions, empirical evaluations on DRL safety, visualization methods, and derived mitigation guidelines. All claims are supported by these sections, equations, and results. We are happy to resubmit the full PDF or direct the referee to specific citations within the arXiv version. revision: no

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper introduces the Adversarial Rate as an adaptation of the ProVe-family metric to partition input domains for quantifying and visualizing adversarial inputs in DRL policies. No derivation chain, equation, or central claim reduces by construction to a fitted parameter, self-definition, or load-bearing self-citation. The framework is presented as a set of tools and algorithms whose empirical results stand independently of any internal redefinition or renaming of prior results. The provided abstract and description contain no evidence of the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review yields no identifiable free parameters, axioms, or invented entities; the central claim rests on the unelaborated applicability of formal verification and domain partitioning.

pith-pipeline@v0.9.0 · 5762 in / 1158 out tokens · 24055 ms · 2026-05-24T03:38:13.911408+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages · 6 internal anchors

  1. [1]

    Solving Rubik's Cube with a Robot Hand

    I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik’s cube with a robot hand. arXiv preprint arXiv:1910.07113,

    work page internal anchor Pith review Pith/arXiv arXiv 1910

  2. [2]

    End to End Learning for Self-Driving Cars

    M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    H., and Ben- gio, Y

    23 Corsi, Amir, Katz, and F arinelli M. Chevalier-Boisvert, D. Bahdanau, S. Lahlou, L. Willems, C. Saharia, T. H. Nguyen, and Y. Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprint arXiv:1810.08272 ,

    work page arXiv

  4. [4]

    Minigrid & mini- world: Modular & customizable reinforcement learning environments for goal-oriented tasks

    M. Chevalier-Boisvert, B. Dai, M. Towers, R. de Lazcano, L. Willems, S. Lahlou, S. Pal, P. S. Castro, and J. Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, abs/2306.13831,

    work page arXiv

  5. [5]

    H.-T. L. Chiang, A. Faust, M. Fiser, and A. Francis. Learning navigation behaviors end-to-end with autorl. IEEE Robotics and Automation Letters , 4(2):2007–2014,

    work page 2007

  6. [6]

    S. G. Clarke and I. Hwang. Deep reinforcement learning control for aerobatic maneuvering of agile fixed-wing aircraft. In AIAA Scitech 2020 Forum , page 0136,

    work page 2020

  7. [7]

    Corsi, R

    D. Corsi, R. Yerushalmi, G. Amir, A. Farinelli, D. Harel, and G. Katz. Constrained reinforcement learning for robotics via scenario-based programming. arXiv preprint arXiv:2206.09603,

    work page arXiv

  8. [8]

    Corsi, L

    D. Corsi, L. Marzari, A. Pore, A. Farinelli, A. Casals, P. Fiorini, and D. Dall’Alba. Con- strained reinforcement learning and formal verification for safe colonoscopy navigation. arXiv preprint arXiv:2303.03207 ,

    work page arXiv

  9. [9]

    G. Katz, D. A. Huang, D. Ibeling, K. Julian, C. Lazarus, R. Lim, P. Shah, S. Thakoor, H. Wu, A. Zelji´ c, et al. The marabou framework for verification and analysis of deep neural networks. In Computer Aided Verification: 31st International Conference, CAV 2019, New York City, NY, USA, July 15-18, 2019, Proceedings, Part I 31 , pages 443–452. Springer,

    work page 2019

  10. [10]

    T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 ,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Marchesini and A

    E. Marchesini and A. Farinelli. Discrete deep reinforcement learning for mapless navigation. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 10688–10694. IEEE,

    work page 2020

  12. [12]

    Marchesini and A

    E. Marchesini and A. Farinelli. Enhancing deep reinforcement learning approaches for multi-robot navigation via single-robot evolutionary policy search. In 2022 International Conference on Robotics and Automation (ICRA) . IEEE,

    work page 2022

  13. [13]

    Marchesini, D

    E. Marchesini, D. Corsi, and A. Farinelli. Benchmarking safe deep reinforcement learning in aquatic navigation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 5590–5595. IEEE,

    work page 2021

  14. [14]

    Marzari, D

    L. Marzari, D. Corsi, F. Cicalese, and A. Farinelli. The #dnn-verification problem: Counting unsafe inputs for deep neural networks. In International Joint Conference on Artificial Intelligence (IJCAI), 2023a. L. Marzari, D. Corsi, E. Marchesini, A. Farinelli, and F. Cicalese. Enumerating safe regions in deep neural networks with provable probabilistic gu...

    work page arXiv

  15. [15]

    A. Pore, D. Corsi, E. Marchesini, D. Dall’Alba, A. Casals, A. Farinelli, and P. Fiorini. Safe reinforcement learning using formal verification for tissue retraction in autonomous robotic-assisted surgery. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 4025–4031. IEEE,

    work page 2021

  16. [16]

    A. Ray, J. Achiam, and D. Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708 , 7(1):2,

    work page internal anchor Pith review Pith/arXiv arXiv 1910

  17. [17]

    High-Dimensional Continuous Control Using Generalized Advantage Estimation

    J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889–1897. PMLR, 2015a. J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 , 2015b. J. S...

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    Srinivasan, B

    K. Srinivasan, B. Eysenbach, S. Ha, J. Tan, and C. Finn. Learning to be safe: Deep rl with a safety critic. arXiv preprint arXiv:2010.14603 ,

    work page arXiv 2010

  19. [19]

    Intriguing properties of neural networks

    C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 ,

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    L. Tai, G. Paolo, and M. Liu. Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 31–36. IEEE,

    work page 2017

  21. [21]

    Tessler, D

    C. Tessler, D. J. Mankowitz, and S. Mannor. Reward constrained policy optimization. In 7th International Conference on Learning Representations, ICLR, 2019 ,

    work page 2019

  22. [22]

    https://arxiv.org/abs/2401.14461

    Technical Report. https://arxiv.org/abs/2401.14461. P. R. Wurman, S. Barrett, K. Kawamoto, J. MacGlashan, K. Subramanian, T. J. Walsh, R. Capobianco, A. Devlic, F. Eckert, F. Fuchs, et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602(7896):223–228,

    work page arXiv

  23. [23]

    Q. Yang, T. D. Sim˜ ao, N. Jansen, S. H. Tindemans, and M. T. Spaan. Training and transferring safe policies in reinforcement learning. In AAMAS 2022 Workshop on Adaptive Learning Agents, 2022a. X. Yang, T. Yamaguchi, H.-D. Tran, B. Hoxha, T. T. Johnson, and D. Prokhorov. Neural network repair with reachability analysis. In International Conference on For...

    work page 2022