REVIEW 4 major objections 6 minor 13 references
Optimal Actuator Attacks on Autonomous Vehicles Using Reinforcement Learning
T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read This paper proposes a reinforcement-learning method for designing stealthy false-data injection attacks on autonomous-vehicle actuators, claiming to simultaneously increase trajectory tracking error, reduce the attacker's energy cost, and…
desk verdict A coherent, incremental extension of Wu et al. with a stealth reward, but the energy objective is mis-specified and the optimality/superiority claims are unsupported as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key machinery is a Markov decision process whose per-step reward is $R_k = J_t - J_e + J_s$, with the stealth term $J_s$ driven by the binary output of the same dynamic chi-square attack detector that is later used for evaluation. This detector-in-the-loop reward is what allows the attacker to learn policies that keep the estimation residue small enough to stay below the alarm threshold. The off-policy PPO and on-policy SAC algorithms serve as the optimizers, and the environment consists of a unicycle vehicle model with a Lyapunov-based tracking controller and an extended Kalman filter for state estimation and residue generation.
What would settle it
Re-evaluate the same trained PPO attacker against a detector with a different threshold, a different residue statistic, or a detector from a different family such as CUSUM or a learned anomaly detector, and measure detector recall; if recall rises substantially above the reported values, the claim that the attack is generally stealthy fails.
Extended reading notes
Core claim
The central claim is that an optimal stealthy actuator false-data injection attack for a nonlinear autonomous vehicle can be found by maximizing $J = \lim_{N\to\infty} \frac{1}{N}\sum_{k=1}^{N}\left(J_t - J_e + J_s\right)$, where $J_t$ is the tracking error, $J_e$ is the attacker's energy expenditure, and $J_s = \alpha(1-D_k)$ rewards the binary output of the attack detector when it fails to raise an alarm. The authors show that both PPO and SAC can optimize this non-convex objective when the detector's binary output is included in the reward, and that the resulting policies produce sparse, low-energy attacks that the dynamic chi-square detector frequently misses. They further claim that their attacker outperforms the previous RL-based attacker and secure controller on tracking error, energy, and stealthiness, and they identify two limitations of that controller: it is tuned to the specific attack learned by RL, and its stability proof does not cover the widened attack space.
Load-bearing premise
The stealthiness result assumes the attacker is trained and evaluated against the very same dynamic chi-square detector with the same threshold, so the learned stealth may be specific to that detector rather than a general property of the attack.
Editorial extensions
If this is right
- The reported PPO policy yields detector recall around 0.08 for a long attack and 0.15 for a short attack, meaning most attack steps go undetected; if true, residue-based detectors do not reliably expose learning-based actuator attacks.
- The same reward structure can be applied to other nonlinear vehicle models, since it needs only the vehicle model, a controller, and a detector output to train the attacker.
- Secure controllers trained against a fixed RL attacker, as in the prior work, will lose effectiveness against attackers that vary frequency, amplitude, or pattern, so defenses must be evaluated against adaptive stealthy attackers.
- The learned PPO attack uses roughly 2.5 times less energy than the SAC attack while also achieving lower tracking error, suggesting the attack is a targeted, efficient strategy rather than brute-force disturbance injection.
Reading between the lines
- Because the reward includes the detector's binary output, the policy may overfit to that specific detector; a defender who changes the detector or its threshold could break the attack's stealthiness, a scenario the paper does not test.
- The comparison with the prior secure controller may be confounded: that controller was designed for its own RL-based attack, so part of the reported gain could come from attacking a controller that is not tuned to this threat model.
- Training and evaluation share the same detector, so the low recall values in the tables could partly be an artifact of the reward hacking that detector; a cleaner test would withhold the detector's parameters during evaluation or use a separate detector for scoring.
- A natural next step is a game-theoretic model in which the defender adapts the detector after observing the attacker, turning the current static-detector assumption into an arms race between stealthy learning attacks and learning-based defenses.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes a reinforcement learning (PPO and SAC) approach for synthesizing actuator false data injection (FDI) attacks on a nonlinear autonomous vehicle model. The attack adds a disturbance d_k to the nominal control input u_k, and the reward is defined as R_k = J_t - J_e + J_s, where J_t is trajectory tracking error, J_e = u_k^T R u_k is claimed to represent attack energy, and J_s = α(1-D_k) is a stealth penalty derived from the binary output of a dynamic chi-square residue detector. The authors train for 50 episodes and report two scenarios with detector miss rates, energy consumption, and tracking error for both algorithms, asserting superiority over the RL-based attacker/controller of Wu et al. [6]. Appendices show detector operation and trajectory visualizations.
Significance. The problem of designing stealthy integrity attacks on autonomous vehicles is timely, and the paper's combination of a kinematic vehicle model, an EKF, and an RL-based attacker is a natural way to explore data-driven attack strategies. The paper offers a concrete reward formulation and a simulation pipeline with training curves and trajectory visualizations. However, the significance of the contribution rests entirely on three claims—optimality, energy efficiency, and stealthiness—and, as detailed in the major comments, none of these is currently supported by the evidence. In particular, the attack-energy term in the objective does not depend on the attacker's decision variable, the stealth evaluation is circular with respect to the detector used in training, and the quantitative comparison to [6] is absent from the tables. If these issues were corrected, the paper would be of interest to the control-theoretic security community; in its present form, the contribution is a preliminary demonstration rather than a verified result.
major comments (4)
- [Section II.B, Eq. (3) and (4)] The objective J in Eq. (3) contains the term J_e = u_k^T R u_k, where u_k is the defender's nominal control input, not the attacker's injected action d_k in Eq. (2). Since the attack model is x_{k+1} = f(x_k, u_k + d_k, w_k), the attacker controls d_k, so J_e is independent of the attack decision at the current step and does not penalize the energy spent by the attacker. Consequently, the RL policy is not optimizing attack energy, the columns labeled 'Energy Consumption' in Tables I and II do not measure it, and the claim that the attack 'spends little energy' is unsupported. The reward should use d_k^T R d_k (or a function of d_k) if the stated objective is to be optimized.
- [Sections III.A-III.B, Eq. (4), Fig. 3] The stealth penalty J_s = α(1 - D_k) is defined through D_k, the binary output of the dynamic chi-square detector described in Eq. (5) and Section III.A. The same detector is then used to compute 'Detector Recall' in Tables I and II. The reported low detection rates therefore reflect the attacker's specialization to this particular detector's statistic and threshold, not a general stealth property. To support the stealthiness claim, the authors should evaluate against different detectors, different thresholds, or a detector not seen during training.
- [Section III.B, Tables I and II] The claimed superiority over the RL-based attacker of [6] is not verifiable because no results from [6] are reported in Tables I or II or elsewhere; the tables contain only the proposed SAC and PPO runs. Moreover, no error bars or multiple seeds are reported, so the stability of the training is unknown. In addition, the metric labeled 'Detector Recall' is described as 'there is an attack but it is not detected,' which is the miss rate (false negative rate), not recall (true positive rate). The values reported (0.07-0.15) are therefore miss rates, and the text should be corrected accordingly.
- [Sections II.C and III.A] The paper repeatedly calls the resulting policy 'optimal' but provides no optimality guarantee or characterization. Training for '50 episodes until convergence' is not a valid convergence criterion for PPO/SAC on a non-convex stochastic problem, and no hyperparameter settings, network architectures, or reward scalings are given. At minimum, the authors should compare against a model predictive or exhaustive search baseline on the training objective, and report learning curves with variance. Absent that, the central optimality claim is not established.
minor comments (6)
- [Section II.C] The text states 'off-policy PPO' and 'on-policy SAC', but PPO is on-policy and SAC is off-policy; the labels are reversed.
- [Section III.A] The simulator parameters (wheelbase L, landmark position p, noise covariances Q_k and R_k, the reference trajectory, the time step, and the attack-cycle definitions for Tables I and II) are not provided, making the experiments unrepeatable.
- [Fig. 2] The y-axis label is missing, and the convergence claim would be stronger with shaded regions across multiple random seeds; as presented, a single run per algorithm is shown.
- [Section I] The font and spacing for 'A V' / 'A Vs' is inconsistent (e.g., 'A Vs' in the abstract, 'A V' throughout the text).
- [Fig. 3] The caption of Fig. 3 says 'the advanced residue-based attack detector misses many attack instances,' but the figure does not show the detection threshold or the chi-square statistic values, so the visual claim is difficult to verify.
- [References [6]] The paper never states the simulation scenario of [6] (e.g., whether the vehicle model, attacker power, or detector is the same), which is needed for a meaningful comparison.
Circularity Check
Detector recall in Tables I–II is the training reward term J_s/alpha restated, so the stealthiness claim is circular; the rest of the derivation is not.
-
self definitional
[Section II.B (Eq. (3)), II.C (Eq. (4)), III.A (Eq. (5)), III.B (Tables I–II)]
"“the corresponding cost function is Js = α(1 − Dk), where α is a scaling parameter, and Dk = {0, 1} is the binary output of the detector. ... We define our reward function as follows: Rk = Jt − Je + Js. ... Following [4], we design a dynamic χ2 attack detector ... For evaluation indicators, the detector recall TP/TP+FN reflects the stealthiness, where there is an attack but it is not detected.”"
The reported detector recall is algebraically the same quantity the policy was optimized to maximize: low recall = 1 − TP/(TP+FN) = FN/(TP+FN) = mean(1 − D_k) = J_s/alpha over attack steps. Because R_k = J_t − J_e + J_s is the training reward and the RL policy maximizes R_k, the policy is directly trained to drive D_k to 0 for this particular dynamic χ2 detector. Reporting low recall in Tables I–II is therefore a restatement of the training objective, not an independent test of stealthiness, and provides no evidence of generalization to other detectors, thresholds, or residue statistics.
full rationale
One central circular step exists: the stealthiness evidence is generated by the same detector whose binary output D_k appears in the optimized reward J_s = alpha(1 − D_k). Low detector recall is mathematically identical to high average J_s/alpha, so the headline stealthiness result reduces to the training reward by construction, giving a partial circularity score of 6. The tracking-error and optimality parts still have independent content in the sense that the tables report realized objective components of a trained policy, which is normal RL evaluation, not by-construction equivalence. A separate concern is that J_e = u_kᵀ R u_k uses the defender's control input, not the attacker's injection d_k, so the reported 'Energy Consumption' does not measure attack energy; however this is a specification/correctness mismatch rather than a circular derivation, so it does not further raise the circularity score. Self-citations [4] and [10] supply the detector family, and [4] is load-bearing because the detector is inside the training loop; the circularity would still hold if the detector came from an unrelated group, so the issue is training/evaluation overlap rather than self-citation per se.
Assumptions & free parameters
free parameters (4)
- alpha (stealth weighting) =
not reported
- detector threshold for chi2 detector =
not reported
- PPO and SAC hyperparameters =
not reported
- noise covariances Q_k, R_k and scenario constants =
not reported
assumptions (5)
- domain assumption Bicycle kinematic model (1) with zero-mean Gaussian noises accurately represents an autonomous vehicle for this attack study.
- domain assumption The dynamic chi2 detector based on EKF prediction residue (5) is representative of attack detectors found on modern AVs.
- ad hoc to paper PPO and SAC converge to the optimal policy after 50 training episodes.
- ad hoc to paper The attacker knows the system model, controller, EKF, and detector, and can add arbitrary d_k to u_k.
- ad hoc to paper Reward weights (alpha and the relative scaling of J_t and J_e) define the correct notion of optimality.
Cite this review
Pith. "Pith review of Optimal Actuator Attacks on Autonomous Vehicles Using Reinforcement Learning." pith.science (2026). https://pith.science/paper/GGMNRUZL
@misc{pith2026250207839,
author = {Pith},
title = {Pith review of: Optimal Actuator Attacks on Autonomous Vehicles Using Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/GGMNRUZL}},
note = {Machine review of arXiv:2502.07839}
}
read the original abstract
With the increasing prevalence of autonomous vehicles (AVs), their vulnerability to various types of attacks has grown, presenting significant security challenges. In this paper, we propose a reinforcement learning (RL)-based approach for designing optimal stealthy integrity attacks on AV actuators. We also analyze the limitations of state-of-the-art RL-based secure controllers developed to counter such attacks. Through extensive simulation experiments, we demonstrate the effectiveness and efficiency of our proposed method.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[6]
A secure robot learning framework for cyber attack scheduling and countermeasure,
C. Wu, W. Yao, W. Luo, W. Pan, G. Sun, H. Xie, and L. Wu, “A secure robot learning framework for cyber attack scheduling and countermeasure,” IEEE Transactions on Robotics , vol. 39, no. 5, pp. 3722–3738, 2023
work page 2023
-
[1]
APF-CPP: An artificial potential field based multi-robot online coverage path planning approach,
Z. Wang, X. Zhao, J. Zhang, N. Yang, P. Wang, J. Tang, J. Zhang, and L. Shi, “APF-CPP: An artificial potential field based multi-robot online coverage path planning approach,” IEEE Robotics and Automation Letters, 2024
work page 2024
-
[2]
MINER-RRT*: A hierarchical and fast trajectory planning framework in 3d cluttered environments,
P. Wang, J. Tang, H. W. Lin, F. Zhang, C. Wang, J. Wang, L. Shi, and M. Q.-H. Meng, “MINER-RRT*: A hierarchical and fast trajectory planning framework in 3d cluttered environments,” IEEE Transactions on Automation Science and Engineering , 2025
work page 2025
-
[3]
U. Ahmad, M. Han, A. Jolfaei, S. Jabbar, M. Ibrar, A. Erbad, H. H. Song, and Y . Alkhrijah, “A comprehensive survey and tutorial on smart vehicles: Emerging technologies, security issues, and solutions using machine learning,” IEEE Transactions on Intelligent Transportation Systems, 2024
work page 2024
-
[4]
Lasso-based detection and identification of actuator integrity attacks in remote control systems,
N. Yang, Z. Wang, T. Yang, Y . Li, and L. Shi, “Lasso-based detection and identification of actuator integrity attacks in remote control systems,” IEEE Transactions on Control Systems Technology , 2023
work page 2023
-
[5]
Learning-based dos attack power allocation in multiprocess systems,
M. Huang, K. Ding, S. Dey, Y . Li, and L. Shi, “Learning-based dos attack power allocation in multiprocess systems,” IEEE Transactions on Neural Networks and Learning Systems , vol. 34, no. 10, pp. 8017– 8030, 2022
work page 2022
-
[7]
Optimal data injection attacks in cyber- physical systems,
G. Wu, J. Sun, and J. Chen, “Optimal data injection attacks in cyber- physical systems,” IEEE Transactions on Cybernetics , vol. 48, no. 12, pp. 3302–3312, 2018
work page 2018
-
[8]
A class of optimal switching mixed data injection attack in cyber-physical systems,
S. Gao, H. Zhang, Z. Wang, and C. Huang, “A class of optimal switching mixed data injection attack in cyber-physical systems,”IEEE Robotics and Automation Letters , vol. 6, no. 2, pp. 1598–1605, 2021
work page 2021
Show all 13 references
-
[9]
Secure pose estimation for autonomous vehicles under cyber attacks,
Q. Liu, Y . Mo, X. Mo, C. Lv, E. Mihankhah, and D. Wang, “Secure pose estimation for autonomous vehicles under cyber attacks,” in IEEE Intelligent V ehicles Symposium, pp. 1583–1588, 2019
2019
-
[10]
QUADFormer: Learning- based detection of cyber attacks in quadrotor UA Vs,
P. Wang, Z. Yang, N. Yang, Z. Wang, J. Li, F. Zhang, C. Wang, J. Wang, M. Q.-H. Meng, and L. Shi, “QUADFormer: Learning- based detection of cyber attacks in quadrotor UA Vs,” arXiv preprint arXiv:2406.00707, 2024
2024 arXiv
-
[11]
Proximal policy optimization algorithms,
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017
2017 arXiv
-
[12]
Soft actor-critic algorithms and applications,
T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V . Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. , “Soft actor-critic algorithms and applications,” arXiv preprint arXiv:1812.05905 , 2018
2018 arXiv
-
[13]
Simultaneous stabilization and tracking of nonholonomic mobile robots: A lyapunov-based ap- proach,
Y . Wang, Z. Miao, H. Zhong, and Q. Pan, “Simultaneous stabilization and tracking of nonholonomic mobile robots: A lyapunov-based ap- proach,” IEEE Transactions on Control Systems Technology , vol. 23, no. 4, pp. 1440–1450, 2015
2015
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.