REVIEW 4 major objections 6 minor 20 references
Continual Adversarial Reinforcement Learning (CARL) of False Data Injection detection: forgetting and explainability
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A reinforcement-learned adversary drops a neural grid-attack detector's accuracy to 3.6%, and the proposed continual training loop must replay old attacks to avoid forgetting them.
desk verdict Solid, honest demonstration that an RL attack can crush an LSTM-based FDIA detector and that continual retraining forgets, but single-seed reporting leaves the headline numbers unproven. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is the CARL iteration loop (Algorithm 1): in each round n, an RL adversary A_n is trained by proximal policy optimization to maximize frequency deviation while being penalized whenever the current detector catches it; then an RL defender D_n (the multiclass classifier, warm-started from D_{n-1}) is trained to label the attacked bus, while the LSTM state predictor stays fixed. Each adversary observes the full frequency deviation and phase-angle state plus a normalized time step, and acts every d=6 timesteps by choosing a bus and a droop-coefficient tamper value from {−1, 0, 1}, with at most one bus modified at a time. The detector observes the window of recent states and outputs one of 11 classes (10 buses plus 'no attack'); detection succeeds when that class matches the attacked bus. The second key mechanism is R-CARL's rehearsal: instead of continuing the sequential loop, a final detector (and adversary) is trained against the entire library of previously generated agents, sampled with a fixed probability (80% synthetic A0, 20% uniform over RL adversaries). This joint-training scheme is what counters catastrophic forgetting.
What would settle it
Train the same adversary reward under a threat model that allows tampering with two or more droop coefficients simultaneously, or restart episodes from a distribution of initial frequency states, and check whether an RL adversary can still push the offline detector's accuracy below 10%; if detection accuracy stays high under these relaxed conditions, the paper's vulnerability claim would not transfer beyond its narrow setup.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a supervised neural FDIA detector can be systematically evaded by an RL-crafted adversary that switches the attacked bus between decision windows: after training A1 against the offline detector D0, detection accuracy collapses from 71.6% to 3.61% while the frequency-deviation reward jumps from 2.76 to 239.31, meaning the attack is both impactful and mostly undetected. The paper further establishes that the continual adversarial training loop it proposes (CARL), in which each new adversary is trained against the latest detector and each new detector against the latest adversary, forgets earlier attacks: already after one iteration D1 detects the original synthetic attack A0 at only 23.6%, and backward transfer degrades by factors of 1.7, 2.3, and 3.1 after successive iterations. The paper's remedy, R-CARL, trains one detector against the entire pool of adversaries sampled with a fixed probability schedule; that detector scores 98.79%, 100%, 98.799%, and 100% on the four RL adversaries and 75.42% on the synthetic attack, beating the offline baseline. Finally, comparing adversaries via transition-matrix differences and action histograms shows the CARL-trained adversaries behave nearly alike, indicating that very small changes in attack pattern suffice to fool a data-based detector.
Load-bearing premise
The weakest link is the threat model the authors adopt: every adversary can tamper with only one droop coefficient at a time, decides once every six timesteps, and all training and testing run from a single initial frequency condition, so the measured vulnerability and forgetting rates may not hold for less constrained real-world attackers or different grid operating points.
Editorial extensions
If this is right
- A single RL-trained adversary can reduce a supervised neural FDIA detector to 3.61% accuracy while inducing large frequency deviations, so offline-trained detectors should not be assumed robust against adaptive attackers.
- Continual adversarial retraining without memory of old attacks loses the ability to detect earlier attack types: after one CARL iteration, accuracy on the synthetic attack A0 drops from 71.6% to 23.6%, so sequential defense updates need a replay buffer or regularization.
- Joint training on the full adversary library (R-CARL) recovers and improves detection: 75.42% on A0 versus the 71.6% baseline, and near-perfect accuracy (98.8–100%) on the RL-trained adversaries, so multi-task adversarial training is the effective mitigation.
- The similarity of successive CARL adversaries (small transition-matrix differences) implies that minimal changes in the attack pattern—mainly which bus is targeted next—are enough to fool a data-based detector, so detector improvements should target sensitivity to attack timing and switching.
Reading between the lines
- We infer that the single-initial-condition and single-bus threat model is the main boundary of the claimed vulnerability; if real adversaries coordinate attacks on several buses or act at arbitrary times, the measured accuracy drop and forgetting rates could change substantially.
- The fact that the R-CARL-trained adversary A never mutes and always attacks a bus suggests that when a detector knows all known attack types, the remaining attack surface shifts toward persistent rather than intermittent tampering—an implicit design hint for defenders.
- The transition-matrix comparison method could be reused as a monitoring tool in deployment: if new adversaries differ from known ones, a defender could flag that the attack library needs updating before retraining.
- The near-identical performance of trained adversaries across iterations hints that the classifier's decision boundary has large flat regions that are easy to cross; a testable extension would be to measure the minimal perturbation in bus-switching probability needed to evade each detector.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CARL, a continual adversarial RL procedure in which an RL adversary learns false data injection attacks against a neural FDIA detector, and the detector is then retrained against that adversary. The authors report in Table I that an RL-trained adversary A1 reduces the offline detector D0's accuracy from 71.6% to 3.61% while inducing large frequency deviations, and that sequential retraining of detectors D1-D4 exhibits catastrophic forgetting. They then propose R-CARL, a rehearsal variant that trains a final detector against the library of adversaries, report high accuracies for that detector, and analyze adversary policies via transition matrices and action histograms as an explainability device.
Significance. If the 3.61% result is robust, this is a practically relevant demonstration that a state-predictor-plus-classifier FDIA detector can be systematically evaded by a learned attack policy, and the continual/rehearsal training comparison is a useful step toward adversarial training for power-system detectors. The paper is clearly written and the framework is specified precisely in Algorithm 1 and Eq. (4); the authors also honestly acknowledge the lack of adversary diversity and the single-initial-condition simplification. The main weakness is statistical: the central numbers are single-realization estimates with no error bars, and the R-CARL accuracy is acknowledged to be inflated by training/test overlap. The explainability contribution is currently descriptive rather than validated. The result is therefore promising but not yet established at the confidence level required by the central claims.
major comments (4)
- [Section III-A, Table I] The central vulnerability result, A1 reducing D0 accuracy from 71.6% to 3.61% (Table I), is based on a single randomly sampled realization of a stochastic PPO adversary; the paper states this explicitly in Section III-A. No training seeds, repeated evaluation episodes, or confidence intervals are reported anywhere in Table I, so the 3.61% value could be an outlier realization rather than a reliable property of the learned policy. Because the forgetting rates in Section III-B and the R-CARL comparison in Section III-C are derived from the same table, this single-realization issue undermines the empirical support for the paper's main claims. Please report means and standard deviations over multiple independent seeds and evaluation rollouts for every entry in Table I.
- [Section III-C, final paragraph] The paper acknowledges that 'a large part of the testing data was seen during training' because of the adversary's lack of diversity. This directly confounds the claim that R-CARL addresses catastrophic forgetting: the near-100% accuracies of D against A1-A4 may simply reflect memorization of the training adversary rollouts. To support the rehearsal advantage, evaluate on held-out realizations of each adversary (for example, attack rollouts generated after training with different random seeds) or report a quantitative diversity measure of the adversary policies. As it stands, the R-CARL comparison is not statistically separated from an overfitting artifact.
- [Section III-D, Eq. (5)] The explainability contribution is not operationalized. Transition-matrix distances and droop-coefficient histograms describe the learned policies, but the paper never defines what a practitioner learns about detector deficiencies or how that information can be validated. The Abstract's claim that the approach 'pinpoint[s] the deficiencies of data-based detection' is not supported by the descriptive analysis in this section. Please specify a concrete explanation task and how the CARL knowledge pool addresses it, or revise the contribution to say that the framework provides a library of adversaries rather than an explainability method.
- [Conclusions and Section II-A] The Conclusions acknowledge that 'we simplified the training process by focusing on a single initial condition.' Because all training and evaluation use the same initial condition for the frequency dynamics, the demonstrated vulnerability and forgetting rates may not transfer to other operating points. Please add a sensitivity study over initial conditions, or clearly qualify the abstract and contribution claims as being for a single operating point.
minor comments (6)
- [Algorithm 1, line 33] The defender's reward is written as rA_t; this should be rD_t to match Eq. (3).
- [Figure 3] The axis labels contain garbled Unicode sequences such as '/uni00000013...'; the figure should be regenerated with readable labels.
- [Table I] The frequency-reward value for a given adversary is repeated across all defender rows; consider reporting these values once in a separate table or column.
- [Eq. (4)] The sampling probabilities P(A0)=0.8 and P(Ai)=0.2/N are fixed without sensitivity analysis; a sentence justifying these choices or showing robustness would strengthen the R-CARL section.
- [References] References [6] and [9] are identified only as 'arXiv preprint' and 'Under Review'; since [9] in particular is not publicly accessible, please provide a DOI or publication status where available.
- [Section II-B3] The asymmetry between A0, which targets the same bus for the entire episode, and the RL adversaries, which can switch buses, should be stated more prominently as a limitation of the baseline comparison.
Circularity Check
No significant circularity: the central adversarial results are empirical co-evolution outcomes, not inputs by construction.
full rationale
The paper's central claims are empirical demonstrations rather than derived predictions. The adversary A1 is trained via PPO to maximize a reward (Eq. 2) that penalizes detection, so evading D0 is part of its objective; however, the reported 3.61% accuracy is a measured outcome of that optimization, not a value fitted to the test result or implied by the reward definition alone. The detector D0 is said to be 'trained following Ref. [9]', a same-author under-review paper, and the LSTM classifier follows Ref. [6], also same-author; these are self-citations, but they supply a benchmark architecture and training procedure rather than a conclusion, and the table reports measurements of that benchmark inside the present paper. The forgetting rates and R-CARL improvements are computed from the authors' own experiments and compared against the same table. The R-CARL evaluation does overlap with training -- the paper explicitly states 'a large part of the testing data was seen during training' -- and this is an acknowledged validity caveat about memorization versus generalization, not a hidden circular reduction of the claim to its own input. Similarly, the single-initial-condition limitation is acknowledged in the Conclusions. No equation in the paper reduces a predicted quantity to an input by construction, and no load-bearing result is imported solely through a self-citation chain. The main concerns are statistical robustness (single stochastic realization, no seeds) and external generality, which are correctness risks, not circularity.
Assumptions & free parameters
free parameters (6)
- reward scaling factor c =
0.1
- detection penalizer p =
0.1
- detection period d =
6
- R-CARL sampling probability P(A0) =
0.8
- RL network architecture =
2 hidden layers of 256 units
- number of training episodes K
assumptions (4)
- domain assumption Swing equation model with droop control accurately represents primary frequency dynamics
- domain assumption Adversary can remotely access all smart inverters and alter droop coefficients, with at most one bus attacked at a time and decisions every d timesteps
- domain assumption The LSTM state predictor remains fixed and only the classifier is retrained
- ad hoc to paper Single initial condition for frequency dynamics
Cite this review
Pith. "Pith review of Continual Adversarial Reinforcement Learning (CARL) of False Data Injection detection: forgetting and explainability." pith.science (2026). https://pith.science/paper/WNQ6SQCR
@misc{pith2026241110367,
author = {Pith},
title = {Pith review of: Continual Adversarial Reinforcement Learning (CARL) of False Data Injection detection: forgetting and explainability},
year = {2026},
howpublished = {\url{https://pith.science/paper/WNQ6SQCR}},
note = {Machine review of arXiv:2411.10367}
}
read the original abstract
False data injection attacks (FDIAs) on smart inverters are a growing concern linked to increased renewable energy production. While data-based FDIA detection methods are also actively developed, we show that they remain vulnerable to impactful and stealthy adversarial examples that can be crafted using Reinforcement Learning (RL). We propose to include such adversarial examples in data-based detection training procedure via a continual adversarial RL (CARL) approach. This way, one can pinpoint the deficiencies of data-based detection, thereby offering explainability during their incremental improvement. We show that a continual learning implementation is subject to catastrophic forgetting, and additionally show that forgetting can be addressed by employing a joint training strategy on all generated FDIA scenarios.
Figures
Reference graph
Works this paper leans on
-
[1]
Inertia and the power grid: A guide without the spin,
P. Denholm, T. Mai, R. W. Kenyon, B. Kroposki, and M. O’Malley, “Inertia and the power grid: A guide without the spin,” National Renewable Energy Lab.(NREL), Golden, CO (United States), Tech. Rep., 2020
work page 2020
-
[2]
Virtual Inertia: Current Trends and Future Directions,
U. Tamrakar, D. Shrestha, M. Maharjan, B. P. Bhattarai, T. M. Hansen, and R. Tonkoski, “Virtual Inertia: Current Trends and Future Directions,” Applied Sciences, vol. 7, no. 7, 29 pages, July 2017
work page 2017
-
[3]
Electric power grid resilience to cyber adversaries,
T. Nguyen, S. Wang, M. Alhazmi et al., “Electric power grid resilience to cyber adversaries,” IEEE Access, vol. 8, pp. 87 592–87 608, 2020
work page 2020
-
[4]
False data injection attacks against state estimation in electric power grids,
Y . Liu, P. Ning, and M. K. Reiter, “False data injection attacks against state estimation in electric power grids,” ACM Trans. on Info. and Syst. Security, vol. 14, no. 1, pp. 1–33, 2011
work page 2011
-
[5]
A survey on the detection algo- rithms for false data injection attacks in smart grids,
A. S. Musleh, G. Chen, and Z. Y . Dong, “A survey on the detection algo- rithms for false data injection attacks in smart grids,” IEEE Transactions on Smart Grid , vol. 11, no. 3, pp. 2218–2234, 2019
work page 2019
-
[6]
A. Sahu, T. Nguyen, K. Chen, X. Zhang, and M. Hassanaly, “Detection of False Data Injection Attacks (FDIA) on Power Dynamical Systems With a State Prediction Method,” arXiv preprint arXiv:2409.04609 , 2024
work page Pith review arXiv 2024
-
[7]
Self-supervised learning of adversarial example: Towards good generalizations for deepfake detection,
L. Chen, Y . Zhang, Y . Song, L. Liu, and J. Wang, “Self-supervised learning of adversarial example: Towards good generalizations for deepfake detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 18 710–18 719
2022
-
[8]
Discovery of false data injection schemes on frequency controllers with reinforcement learning,
R. Prasad, M. Hassanaly, X. Zhang, and A. Sahu, “Discovery of false data injection schemes on frequency controllers with reinforcement learning,” in 2024 IEEE Power & Energy Society General Meeting (PESGM). IEEE, 2024, pp. 1–5
work page 2024
Show all 20 references
-
[9]
An adversarial multi-agent reinforcement learning framework for proactive false data injection detection,
K. Chen, T. Nguyen, and M. Hassanaly, “An adversarial multi-agent reinforcement learning framework for proactive false data injection detection,” Under Review, 2024
2024
-
[10]
Explainability in deep reinforcement learning,
A. Heuillet, F. Couthouis, and N. D ´ıaz-Rodr´ıguez, “Explainability in deep reinforcement learning,” Knowledge-Based Systems , vol. 214, p. 106685, 2021
2021
-
[11]
Autonomous self- explanation of behavior for interactive reinforcement learning agents,
Y . Fukuchi, M. Osawa, H. Yamakawa, and M. Imai, “Autonomous self- explanation of behavior for interactive reinforcement learning agents,” in Proceedings of the 5th International Conference on Human Agent Interaction, 2017, pp. 97–101
2017
-
[12]
An empirical investigation of catastrophic forgetting in gradient-based neural networks,
I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y . Bengio, “An empirical investigation of catastrophic forgetting in gradient-based neural networks,” arXiv preprint arXiv:1312.6211 , 2013
2013 arXiv
-
[13]
Reinforcement learning for optimal primary frequency control: A Lyapunov approach,
W. Cui, Y . Jiang, and B. Zhang, “Reinforcement learning for optimal primary frequency control: A Lyapunov approach,” IEEE Transactions on Power Systems, vol. 38, no. 2, pp. 1676–1688, 2022
2022
-
[14]
Overcoming catastrophic forgetting in neural networks,
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska et al. , “Overcoming catastrophic forgetting in neural networks,” Pro- ceedings of the national academy of sciences , vol. 114, no. 13, pp. 3521...
2017
-
[15]
Learning without forgetting,
Z. Li and D. Hoiem, “Learning without forgetting,” IEEE transactions on pattern analysis and machine intelligence , vol. 40, no. 12, pp. 2935– 2947, 2017
2017
-
[16]
Progressive neural networks,
A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell, “Progressive neural networks,” arXiv preprint arXiv:1606.04671 , 2016
2016 arXiv
-
[17]
Gradient episodic memory for continual learning,
D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for continual learning,” Advances in neural information processing systems , vol. 30, 2017
2017
-
[18]
A practical method for the direct analysis of transient stability,
T. Athay, R. Podmore, and S. Virmani, “A practical method for the direct analysis of transient stability,” IEEE Transactions on Power Apparatus and Systems, vol. PAS-98, no. 2, pp. 573–584, 1979
1979
-
[19]
Ray: A distributed framework for emerging AI applications,
P. Moritz, R. Nishihara, Wang et al., “Ray: A distributed framework for emerging AI applications,” in 13th USENIX symp. on operating syst. design and impl. , 2018, pp. 561–577
2018
-
[20]
Policy distillation,
A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirk- patrick, R. Pascanu, V . Mnih, K. Kavukcuoglu, and R. Hadsell, “Policy distillation,” arXiv preprint arXiv:1511.06295 , 2015
2015 arXiv
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.