Pith. sign in

REVIEW 4 major objections 6 minor 20 references

Continual Adversarial Reinforcement Learning (CARL) of False Data Injection detection: forgetting and explainability

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read A reinforcement-learned adversary drops a neural grid-attack detector's accuracy to 3.6%, and the proposed continual training loop must replay old attacks to avoid forgetting them.

desk verdict Solid, honest demonstration that an RL attack can crush an LSTM-based FDIA detector and that continual retraining forgets, but single-seed reporting leaves the headline numbers unproven. read the letter →

arxiv 2411.10367 v1 pith:WNQ6SQCR submitted 2024-11-15 cs.LG cs.AI

classification cs.LGcs.AI
keywords falsedatainjectionattackreinforcementlearningcontinualcatastrophicforgettingfrequencycontrolsmartinverteradversarialdetectionexplainability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that data-based detectors of false data injection attacks (FDIAs) on smart inverters remain vulnerable to impactful and stealthy adversarial examples, even when the detector uses an LSTM state predictor plus a trained classifier. Its core demonstration is quantitative: one reinforcement-learned adversary, A1, drops the offline detector D0's accuracy from 71.6% to 3.61% while driving large frequency deviations. The paper then proposes a continual adversarial RL (CARL) loop that alternately trains an attacker and a defender, and shows this loop suffers catastrophic forgetting—after one iteration, detection of the original synthetic attack falls to 23.6%. A rehearsal variant, R-CARL, that jointly trains a detector against the whole library of generated adversaries, restores and even exceeds the original performance, while the library of adversaries doubles as an explainable record of how attack strategies evolve.

What carries the argument

The central machinery is the CARL iteration loop (Algorithm 1): in each round n, an RL adversary A_n is trained by proximal policy optimization to maximize frequency deviation while being penalized whenever the current detector catches it; then an RL defender D_n (the multiclass classifier, warm-started from D_{n-1}) is trained to label the attacked bus, while the LSTM state predictor stays fixed. Each adversary observes the full frequency deviation and phase-angle state plus a normalized time step, and acts every d=6 timesteps by choosing a bus and a droop-coefficient tamper value from {−1, 0, 1}, with at most one bus modified at a time. The detector observes the window of recent states and outputs one of 11 classes (10 buses plus 'no attack'); detection succeeds when that class matches the attacked bus. The second key mechanism is R-CARL's rehearsal: instead of continuing the sequential loop, a final detector (and adversary) is trained against the entire library of previously generated agents, sampled with a fixed probability (80% synthetic A0, 20% uniform over RL adversaries). This joint-training scheme is what counters catastrophic forgetting.

What would settle it

Train the same adversary reward under a threat model that allows tampering with two or more droop coefficients simultaneously, or restart episodes from a distribution of initial frequency states, and check whether an RL adversary can still push the offline detector's accuracy below 10%; if detection accuracy stays high under these relaxed conditions, the paper's vulnerability claim would not transfer beyond its narrow setup.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a supervised neural FDIA detector can be systematically evaded by an RL-crafted adversary that switches the attacked bus between decision windows: after training A1 against the offline detector D0, detection accuracy collapses from 71.6% to 3.61% while the frequency-deviation reward jumps from 2.76 to 239.31, meaning the attack is both impactful and mostly undetected. The paper further establishes that the continual adversarial training loop it proposes (CARL), in which each new adversary is trained against the latest detector and each new detector against the latest adversary, forgets earlier attacks: already after one iteration D1 detects the original synthetic attack A0 at only 23.6%, and backward transfer degrades by factors of 1.7, 2.3, and 3.1 after successive iterations. The paper's remedy, R-CARL, trains one detector against the entire pool of adversaries sampled with a fixed probability schedule; that detector scores 98.79%, 100%, 98.799%, and 100% on the four RL adversaries and 75.42% on the synthetic attack, beating the offline baseline. Finally, comparing adversaries via transition-matrix differences and action histograms shows the CARL-trained adversaries behave nearly alike, indicating that very small changes in attack pattern suffice to fool a data-based detector.

Load-bearing premise

The weakest link is the threat model the authors adopt: every adversary can tamper with only one droop coefficient at a time, decides once every six timesteps, and all training and testing run from a single initial frequency condition, so the measured vulnerability and forgetting rates may not hold for less constrained real-world attackers or different grid operating points.

Editorial extensions

If this is right

  • A single RL-trained adversary can reduce a supervised neural FDIA detector to 3.61% accuracy while inducing large frequency deviations, so offline-trained detectors should not be assumed robust against adaptive attackers.
  • Continual adversarial retraining without memory of old attacks loses the ability to detect earlier attack types: after one CARL iteration, accuracy on the synthetic attack A0 drops from 71.6% to 23.6%, so sequential defense updates need a replay buffer or regularization.
  • Joint training on the full adversary library (R-CARL) recovers and improves detection: 75.42% on A0 versus the 71.6% baseline, and near-perfect accuracy (98.8–100%) on the RL-trained adversaries, so multi-task adversarial training is the effective mitigation.
  • The similarity of successive CARL adversaries (small transition-matrix differences) implies that minimal changes in the attack pattern—mainly which bus is targeted next—are enough to fool a data-based detector, so detector improvements should target sensitivity to attack timing and switching.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the single-initial-condition and single-bus threat model is the main boundary of the claimed vulnerability; if real adversaries coordinate attacks on several buses or act at arbitrary times, the measured accuracy drop and forgetting rates could change substantially.
  • The fact that the R-CARL-trained adversary A never mutes and always attacks a bus suggests that when a detector knows all known attack types, the remaining attack surface shifts toward persistent rather than intermittent tampering—an implicit design hint for defenders.
  • The transition-matrix comparison method could be reused as a monitoring tool in deployment: if new adversaries differ from known ones, a defender could flag that the attack library needs updating before retraining.
  • The near-identical performance of trained adversaries across iterations hints that the classifier's decision boundary has large flat regions that are easy to cross; a testable extension would be to measure the minimal perturbation in bus-switching probability needed to evade each detector.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes CARL, a continual adversarial RL procedure in which an RL adversary learns false data injection attacks against a neural FDIA detector, and the detector is then retrained against that adversary. The authors report in Table I that an RL-trained adversary A1 reduces the offline detector D0's accuracy from 71.6% to 3.61% while inducing large frequency deviations, and that sequential retraining of detectors D1-D4 exhibits catastrophic forgetting. They then propose R-CARL, a rehearsal variant that trains a final detector against the library of adversaries, report high accuracies for that detector, and analyze adversary policies via transition matrices and action histograms as an explainability device.

Significance. If the 3.61% result is robust, this is a practically relevant demonstration that a state-predictor-plus-classifier FDIA detector can be systematically evaded by a learned attack policy, and the continual/rehearsal training comparison is a useful step toward adversarial training for power-system detectors. The paper is clearly written and the framework is specified precisely in Algorithm 1 and Eq. (4); the authors also honestly acknowledge the lack of adversary diversity and the single-initial-condition simplification. The main weakness is statistical: the central numbers are single-realization estimates with no error bars, and the R-CARL accuracy is acknowledged to be inflated by training/test overlap. The explainability contribution is currently descriptive rather than validated. The result is therefore promising but not yet established at the confidence level required by the central claims.

major comments (4)
  1. [Section III-A, Table I] The central vulnerability result, A1 reducing D0 accuracy from 71.6% to 3.61% (Table I), is based on a single randomly sampled realization of a stochastic PPO adversary; the paper states this explicitly in Section III-A. No training seeds, repeated evaluation episodes, or confidence intervals are reported anywhere in Table I, so the 3.61% value could be an outlier realization rather than a reliable property of the learned policy. Because the forgetting rates in Section III-B and the R-CARL comparison in Section III-C are derived from the same table, this single-realization issue undermines the empirical support for the paper's main claims. Please report means and standard deviations over multiple independent seeds and evaluation rollouts for every entry in Table I.
  2. [Section III-C, final paragraph] The paper acknowledges that 'a large part of the testing data was seen during training' because of the adversary's lack of diversity. This directly confounds the claim that R-CARL addresses catastrophic forgetting: the near-100% accuracies of D against A1-A4 may simply reflect memorization of the training adversary rollouts. To support the rehearsal advantage, evaluate on held-out realizations of each adversary (for example, attack rollouts generated after training with different random seeds) or report a quantitative diversity measure of the adversary policies. As it stands, the R-CARL comparison is not statistically separated from an overfitting artifact.
  3. [Section III-D, Eq. (5)] The explainability contribution is not operationalized. Transition-matrix distances and droop-coefficient histograms describe the learned policies, but the paper never defines what a practitioner learns about detector deficiencies or how that information can be validated. The Abstract's claim that the approach 'pinpoint[s] the deficiencies of data-based detection' is not supported by the descriptive analysis in this section. Please specify a concrete explanation task and how the CARL knowledge pool addresses it, or revise the contribution to say that the framework provides a library of adversaries rather than an explainability method.
  4. [Conclusions and Section II-A] The Conclusions acknowledge that 'we simplified the training process by focusing on a single initial condition.' Because all training and evaluation use the same initial condition for the frequency dynamics, the demonstrated vulnerability and forgetting rates may not transfer to other operating points. Please add a sensitivity study over initial conditions, or clearly qualify the abstract and contribution claims as being for a single operating point.
minor comments (6)
  1. [Algorithm 1, line 33] The defender's reward is written as rA_t; this should be rD_t to match Eq. (3).
  2. [Figure 3] The axis labels contain garbled Unicode sequences such as '/uni00000013...'; the figure should be regenerated with readable labels.
  3. [Table I] The frequency-reward value for a given adversary is repeated across all defender rows; consider reporting these values once in a separate table or column.
  4. [Eq. (4)] The sampling probabilities P(A0)=0.8 and P(Ai)=0.2/N are fixed without sensitivity analysis; a sentence justifying these choices or showing robustness would strengthen the R-CARL section.
  5. [References] References [6] and [9] are identified only as 'arXiv preprint' and 'Under Review'; since [9] in particular is not publicly accessible, please provide a DOI or publication status where available.
  6. [Section II-B3] The asymmetry between A0, which targets the same bus for the entire episode, and the RL adversaries, which can switch buses, should be stated more prominently as a limitation of the baseline comparison.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central adversarial results are empirical co-evolution outcomes, not inputs by construction.

full rationale

The paper's central claims are empirical demonstrations rather than derived predictions. The adversary A1 is trained via PPO to maximize a reward (Eq. 2) that penalizes detection, so evading D0 is part of its objective; however, the reported 3.61% accuracy is a measured outcome of that optimization, not a value fitted to the test result or implied by the reward definition alone. The detector D0 is said to be 'trained following Ref. [9]', a same-author under-review paper, and the LSTM classifier follows Ref. [6], also same-author; these are self-citations, but they supply a benchmark architecture and training procedure rather than a conclusion, and the table reports measurements of that benchmark inside the present paper. The forgetting rates and R-CARL improvements are computed from the authors' own experiments and compared against the same table. The R-CARL evaluation does overlap with training -- the paper explicitly states 'a large part of the testing data was seen during training' -- and this is an acknowledged validity caveat about memorization versus generalization, not a hidden circular reduction of the claim to its own input. Similarly, the single-initial-condition limitation is acknowledged in the Conclusions. No equation in the paper reduces a predicted quantity to an input by construction, and no load-bearing result is imported solely through a self-citation chain. The main concerns are statistical robustness (single stochastic realization, no seeds) and external generality, which are correctness risks, not circularity.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central results rest on a handful of hand-chosen hyperparameters (p, c, d, sampling probabilities) and on several domain simplifications: a single-bus droop-attack threat model, a fixed LSTM predictor, and a single initial condition. None of these are fitted to the test table, so the circularity burden is low, but they constrain the generality of the findings.

free parameters (6)
  • reward scaling factor c = 0.1
    Hand-chosen in Sec. II-B.1 to make detection penalty commensurate with frequency reward; affects the adversary/defender training trade-off.
  • detection penalizer p = 0.1
    Hand-chosen in Sec. II-B; magnitude of reward/penalty for detection success and failure.
  • detection period d = 6
    Chosen in Sec. III; the fixed interval between adversary decisions and detector classifications.
  • R-CARL sampling probability P(A0) = 0.8
    Chosen in Eq. 4b; biases rehearsal training toward the synthetic attack A0; no sensitivity analysis.
  • RL network architecture = 2 hidden layers of 256 units
    Chosen in Sec. III for all agents; not justified against alternatives.
  • number of training episodes K
    Algorithm 1 loops to K, but K is never specified, hindering reproduction and quantitative understanding of training effort.
assumptions (4)
  • domain assumption Swing equation model with droop control accurately represents primary frequency dynamics
    Used as the environment in Sec. II-A1, following Ref. [13]; the paper does not validate this model against a more detailed simulator.
  • domain assumption Adversary can remotely access all smart inverters and alter droop coefficients, with at most one bus attacked at a time and decisions every d timesteps
    Threat model in Sec. II-A2; restricts the attack space and enables the detection definition.
  • domain assumption The LSTM state predictor remains fixed and only the classifier is retrained
    Detector design in Sec. II-A3; the authors do not adapt the predictor, so improvements are limited to the classifier boundary.
  • ad hoc to paper Single initial condition for frequency dynamics
    Acknowledged in Conclusions ('we simplified the training process by focusing on a single initial condition'); limits generalization claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Continual Adversarial Reinforcement Learning (CARL) of False Data Injection detection: forgetting and explainability." pith.science (2026). https://pith.science/paper/WNQ6SQCR

@misc{pith2026241110367,
  author       = {Pith},
  title        = {Pith review of: Continual Adversarial Reinforcement Learning (CARL) of False Data Injection detection: forgetting and explainability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WNQ6SQCR}},
  note         = {Machine review of arXiv:2411.10367}
}
read the original abstract

False data injection attacks (FDIAs) on smart inverters are a growing concern linked to increased renewable energy production. While data-based FDIA detection methods are also actively developed, we show that they remain vulnerable to impactful and stealthy adversarial examples that can be crafted using Reinforcement Learning (RL). We propose to include such adversarial examples in data-based detection training procedure via a continual adversarial RL (CARL) approach. This way, one can pinpoint the deficiencies of data-based detection, thereby offering explainability during their incremental improvement. We show that a continual learning implementation is subject to catastrophic forgetting, and additionally show that forgetting can be addressed by employing a joint training strategy on all generated FDIA scenarios.

Figures

Figures reproduced from arXiv: 2411.10367 by the authors.

Figure 1
Figure 1. (i)Schematic of the CARL process, (ii) framework of CARL process. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Left: bus under FDIA from A1 ( ) and corresponding successful detection from D0 ( ) and D1 ( ). Right: time-series of frequency deviations induced for each bus induced by A1. to 3.61% while inducing large frequency deviation rewards. This shows that even detections that use a minimal amount of supervision [6] are still vulnerable to adversarial examples that can be crafted with RL [PITH_FULL_IMAGE:figures/full_fig_… view at source ↗
Figure 3
Figure 3. Transition matrix differences (Eq. 5) between pairs of adversaries. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Histogram of droop coefficient modification for all adversaries. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 13 canonical work pages

  1. [1]

    Inertia and the power grid: A guide without the spin,

    P. Denholm, T. Mai, R. W. Kenyon, B. Kroposki, and M. O’Malley, “Inertia and the power grid: A guide without the spin,” National Renewable Energy Lab.(NREL), Golden, CO (United States), Tech. Rep., 2020

  2. [2]

    Virtual Inertia: Current Trends and Future Directions,

    U. Tamrakar, D. Shrestha, M. Maharjan, B. P. Bhattarai, T. M. Hansen, and R. Tonkoski, “Virtual Inertia: Current Trends and Future Directions,” Applied Sciences, vol. 7, no. 7, 29 pages, July 2017

  3. [3]

    Electric power grid resilience to cyber adversaries,

    T. Nguyen, S. Wang, M. Alhazmi et al., “Electric power grid resilience to cyber adversaries,” IEEE Access, vol. 8, pp. 87 592–87 608, 2020

  4. [4]

    False data injection attacks against state estimation in electric power grids,

    Y . Liu, P. Ning, and M. K. Reiter, “False data injection attacks against state estimation in electric power grids,” ACM Trans. on Info. and Syst. Security, vol. 14, no. 1, pp. 1–33, 2011

  5. [5]

    A survey on the detection algo- rithms for false data injection attacks in smart grids,

    A. S. Musleh, G. Chen, and Z. Y . Dong, “A survey on the detection algo- rithms for false data injection attacks in smart grids,” IEEE Transactions on Smart Grid , vol. 11, no. 3, pp. 2218–2234, 2019

  6. [6]

    Detection of False Data Injection Attacks (FDIA) on Power Dynamical Systems With a State Prediction Method

    A. Sahu, T. Nguyen, K. Chen, X. Zhang, and M. Hassanaly, “Detection of False Data Injection Attacks (FDIA) on Power Dynamical Systems With a State Prediction Method,” arXiv preprint arXiv:2409.04609 , 2024

  7. [7]

    Self-supervised learning of adversarial example: Towards good generalizations for deepfake detection,

    L. Chen, Y . Zhang, Y . Song, L. Liu, and J. Wang, “Self-supervised learning of adversarial example: Towards good generalizations for deepfake detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 18 710–18 719

  8. [8]

    Discovery of false data injection schemes on frequency controllers with reinforcement learning,

    R. Prasad, M. Hassanaly, X. Zhang, and A. Sahu, “Discovery of false data injection schemes on frequency controllers with reinforcement learning,” in 2024 IEEE Power & Energy Society General Meeting (PESGM). IEEE, 2024, pp. 1–5

Show all 20 references
  1. [9]

    An adversarial multi-agent reinforcement learning framework for proactive false data injection detection,

    K. Chen, T. Nguyen, and M. Hassanaly, “An adversarial multi-agent reinforcement learning framework for proactive false data injection detection,” Under Review, 2024

  2. [10]

    Explainability in deep reinforcement learning,

    A. Heuillet, F. Couthouis, and N. D ´ıaz-Rodr´ıguez, “Explainability in deep reinforcement learning,” Knowledge-Based Systems , vol. 214, p. 106685, 2021

  3. [11]

    Autonomous self- explanation of behavior for interactive reinforcement learning agents,

    Y . Fukuchi, M. Osawa, H. Yamakawa, and M. Imai, “Autonomous self- explanation of behavior for interactive reinforcement learning agents,” in Proceedings of the 5th International Conference on Human Agent Interaction, 2017, pp. 97–101

  4. [12]

    An empirical investigation of catastrophic forgetting in gradient-based neural networks,

    I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y . Bengio, “An empirical investigation of catastrophic forgetting in gradient-based neural networks,” arXiv preprint arXiv:1312.6211 , 2013

  5. [13]

    Reinforcement learning for optimal primary frequency control: A Lyapunov approach,

    W. Cui, Y . Jiang, and B. Zhang, “Reinforcement learning for optimal primary frequency control: A Lyapunov approach,” IEEE Transactions on Power Systems, vol. 38, no. 2, pp. 1676–1688, 2022

  6. [14]

    Overcoming catastrophic forgetting in neural networks,

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska et al. , “Overcoming catastrophic forgetting in neural networks,” Pro- ceedings of the national academy of sciences , vol. 114, no. 13, pp. 3521...

  7. [15]

    Learning without forgetting,

    Z. Li and D. Hoiem, “Learning without forgetting,” IEEE transactions on pattern analysis and machine intelligence , vol. 40, no. 12, pp. 2935– 2947, 2017

  8. [16]

    Progressive neural networks,

    A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell, “Progressive neural networks,” arXiv preprint arXiv:1606.04671 , 2016

  9. [17]

    Gradient episodic memory for continual learning,

    D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for continual learning,” Advances in neural information processing systems , vol. 30, 2017

  10. [18]

    A practical method for the direct analysis of transient stability,

    T. Athay, R. Podmore, and S. Virmani, “A practical method for the direct analysis of transient stability,” IEEE Transactions on Power Apparatus and Systems, vol. PAS-98, no. 2, pp. 573–584, 1979

  11. [19]

    Ray: A distributed framework for emerging AI applications,

    P. Moritz, R. Nishihara, Wang et al., “Ray: A distributed framework for emerging AI applications,” in 13th USENIX symp. on operating syst. design and impl. , 2018, pp. 561–577

  12. [20]

    Policy distillation,

    A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirk- patrick, R. Pascanu, V . Mnih, K. Kavukcuoglu, and R. Hadsell, “Policy distillation,” arXiv preprint arXiv:1511.06295 , 2015

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.