Mitigating Multi-Stage Cascading Failure by Reinforcement Learning

Chengxi Liu; Yongli Zhu

REVIEW 4 major objections 5 minor 21 references

Mitigating Multi-Stage Cascading Failure by Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A reinforcement learning policy trained on a cascade simulator keeps an IEEE 118-bus test grid stable in 77% of multi-stage attack episodes.

desk verdict A plausible proof-of-concept for RL-based cascading mitigation that undercuts itself by omitting any baseline and by using a scalar flow-limit multiplier whose physical interpretation is never validated. read the letter →

arxiv 1908.06599 v1 pith:H4ESXI3I submitted 2019-08-19 cs.LG cs.SYeess.SYmath.DSstat.ML

Yongli Zhu , Chengxi Liu This is my paper · ORCID

classification cs.LGcs.SYeess.SYmath.DSstat.ML

keywords reinforcementlearningmulti-stagecascadingfailurepowergridDCoptimalflowdeepQ-networkSARSAIEEE118-bussystemcorrectivecontrol

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a reinforcement learning agent can be trained to reduce the risk of multi-stage cascading failure in a power grid. It frames one cascade as an RL episode: an attacker breaks a line, the grid re-dispatches itself, overloaded lines may trip, and the system either settles or collapses. The agent's control is to rescale the branch flow limits used in the DC-OPF corrective re-dispatch, which changes how generation and load are adjusted at each stage. On a version of the IEEE 118-bus test system, the trained shallow and deep networks end 74.50% and 77.25% of 400 attack episodes in a stable "win" state. A sympathetic reader would take this as evidence that model-free RL can learn a useful emergency-control policy for cascades.

What carries the argument

The action is the scalar branch-flow-limit multiplier. Rather than directly switching lines or setting generator outputs, the agent alters the feasible region of the DC-OPF that mimics operator corrective control; that changes the generator re-dispatch and load shedding, and therefore the loading pattern passed to the next cascade stage. The state is a vector of branch loading percentages plus per-bus voltage magnitude, angle, active and reactive injection; for the deep variant this is zero-padded to 28x28 and fed to a CNN. Learning uses temporal-difference updates: SARSA for the one-hidden-layer network, Q-learning for the deep network.

What would settle it

Run the trained policies on a different cascade model or on recorded historical cascades, with the same attack distribution, and compare collapse rates against a no-mitigation baseline; if the RL agent's win rate is not clearly above baseline in that setting, the paper's central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that the multi-stage cascading failure problem can be addressed by model-free reinforcement learning, with the DC-OPF as a stand-in for operator corrective control. In the proposed setup, the state is the grid's branch loading plus bus voltage, angle, and injection information; the action is a single multiplier in [0.80, 1.25] applied to all branch flow limits before each DC-OPF solve; and the reward combines negative re-dispatch cost, a -1000 penalty for divergence, and a +1000 terminal reward for reaching a new steady state. After 400 training episodes, the shallow-network SARSA agent wins 74.50% of episodes and the deep Q-network agent wins 77.25%, with average rewards of 579.32 and 626.35. The paper interprets these numbers as lower cascading risk and positive learning ability.

Load-bearing premise

The load-bearing premise is that the simulator in Figure 2—random single-line attack, overload tripping, ACPF/DC-OPF re-dispatch—captures how real multi-stage cascading failures behave, and that rescaling all branch flow limits by one number is enough for a useful mitigation action.

Editorial extensions

If this is right

If correct, cascading failures can be treated as single RL episodes, so a mitigation action at an early stage can be shaped by its downstream consequences.
Encoding corrective action as an OPF parameter, rather than as direct line switching or generation setpoints, gives RL a compact action space.
Image-like state encoding lets deep Q-networks transfer from video-game-style inputs to power-system states.
The rising reward curves reported during training imply the policy improves systematically, not merely by chance, within the simulator.
The same reward and action design could be applied to other test systems where DC-OPF re-dispatch models exist.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported win rates are measured only in the paper's own simulator; transferring them to a real grid would require validation against historical cascades or a higher-fidelity transient model, which the paper does not provide.
Because the action is a single global multiplier, the policy cannot target individual bottlenecks; a natural extension would be per-branch or per-zone multipliers, which the current state and action design do not support.
The reward places a large terminal bonus on reaching steady state; an operator might need additional terms for equipment stress, blackout size, or speed, and those would likely change the learned policy.
The same RL formulation could be applied to security-constrained OPF or restoration problems wherever a corrective re-dispatch model exists.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper is a proof-of-concept for RL-based mitigation of multi-stage cascading failures. The MSCF sequential formulation is clearly laid out, and the authors map state, action, reward to the problem in a sensible way. They also report learning curves for both a shallow SARSA network and a deeper Q-learning CNN on a modified IEEE 118-bus system, and the moving-average reward trends upward. That is real work and worth acknowledging.

But the central claim—that RL reduces system collapse rates—is not supported by the reported numbers. The biggest missing piece is any baseline: there is no 'do nothing' policy, no fixed-multiplier policy, no random-action control. Without one, win rates of 74.5% and 77.25% over 400 episodes tell us the agent learned to maximize its reward, which is exactly what it was trained to do. That is circular rather than predictive.

The second soft spot is the action representation. The only control is a single scalar multiplier applied to every branch flow limit in the DC-OPF (Section III-C.2). The learned actions are never reported. If the simulator treats the inflated limits as the actual thermal limits when deciding whether lines trip, then a policy that simply sets the multiplier to 1.25 will appear to 'win' by making the problem easier. If the simulator uses the original limits, then a high multiplier tends to push flows closer to their real limits and should make things worse—yet the paper does not explain why the agent still wins. The paper even acknowledges in III-C.2 that too-high limits can lead to a tighter operating point and cascading failure, so the observed win rate is internally puzzling.

There are also the usual proof-of-concept omissions: no error bars, no train/test split, hand-tuned reward constants, and no public code or data. These are not fatal for a conference-style position paper, but they do prevent the reader from verifying the quantitative claim.

Who is this for? Researchers working on RL for power system emergency control. It is a useful entry point into the MSCF problem and a good illustration of the kinds of pitfalls you hit when the RL action space touches the simulator's constraints. I would not cite it as evidence that RL works for cascading mitigation, and I would not accept it in its current form. But I would send it to peer review—not to rubber-stamp it, but because the flaws are fixable. The right referee request would be: add a no-action baseline, report the distribution of learned multipliers, and verify that the post-OPF operating points respect the original thermal limits.

I'd bring it to a reading group if the goal is to discuss RL-on-physical-simulator issues, but it's not a citation-worthy result yet.

Referee Report

4 major / 5 minor

Summary. The paper proposes a reinforcement learning (RL) framework for mitigating multi-stage cascading failures (MSCF) in power systems. It formulates MSCF as a staged process with random initial attacks, overload-driven line tripping, and DC-OPF-based corrective re-dispatch. The RL action is a scalar multiplier applied to all branch flow limits in the DC-OPF; the state includes branch loading and bus quantities; the reward combines negative operating cost with terminal bonuses/penalties of ±1000. Experiments on a modified IEEE 118-bus test system use SARSA with a shallow network and Q-learning with a CNN, reporting win rates of 74.50% and 77.25% over 400 episodes, interpreted as reduced system collapse rates. The authors claim this is the first RL/DL application to cascading failure mitigation.

Significance. If the reported results were properly validated, the paper would provide a useful demonstration that reinforcement learning can be applied to multi-stage cascading failure mitigation with a relatively simple state/action formulation. The paper explicitly contributes a formulation of the MSCF problem, clear pseudo-code for SARSA and Q-learning, and a co-simulation architecture using DIgSILENT and MATLAB on a publicly known test system. However, the significance is currently limited by the absence of baseline comparisons, lack of train/test separation, and an under-specified relationship between the control action and the physical cascade model. The central claim of 'reduced system collapse rates' is not supported without these elements, so the contribution remains at the level of a promising proof-of-concept rather than a validated mitigation method.

major comments (4)

[Section IV-C, Table VI] The win rates in Table VI are training outcomes, not test-set evaluations: they are moving averages over 400 training episodes, with no separate evaluation episodes, no multiple random seeds, and no error bars. This does not establish that the trained policy reduces collapse rates relative to no control. The paper should report test-set win rates under the frozen learned policy, with at least 10 seeds and standard deviations, and compare against at least two baselines: no mitigation (always multiplier = 1.0) and random actions among the 10 candidates. Without such a control, the phrase 'reduced system collapse rates' in the abstract is unsupported.
[Section III-C.2 and Figure 2] The action is a single scalar multiplier applied to the branch flow limit Flmax in the DC-OPF, but the paper never states whether the line-overflow check that trips lines in Figure 2 uses the original thermal limits or the RL-modified limits. If the modified limits are used for tripping, then increasing the multiplier inflates the apparent thermal capacity and can trivially suppress overloads; if the original limits are used, a high multiplier should cause more overload trips, and the reported win rates lack a physical mechanism. The paper must clarify which limits are used in the tripping check, report the distribution of actions actually selected by the learned policies, and include a sensitivity study over fixed multipliers (e.g., 0.80, 1.00, 1.25) to rule out an artifact of the action representation.
[Section III-C.1 and Table VI] The terminal reward terms +1000 (steady state) and -1000 (collapse) are chosen by hand and by trial-and-error, and they dominate the negative generation-cost term. Because the learning objective is exactly the win/loss outcome used to compute the win rate, the reported win rate is a measure of how well the agent optimized its reward, not an independent validation of mitigation. This concern is compounded by the absence of any reward-sensitivity analysis or comparison with alternative reward weightings. The authors should show that the reported win rates are robust to reasonable changes in the reward constants, or at minimum report the average generation cost and load-shedding penalties separately from the terminal bonuses.
[Section IV-C and Figure 7] No convergence analysis is provided: Figure 7 shows only a single moving-average reward curve for the deep network, with no corresponding curve for the shallow network, no convergence criterion, and no discussion of how the 400-episode limit was chosen. It is therefore unclear whether the final win rates reflect converged policies or an arbitrary stopping point. The paper should provide learning curves for both methods, report the variance across episodes and seeds, and specify a convergence rule or verify that longer training does not materially change the results.

minor comments (5)

[Section II-B/C, Eq. (1)] Equation (1) appears garbled in the typeset version and should be rewritten; the ε-greedy formula can be stated more clearly, and the learning rate α used in the SARSA and Q-learning update rules is never defined in the text.
[Section IV, first paragraph] The testbed is called a 'modified IEEE 118-bus system' but is then described as containing 137 buses; please clarify the modification or reconcile the numbers with the standard IEEE 118-bus test case.
[References, [10]] Reference [10] lists the authors of the DQN Nature paper in incorrect order; the first author should be Mnih et al., not Kavukcuoglu et al.
[Section III-C.4 and Figure 2] The flowchart includes 'Sequentially pick all islanded grids', but the text does not explain how islanding is detected or how the DC-OPF and AC power flow are applied to islanded portions of the grid; please clarify.
[Section II-C] The paper describes neural networks as 'nonparametric models', which is not standard terminology for a fixed-structure network with tunable weights; this is a minor terminology issue but should be corrected for precision.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports a standard RL training/evaluation loop, and no claimed prediction reduces by construction to its inputs or to a load-bearing self-citation.

full rationale

The paper does not present a closed-form derivation or a fitted-parameter prediction. It defines an RL framework whose reward includes a +1000 terminal bonus for the same 'win' states that define the reported win rate, so the win rate is the training objective rather than an independent, out-of-sample prediction. However, this is the standard way RL performance is measured and is not a circular reduction: the agent is not guaranteed to maximize the reward, and the paper's claim is that training succeeded. No load-bearing self-citation appears; reference [8] by the first author is used only as a general example of CNN image embedding and is not needed to justify the MSCF formulation. The main weaknesses are the absence of a no-control baseline and the ambiguity of whether the line-overflow check in Fig. 2 uses physical or action-modified flow limits, but these are correctness and validation gaps, not circularity. The reported 'reduced system collapse rates' are therefore under-supported empirically, but they do not reduce by definition to the paper's inputs.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central result rests on the fidelity of the DC-OPF and cascade simulation, on hand-chosen reward and action bounds, and on the sufficiency of a scalar flow-limit multiplier as the only control. These are domain assumptions and ad hoc settings rather than derived facts. No new physical entities are introduced.

free parameters (6)

Reward shaping constants = -1000 for divergence, +1000 for terminal steady state
Section III.C.1 states these are determined by trial-and-error; they directly shape the policy and the reported win rate.
Action multiplier range = [0.80, 1.25]
Section IV.A bounds the action and Section IV.B uses 10 discrete values; no justification or sensitivity analysis is given.
Discount factor gamma = 0.7
Table V; fixed without sensitivity analysis.
Exploration epsilon = 1e-4
Table V; fixed, no decay schedule or sensitivity analysis.
Number of episodes = 400
Table V; no convergence criterion or train/test split is reported.
Network hyperparameters = Shallow: 10 hidden units; deep: conv channels 8/16/32 and 1x10 action layer
Section IV.A and IV.B; chosen ad hoc, only one setting is tested.

assumptions (4)

domain assumption DC-OPF (Eq. 2) faithfully represents operator corrective control via generation re-dispatch and load shedding.
Invoked in Section III.B; the reward and the action both act through this DC-OPF, so its fidelity is load-bearing.
domain assumption The simulation workflow of Fig. 2 (ACPF convergence, overloaded line tripping, DC-OPF re-dispatch, three-stage limit) models real multi-stage cascading failures.
Defines the environment in Section III.A and Fig. 2; no validation against historical cascades or higher-fidelity models is provided.
ad hoc to paper A single scalar multiplier applied to all branch flow limits is a sufficient control action for mitigation.
Section III.C.2 selects F_l^max as the action; Section IV.A bounds it to [0.80, 1.25]. The paper does not justify that this low-dimensional action can express the needed corrective strategies.
domain assumption The environment is stationary and Markov across the 400 training episodes, so Q-learning and SARSA converge to a useful policy.
Implicit in Section II for the RL algorithms and in Table V for the episode count; no convergence test or non-stationarity analysis is given.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Mitigating Multi-Stage Cascading Failure by Reinforcement Learning." pith.science (2026). https://pith.science/paper/H4ESXI3I

@misc{pith2026190806599,
  author       = {Pith},
  title        = {Pith review of: Mitigating Multi-Stage Cascading Failure by Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H4ESXI3I}},
  note         = {Machine review of arXiv:1908.06599}
}

read the original abstract

This paper proposes a cascading failure mitigation strategy based on Reinforcement Learning (RL) method. Firstly, the principles of RL are introduced. Then, the Multi-Stage Cascading Failure (MSCF) problem is presented and its challenges are investigated. The problem is then tackled by the RL based on DC-OPF (Optimal Power Flow). Designs of the key elements of the RL framework (rewards, states, etc.) are also discussed in detail. Experiments on the IEEE 118-bus system by both shallow and deep neural networks demonstrate promising results in terms of reduced system collapse rates.

Figures

Figures reproduced from arXiv: 1908.06599 by the authors.

**Figure 1.** The basic concept of Reinforcement Learning. Basically, there are two categories of RL methods: modelbased and model-free, depending on whether the model structure information is available or not. This work is supported by State Grid Corporation Project SG5455HJ180021 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 3.** The topology of IEEE 118-bus system [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

**Figure 4.** The shallow neural network structure used in RL. B. Deep Neural Network 1) Feature engineering To create an image-like input to utilize the CNN, the original input (length = 753) is extended to the length of 784 = 28×28 by appending extra zeros. One example of such encoded “images” for the input state is shown in [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Figures from the paper (4 more)

**Figure 2.** Figure 2: The overall workflow of grid simulation for MSCF study. IV. CASE STUDY In this section, a modified IEEE 118-bus system is adopted as the testbed for the proposed MSCF mitigation strategy. The maximum stage number is set to 3. It contains 137 buses and 177 lines (parall…

**Figure 5.** Figure 5: The image feature of one example input (state). 2) Network structure [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

**Figure 6.** Figure 6: The network structure for Deep Reinforcement Learning. C. Results by RL The parameter settings regarding the simulation and RL framework are shown in Table. V. The learning results are shown in Table. VI. The plot of moving average reward (window size = 50) for deep ne…

**Figure 7.** Figure 7: The moving-average reward by the DRL. TABLE V. PARAMETER SETTINGS FOR RL Shallow Network Deep Network Episodes 400 400 Max Stage 3 3 γ 0.7 0.7 ε 1e-4 1e-4 TABLE VI. LEARNING RESULTS Shallow Network Deep Network Win rate (moving_avg.) 74.50% 77.25% Total Win times 298 3…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 21 canonical work pages

[1]

Propose and formulate the concept of the Multi-Stage Cascading Failure (MSCF) problem

work page
[2]

environment

Present a systematic reinforcement learning framework to fulfill the mitigation control for the MSCF problem. The remaining parts of this paper are organized as follows. Section II introduces the principles of deep reinforcement learning, including two basic types of learning strategies. Section III investigates and proposes an RL-based control framework ...

work page
[3]

firstly a “value” Q(s,a) is constructed concretely (e.g

Value-based RL The value-based RL is indeed an indirect control approach, i.e. firstly a “value” Q(s,a) is constructed concretely (e.g. a 2D look-up table of state-action pairs for discrete cases; or approximated by abstract means, e.g. by high-order nonlinear functions (parametric model), or by neural networks (nonparametric model). In this paper, the ne...

work page
[4]

discount factor

Policy-based RL The idea of policy-based RL is to model the policy (i.e. the probabilistic distribution of action) directly by neural network and train such model by RL. It can handle problems with continuous action space. This type of RL method is out of the scope of this paper. More details about this method can be found in [5]. B. On Policy Temporal Di...

work page
[5]

Optimal and autonomous control using reinforcement learning: a survey,

B. Kiumarsi, K. G. Vamvoudakis, H. Modares and F. L. Lewis, “Optimal and autonomous control using reinforcement learning: a survey,” IEEE Transactions on Neural Networks and Learning Systems, vol. 29, no. 6, pp. 2042-2062, Jun. 2018

work page 2018
[6]

Reward design (of each Stage) - −Total generation cost (i.e. minus the objective value of DC-OPF) (if converge); - −1000 (minus one thousand if DC-OPF or AC-PF diverge); +1000 (plus one thousand if system finally reaches a new steady state at the last stage. These values are determined by trial-and-error test

work page
[7]

branch flow limit

Action design In the OPF, if the line flow limit is too low, the OPF might not converge due to the narrow feasible region. On the contrary, if the line flow limit is too high, the feasible region also becomes large. However, the obtained optimal solution might lead to an operation point with tighter power flow status on each branch, which may result in ca...

work page
[8]

lines and transformers) are chosen and packed as the state in the RL learning framework, i.e

State design: Several quantities of each bus and the power flow of each branch (i.e. lines and transformers) are chosen and packed as the state in the RL learning framework, i.e. state=[branch_loading_status, V 1, θ1, P1, Q1,…, Vn, θn, Pn, Qn], where, branch_loading_status are the percentage values calculated by dividing each branch flow by its loading li...

work page

Show all 21 references

[9]

Generation

Environment In this study, the learning environment in the RL framework is just the power grid itself. Thus, a co-simulation platform based on the combination of DIgSILENT and MATLAB is implemented, where the commercial power system tool DIgSILENT is mainly used as the simulat...

1934
[10]

In this paper, a DRL approach, similar to the original DQN but with a simplified training procedure is adopted in the mitigation of cascading failures

use screenshots of a video game as the training input and the trained networks outperform human-beings in playing that game, and thereafter the Deep Reinforcement Learning (DRL) concept is established. In this paper, a DRL approach, similar to the original DQN but with a simpl...
[11]

One example of such encoded “images” for the input state is shown in Fig

Feature engineering To create an image-like input to utilize the CNN, the original input (length = 753) is extended to the length of 784 = 28×28 by appending extra zeros. One example of such encoded “images” for the input state is shown in Fig. 5. 5 1 01 52 02 5 5 10 15 20 25 ...
[12]

ε-greedy

Network structure Typically, deeper network and more layers might lead to overfitting in practice. Thus, the network structure used in this paper is shown in Fig. 6. The Q-learning (Off-policy TD) method is applied to it. The output of the 2nd-last layer (dimension 1×10) will ...
[13]

Modeling the dynamics of cascading failures in power systems,

X. Zhang, C. Zhan and C. K. Tse, “Modeling the dynamics of cascading failures in power systems,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 7, no. 2, pp. 192-204, Jun. 2017

2017
[14]

Modeling cascading failures in the North American power grid,

R. Kinney, P. Crucitti, R. Albert, et al., “Modeling cascading failures in the North American power grid,” Eur. Phys. J. B , vol. 46, no.1, pp. 101-107, 2005

2005
[15]

Lu and P

N. Lu and P. Du, Energy Storage for Smart Grids , 1st ed. Cambridge, MA: Academic Press, 2015, p. 115-116

2015
[16]

Modeling cascading failures and mitigation strategies in PMU based cyber-physical power systems,

Y. Han, C. Guo, S. Ma, et al. , “Modeling cascading failures and mitigation strategies in PMU based cyber-physical power systems,” J. Mod. Power Syst. Clean Energy, vol.6, no. 5, pp. 944-957, 2018

2018
[17]

Reinforcement learning for reactive power control,

J. G. Vlachogiannis and N. D. Hatziargyriou, “Reinforcement learning for reactive power control,” IEEE Transactions on Power Systems, vol. 19, no. 3, pp. 1317-1325, Aug. 2004

2004
[18]

Distributed reinforcement learning to coordinate current sharing and voltage restoration for islanded DC microgrid,

Z. Liu, Y. Luo, R. Zhou, et al., “Distributed reinforcement learning to coordinate current sharing and voltage restoration for islanded DC microgrid,” J. Mod. Power Syst. Clean Energy, vol.6, no.2, pp.364-374, Sep. 2017

2017
[19]

Image embedding of PMU data for deep learning towards transient disturbance classification,

Y. Zhu, C. Liu and K. Sun, “Image embedding of PMU data for deep learning towards transient disturbance classification,” in Proc. of IEEE International Conference on Energy Internet (ICEI), Beijing, 2018, pp. 169-174

2018
[20]

Multi-Step Short-Term Power Consumption Forecasting with a Hybrid Deep Learning Strategy,

K. Yan, X. Wang, Y. Du, et al. "Multi-Step Short-Term Power Consumption Forecasting with a Hybrid Deep Learning Strategy," Energies, vol.11, no.11, 2018

2018
[21]

Human-level control through deep reinforcement learning,

K. Kavukcuoglu, D. Silver, A. A. Rusu, et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, pp. 529-533, Feb. 2015

2015

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Propose and formulate the concept of the Multi-Stage Cascading Failure (MSCF) problem

work page

[2] [2]

environment

Present a systematic reinforcement learning framework to fulfill the mitigation control for the MSCF problem. The remaining parts of this paper are organized as follows. Section II introduces the principles of deep reinforcement learning, including two basic types of learning strategies. Section III investigates and proposes an RL-based control framework ...

work page

[3] [3]

firstly a “value” Q(s,a) is constructed concretely (e.g

Value-based RL The value-based RL is indeed an indirect control approach, i.e. firstly a “value” Q(s,a) is constructed concretely (e.g. a 2D look-up table of state-action pairs for discrete cases; or approximated by abstract means, e.g. by high-order nonlinear functions (parametric model), or by neural networks (nonparametric model). In this paper, the ne...

work page

[4] [4]

discount factor

Policy-based RL The idea of policy-based RL is to model the policy (i.e. the probabilistic distribution of action) directly by neural network and train such model by RL. It can handle problems with continuous action space. This type of RL method is out of the scope of this paper. More details about this method can be found in [5]. B. On Policy Temporal Di...

work page

[5] [5]

Optimal and autonomous control using reinforcement learning: a survey,

B. Kiumarsi, K. G. Vamvoudakis, H. Modares and F. L. Lewis, “Optimal and autonomous control using reinforcement learning: a survey,” IEEE Transactions on Neural Networks and Learning Systems, vol. 29, no. 6, pp. 2042-2062, Jun. 2018

work page 2018

[6] [6]

Reward design (of each Stage) - −Total generation cost (i.e. minus the objective value of DC-OPF) (if converge); - −1000 (minus one thousand if DC-OPF or AC-PF diverge); +1000 (plus one thousand if system finally reaches a new steady state at the last stage. These values are determined by trial-and-error test

work page

[7] [7]

branch flow limit

Action design In the OPF, if the line flow limit is too low, the OPF might not converge due to the narrow feasible region. On the contrary, if the line flow limit is too high, the feasible region also becomes large. However, the obtained optimal solution might lead to an operation point with tighter power flow status on each branch, which may result in ca...

work page

[8] [8]

lines and transformers) are chosen and packed as the state in the RL learning framework, i.e

State design: Several quantities of each bus and the power flow of each branch (i.e. lines and transformers) are chosen and packed as the state in the RL learning framework, i.e. state=[branch_loading_status, V 1, θ1, P1, Q1,…, Vn, θn, Pn, Qn], where, branch_loading_status are the percentage values calculated by dividing each branch flow by its loading li...

work page