Heuristic Dynamic Programming for Adaptive Virtual Synchronous Generators

Donald C. Wunsch; Mehdi Ferdowsi; Mohammad Saleh Sanjarinia; Pourya Shamsi; Sepehr Saadatmand

REVIEW 5 major objections 8 minor 14 references

Heuristic Dynamic Programming for Adaptive Virtual Synchronous Generators

T0 review · 5 major / 8 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a two-network heuristic dynamic programming controller, trained online, outperforms conventional PI voltage and reactive-power controllers for virtual synchronous generators, especially in resistive grids, and can…

desk verdict A credible VSG/HDP combination with a sound resistive-grid motivation, but the action-network update as written is wrong and the simulation evidence is too thin for the 'any impedance angle' claim. read the letter →

arxiv 1908.05744 v1 pith:KGOYWHFQ submitted 2019-08-14 cs.LG cs.NEcs.SYeess.SYstat.ML

Sepehr Saadatmand , Mohammad Saleh Sanjarinia , Pourya Shamsi , Mehdi Ferdowsi , Donald C. Wunsch This is my paper · ORCID

classification cs.LGcs.NEcs.SYeess.SYstat.ML

keywords grid-connectedinverterheuristicdynamicprogrammingneuralnetworkvirtualsynchronousgeneratorinertiacontroladaptivereactivepowerresistivegrid

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard virtual synchronous generator (VSG) controller, which assumes the grid impedance is mostly inductive, breaks down in low-voltage resistive grids, and that a neural-network heuristic dynamic programming (HDP) controller can restore good tracking of active and reactive power there. The proposed controller replaces the conventional voltage and reactive-power PI loop with two small feedforward networks — an action network that sets the inverter voltage magnitude and a critic network that learns a cost-to-go value — trained together online. In simulation comparisons on inductive and resistive grids, the HDP controller reaches steady state faster than the PI controller in both cases and is reported to track power references better in the resistive case. If the claim holds, adaptive critic control would give inverter-based energy resources the frequency and voltage support of synchronous machines without retuning the controller for each unknown line impedance.

What carries the argument

The load-bearing mechanism is the heuristic dynamic programming pair: an action network and a critic network. The critic network learns the cost-to-go $J(k)$ from the Bellman equation, with a discount factor $\gamma$, by minimizing $[J(k) - \gamma J(k+1) - U(k)]^2$ in gradient descent; the action network is trained through backpropagation to choose the inverter peak voltage $E(k)$ that minimizes the future cost. The utility function is $U(k) = \sqrt{K_P e_P^2 + K_Q e_Q^2 + K_f e_f^2}$, built from active-power, reactive-power, and frequency errors. Because the action network takes both active and reactive power errors as inputs, it can capture the cross-coupling that appears when the line resistance is no longer negligible, which is the specific failure that motivates the paper.

What would settle it

Apply the same trained HDP networks to a grid with a 45-degree impedance angle (equal resistance and reactance) and to a step change in line resistance during operation; if settling time or reactive-power tracking is no better than the PI controller, or if the online weight updates fail to recover, the 'adapt to any impedance angle' claim is refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a heuristic dynamic programming controller — two fully connected neural networks with two hidden layers of eight nodes each — can replace the integral/PI voltage-reactive power controller in a virtual synchronous generator and perform better than the conventional design under non-inductive line conditions. The paper derives power-flow equations showing that in inductive grids active power is approximately proportional to the power angle and reactive power to the voltage magnitude, so the conventional decoupled PI design works; in resistive grids that decoupling disappears, and the reactive power depends on both angle and voltage. The HDP action network receives active power, reactive power, frequency errors, and the inverter angle and outputs the voltage magnitude, while the critic network approximates the discounted cost-to-go $J(k)$ and is trained to satisfy the Bellman equation. Simulation results for the two fixed line configurations show the HDP reaching steady state faster in the inductive grid and tracking better in the resistive grid than the conventional PI controller, supporting the paper's conclusion that a well-trained HDP performs better than conventional PIs particularly in resistive grids.

Load-bearing premise

The controller's promised ability to adapt to any grid impedance rests on networks trained only in the same simulation at two fixed line configurations (inductive and resistive); if the training simply memorizes those two cases, the advantage over the PI controller would not carry over to other impedance angles.

Editorial extensions

If this is right

For resistive-grid microgrids, the HDP controller would keep reactive power and voltage tracking without relying on the inductive-line approximation that invalidates conventional PI tuning.
The joint training of action and critic networks in the same cycle could shorten the adaptation time when the operating point changes.
Using both active- and reactive-power errors as network inputs removes the single-input single-output restriction that limits conventional PI voltage control in non-inductive grids.
The same control structure could be retrained rather than redesigned when line parameters change, because no analytic model of the impedance is needed for the neural network.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper trains and evaluates at only two impedance angles (inductive and resistive); the natural next test is intermediate angles such as R/X = 1, where the coupling is strongest, to see whether the trained networks interpolate or need retraining.
Since the frequency error is present in the utility and input vectors but its coefficient $K_f$ is set to zero in the simulations, the same architecture could be exercised for simultaneous voltage and frequency control by activating that term.
A hardware-in-the-loop or field test with a step change in line impedance would separate genuine online adaptation from memorization of the two simulated operating points.
The random-initial-state training procedure suggests the controller may be robust to operating-point changes, but comparing it against a PI controller with adaptive gains under the same disturbances would give a stricter baseline.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper is a legitimate extension — HDP applied to VSG voltage-magnitude control — and its motivation from P/Q coupling in resistive grids is sound. But the algorithm section as written has a real problem, and the empirical case is too thin to support the abstract's claims.

What's genuinely there: the power-flow equations in Section II correctly show that in low-voltage/resistive grids, the conventional assumption that Q tracks voltage magnitude and P tracks angle breaks down. That is a real issue for VSG control. The idea of replacing the reactive-power PI with an HDP action-critic that takes both P and Q errors as inputs is sensible, and it is not in the cited HDP papers. The qualitative figures do show faster settling for the HDP in the resistive case.

Soft spots, in decreasing order of severity:

1. The action-network update in Section III.B is not a gradient of J. The paper defines ζ = Σ ∂J(k+1)/∂E(k) and then updates W_action by −α ζ ∂ζ/∂W_action, which minimizes ζ², not the cost-to-go J. If that equation is literal, the controller is not doing HDP. This needs a rewrite or a clarification.

2. The comparison is entirely qualitative. No numerical metrics, no settling-time or overshoot values, no multiple runs. Training and evaluation use the same simulation model, and only two R/X cases are shown, so "can adapt to any impedance angle" is an overclaim. There is no held-out impedance angle or disturbance scenario.

3. The PI baseline is underspecified — no gains, no tuning method — so "better than PI" cannot be audited.

4. Minor but telling: the intro has a literal placeholder "[some references that have used other controllers]". That should have been resolved before submission.

5. No code or data. Training details (learning rates, network initialization, number of episodes) are almost absent; the four-step procedure in Section IV could not be reproduced from the paper alone.

Who is this for: people working on ADP/RL for power-electronic control, and VSG control in low-voltage microgrids. The idea is worth knowing, but the evidence is not.

Recommendation: if I were an editor, I'd send it to review rather than desk reject, because the combination is new and the resistive-grid motivation is correct — but I'd expect heavy revision. The authors should fix the update rule, add metrics, report a range of R/X ratios, and tone down the generalization claim.

Referee Report

5 major / 8 minor

Summary. The paper proposes a heuristic dynamic programming (HDP) controller, with critic and action neural networks, to replace the conventional integrator/droop controllers in a virtual synchronous generator (VSG) for grid-connected inverters. The authors argue that conventional VSG control, which relies on the inductive-grid assumption that active power depends on angle and reactive power on voltage magnitude, fails in resistive or semi-resistive grids. They derive the VSG power-flow equations, describe the HDP training procedure over a one-second horizon with random initial states and power setpoints, and compare the HDP controller with a conventional PI-based controller in two simulated cases: an inductive grid and a resistive grid. The paper concludes that the HDP controller performs better than the conventional controller, especially in the resistive case, and claims that the approach can adapt to any impedance angle.

Significance. If the central claim were established, the paper would offer an adaptive, model-free alternative to fixed-parameter VSG controllers in low-voltage resistive grids, where the standard P/Q decoupling assumption fails. The paper correctly identifies a real limitation of conventional VSG droop control and makes a reasonable conceptual case for using an adaptive critic design. The simulation setup covers two impedance regimes, and the use of standard VSG and HDP formulations gives the work a sound starting point. However, the evidence for the claimed superiority is currently qualitative only, the comparison baseline is not fully specified, the action-network update appears to be mis-derived, and the 'any impedance angle' claim is not tested. The significance is therefore conditional on substantial additional validation and correction.

major comments (5)

[Section IV, Figures 7-8] The central claim that the HDP controller 'performs better' than the conventional controller is supported only by qualitative time-domain plots. No numerical metrics such as overshoot, settling time, steady-state error, or integrated squared error are reported, and there are no multiple runs, error bars, or statistical tests. The paper should provide quantitative comparison tables with clearly defined metrics and a statement of how many random initializations/experiments were used.
[Section IV, training steps 1-4 and Table I] The training and evaluation are performed in the same simulation model at only two fixed line configurations (inductive and resistive, Table I). There is no held-out impedance angle, no setpoint-change scenario, no disturbance test, and no robustness measure. The Abstract's claim that the controller 'can adapt to any impedance angle' is therefore unsupported by the presented evidence; at most, the two cases are consistent with that conjecture. The paper should validate on multiple R/X ratios and on conditions not seen during training.
[Section III.B, paragraphs on action network update] The action-network weight update is written as ΔW_action = −α_a ζ ∂ζ/∂W_action with ζ = ∑_k ∂J(k+1)/∂E(k). This is not the gradient of the cost-to-go J with respect to the action weights; the correct backpropagation expression requires ∂J/∂W_action = (∂J/∂E)(∂E/∂W_action). As written, the update does not minimize the cost-to-go, so the optimality claim for the action network is not justified. Please correct the derivation or clarify the exact update rule implemented and how ∂J/∂E is computed.
[Section III.A and Table I] The text states that the discount factor satisfies 0 < γ < 1, but Table I lists γ = 1. With an infinite-horizon cost and γ = 1, the cost-to-go J can diverge unless the system is asymptotically stable and the utility decays to zero; the paper does not address this. Additionally, the utility weights in Table I are given as [kP kQ kf] = [1 1 0], so the frequency error term is not actually used in the utility function despite being included in the definition. These inconsistencies should be resolved and the choices justified.
[Section IV and Section II.A] The PI baseline against which the HDP controller is compared is under-specified. No PI gains, tuning procedure, or exact controller structure are given, and the conventional controller described in Section II.A is an integrator with voltage droop rather than a standard PI. Without a precise definition of the baseline controller and its parameters, the claim that the HDP controller 'performs better than the conventional PIs' cannot be audited or reproduced.

minor comments (8)

[Abstract] The abstract contains a typo: 'an/or' should be 'and/or'.
[Section I, Introduction] The text includes the placeholder '[some references that have used other controllers]', which must be replaced with actual citations.
[Section III, paragraph 2] The term 'ADCs' appears where 'ACDs' (adaptive critic designs) is intended; please fix the abbreviation.
[Section III.A, cost-to-go equation] The Bellman equation is written with summation index k over the same symbol as the time index, which is confusing; the right-hand side should use a dummy summation index and shifted time arguments, e.g., J(k) = ∑_{i=0}^∞ γ^i U(k+i+1).
[Section IV.B heading] The resistive-grid subsection is labeled 'A. Resistive grid' but should be 'B. Resistive grid'.
[Section III.A, utility function] The phrase 'These coefficients can also be defined as the weight matrix in a normalized function' is unclear, as no normalization procedure is described.
[Table I] The row 'ℽ 1' should be labeled as the discount factor γ, and the learning-rate row '[αc αc] [1 1]' appears to conflate the critic and action learning rates; the notation should be clarified.
[References] Reference [14] is missing publication year and publisher details; several references also lack page numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the HDP controller is an empirical application of a known adaptive-dynamic-programming method, and its comparison with the PI controller is an independent simulation result, not a construction-level reduction.

full rationale

The paper defines the HDP cost-to-go and utility function in Section III, then trains the critic and action networks using the associated update rules, and evaluates the resulting controller against a conventional PI controller in inductive and resistive grids. The reported active and reactive power responses are physical simulation outputs, not re-statements of the training objective, and the PI baseline is a separate controller rather than a disguised version of the HDP loss. No parameter is fitted to the evaluation data and then renamed as a prediction; no uniqueness theorem or prior result by the authors is invoked to forbid alternatives; and the cited works by co-author Wunsch are background references for adaptive critic designs, not load-bearing support for the central claim. The paper's main weakness is empirical: the claim that the controller can 'adapt to any impedance angle' is supported only by two simulated line configurations, with training and evaluation sharing the same plant model and no held-out impedance angle or quantitative metric. This is a generalization and validation limitation, not circularity under the criteria requiring a quoted reduction of the result to its inputs by construction.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

All the physical equations are standard. The main cost is in the boldest claim, 'adapt to any impedance angle', which requires an additional generalization assumption that the paper does not test. The hyperparameters of the HDP controller are chosen by hand and are free parameters. No invented entities are introduced.

free parameters (6)

Utility weights K_P, K_Q, K_f = [1, 1, 0]
Chosen weights in utility function U(k) = sqrt(K_P e_P^2 + K_Q e_Q^2 + K_f e_f^2), Section III.A; no tuning procedure given, and K_f=0 means frequency error is ignored.
Learning rates alpha_c, alpha_a = [1, 1] (as printed in Table I)
Update rates for critic and action networks; values appear implausibly high and may be a typo, but no other values are given.
Discount factor gamma = 1 (Table I, printed as 'ℽ 1')
Discount factor in Bellman cost; gamma=1 gives an unbounded infinite-horizon sum unless a finite horizon is imposed, which is not fully specified.
Training horizon = 1 s (1000 ms)
Control and optimization horizon used in training, stated in Section III.B; chosen without justification.
Network architecture = two hidden layers, 8 nodes each
Architecture for critic and action networks, Section III.A/B; no sensitivity analysis or selection rationale is provided.
Sampling time = 1 ms
Time step for simulation and training in Table I; no stability or discretization analysis is given.

assumptions (6)

domain assumption Virtual synchronous generator dynamics follow the swing equation P_in - P_out = J omega_i domega_i/dt + D Delta_omega (Eq. 1)
This models the inverter as a virtual rotor; used throughout Section II.A. It is a standard model but a modeling choice for how the inverter emulates inertia.
domain assumption Grid can be represented as a stiff voltage source behind an equivalent R-L impedance per phase (Figure 3, Section II.B)
Power flow equations (2)-(3) rely on this lumped circuit; real grids have shunt elements, harmonics, and dynamics.
standard math For inductive grids, X_eq >> R_eq and small-angle approximations sin(delta) approximately delta and cos(delta) approximately 1 hold (Eqs. 4-5)
Used to justify conventional VSG controller; not valid for resistive grids, which motivates the paper.
domain assumption Bellman optimality and gradient-descent training converge to a useful cost-to-go J (Section III.A-B)
No convergence proof or stability guarantee is given; HDP training is assumed to find good weights within one-second horizons.
domain assumption Inverter is connected to an ideal DC source (Section II.A)
The authors explicitly state this is not applicable to solar PV inverters, so the results do not cover one of the main renewable applications.
ad hoc to paper Random training over a one-second horizon in the same simulation environment generalizes to unseen conditions and to any impedance angle (Section IV)
The central claim of adaptivity relies on generalization that is not demonstrated outside the training and evaluation environment.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Heuristic Dynamic Programming for Adaptive Virtual Synchronous Generators." pith.science (2026). https://pith.science/paper/KGOYWHFQ

@misc{pith2026190805744,
  author       = {Pith},
  title        = {Pith review of: Heuristic Dynamic Programming for Adaptive Virtual Synchronous Generators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KGOYWHFQ}},
  note         = {Machine review of arXiv:1908.05744}
}

read the original abstract

In this paper a neural network heuristic dynamic programing (HDP) is used for optimal control of the virtual inertia based control of grid connected three phase inverters. It is shown that the conventional virtual inertia controllers are not suited for non inductive grids. A neural network based controller is proposed to adapt to any impedance angle. Applying an adaptive dynamic programming controller instead of a supervised controlled method enables the system to adjust itself to different conditions. The proposed HDP consists of two subnetworks, critic network and action network. These networks can be trained during the same training cycle to decrease the training time. The simulation results confirm that the proposed neural network HDP controller performs better than the traditional direct fed voltage and reactive power controllers in virtual inertia control schemes.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 8 canonical work pages

[1]

Overview of Control and Grid Synchronization for Distributed Power Generation Systems,

F. Blaabjerg, R. Teodorescu, M. Liserre, and A. Timbus, “Overview of Control and Grid Synchronization for Distributed Power Generation Systems,” IEEE Transactions on Industrial Electronics, vol. 53, no. 5, pp. 1398–1409, 2006

2006
[2]

Impact of Distributed Generations With Energy Storage Devices on the Electric Grid,

A. K. Srivastava, A. A. Kumar, and N. N. Schulz, “Impact of Distributed Generations With Energy Storage Devices on the Electric Grid,” IEEE Systems Journal, vol. 6, no. 1, pp. 110–117, 2012

2012
[3]

Virtual synchronous machine,

H.-P. Beck and R. Hesse, “Virtual synchronous machine,” in Proc. 9th Int. Conf. Elect. Power Qual. Utilisation, 2007, pp. 1–6

2007
[4]

Equivalence of virtual synchronous machines and frequency -droops for converter -based microgrids,

S. D’Arco and J. A. Suul, “Equivalence of virtual synchronous machines and frequency -droops for converter -based microgrids,” IEEE Trans. Smart Grid, vol. 5, no. 1, pp. 394–395, Jan. 2014

2014
[5]

Research on the impact of DFIG virtual inertia control on power system small -signal stability considering the phase -locked loop

Ma, Jing, Yang Qiu, Yinan Li, Weibo Zhang, Zhanxiang Song, and James S. Thorp. "Research on the impact of DFIG virtual inertia control on power system small -signal stability considering the phase -locked loop." IEEE Transactions on Power Systems 32, no. 3 (2017)

work page 2017
[6]

Estimating the Rating of the Reviews Based on the Text,

M. Kavousi and S. Saadatmand, “Estimating the Rating of the Reviews Based on the Text,” Data Analyt ics and Learning Lecture Notes in Networks and Systems, pp. 257–267, 2018

work page 2018
[7]

E., Optimal Control Theory: An Introduction, Prentice –Hall, Englewood Cliffs, NJ, 1970, Chaps

Kirk, D. E., Optimal Control Theory: An Introduction, Prentice –Hall, Englewood Cliffs, NJ, 1970, Chaps. 1–3

work page 1970
[8]

Adaptive Critic Designs,

D. V. Prokorov and D. C. Wunsch, “Adaptive Critic Designs,” IEEE Trans. on Neural Networks, vol. 8, pp. 997–1007, Sep. 1997

work page 1997

Show all 14 references

[9]

Heuristic dynamic programming for neural network vector control of a grid-connected converter

Fu, Xingang, and Shuihui Li. “ Heuristic dynamic programming for neural network vector control of a grid-connected converter.” 2014 IEEE PES General Meeting Conference & Exposition. IEEE, 2014

2014
[10]

Direct heuristic dynamic programming method for power system stability enhancement,

M. Yu, C. Lu, and Y. Liu, “Direct heuristic dynamic programming method for power system stability enhancement,” 2014 American Control Conference, 2014

2014
[11]

Nested -loop neural network vector control of permanent magnet synchronous motors,

S. Li, M. Fairbank, X. Fu, D. C. Wunsch, and E. Alonso, “Nested -loop neural network vector control of permanent magnet synchronous motors,” The 2013 International Joint Conference on Neural Networks (IJCNN), 2013

2013
[12]

Novel comprehensive control framework for incorporating VSCs to smart power grids using bidirectional synchronous-VSC,

M. Ashabani and Y. A. R. I. Mohamed, “Novel comprehensive control framework for incorporating VSCs to smart power grids using bidirectional synchronous-VSC,” IEEE Trans. Power Syst., vol. 29, no. 2, pp. 943–957, Mar. 2014

2014
[13]

Wind farms with HVDC delivery in inertial response and primary frequency control,

M. Zhixin, F. Lingling, D. Osborn, and S. Yuvarajan, “Wind farms with HVDC delivery in inertial response and primary frequency control,” IEEE Trans. Energy Convers., vol. 25, no. 4, pp. 1171–1178, Dec. 2010

2010
[14]

Approximate dynamic programming for real -time control and neural modeling,

P. Werbos, “Approximate dynamic programming for real -time control and neural modeling,” in Handbook of Intelligent Control, White and Sofge, Eds. New York: Van Nostrand Reinhold, pp. 493–525. (a) (b) (c) (d) Figure. 8. HDP controller performance for VSG connected to the resis...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Overview of Control and Grid Synchronization for Distributed Power Generation Systems,

F. Blaabjerg, R. Teodorescu, M. Liserre, and A. Timbus, “Overview of Control and Grid Synchronization for Distributed Power Generation Systems,” IEEE Transactions on Industrial Electronics, vol. 53, no. 5, pp. 1398–1409, 2006

2006

[2] [2]

Impact of Distributed Generations With Energy Storage Devices on the Electric Grid,

A. K. Srivastava, A. A. Kumar, and N. N. Schulz, “Impact of Distributed Generations With Energy Storage Devices on the Electric Grid,” IEEE Systems Journal, vol. 6, no. 1, pp. 110–117, 2012

2012

[3] [3]

Virtual synchronous machine,

H.-P. Beck and R. Hesse, “Virtual synchronous machine,” in Proc. 9th Int. Conf. Elect. Power Qual. Utilisation, 2007, pp. 1–6

2007

[4] [4]

Equivalence of virtual synchronous machines and frequency -droops for converter -based microgrids,

S. D’Arco and J. A. Suul, “Equivalence of virtual synchronous machines and frequency -droops for converter -based microgrids,” IEEE Trans. Smart Grid, vol. 5, no. 1, pp. 394–395, Jan. 2014

2014

[5] [5]

Research on the impact of DFIG virtual inertia control on power system small -signal stability considering the phase -locked loop

Ma, Jing, Yang Qiu, Yinan Li, Weibo Zhang, Zhanxiang Song, and James S. Thorp. "Research on the impact of DFIG virtual inertia control on power system small -signal stability considering the phase -locked loop." IEEE Transactions on Power Systems 32, no. 3 (2017)

work page 2017

[6] [6]

Estimating the Rating of the Reviews Based on the Text,

M. Kavousi and S. Saadatmand, “Estimating the Rating of the Reviews Based on the Text,” Data Analyt ics and Learning Lecture Notes in Networks and Systems, pp. 257–267, 2018

work page 2018

[7] [7]

E., Optimal Control Theory: An Introduction, Prentice –Hall, Englewood Cliffs, NJ, 1970, Chaps

Kirk, D. E., Optimal Control Theory: An Introduction, Prentice –Hall, Englewood Cliffs, NJ, 1970, Chaps. 1–3

work page 1970

[8] [8]

Adaptive Critic Designs,

D. V. Prokorov and D. C. Wunsch, “Adaptive Critic Designs,” IEEE Trans. on Neural Networks, vol. 8, pp. 997–1007, Sep. 1997

work page 1997