{"id":"10777759-4971-4b60-b889-a112c106e635","arxiv_id":"1908.05744","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A neural heuristic dynamic programming controller for virtual synchronous generators is shown in simulation to outperform PI control, especially in resistive grids.","lead":"This paper replaces the conventional PI voltage controller in a virtual synchronous generator with a neural heuristic dynamic programming (HDP) controller for grid-connected inverters. Simulations show the HDP controller settles faster and handles resistive grids better, pointing toward inverters that adapt to changing grid impedance without manual retuning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'any impedance angle' and 'better than PI' claims are untested: training and evaluation share the same simulation model, only two R/X ratios are shown, and the PI benchmark and update rule are not specified.","rationale":"The reader's weakest assumption identifies the same load-bearing gap: the HDP networks are only demonstrated in the training environment, and no generalization test supports the abstract's 'any impedance angle' claim. My pass adds two concrete aggravating details: the comparison PI is not parameterized, and the written action-network update rule is formally inconsistent with minimizing J (it uses ∂ζ/∂W instead of ∂E/∂W). The VSG power-flow modeling in Section II is standard, and the HDP/critic structure is a recognizable ADP formulation, so I do not see an internal contradiction in the model itself. The issue is evidence quality: without numerical metrics, a held-out impedance sweep, explicit PI tuning, and a corrected update rule, the central empirical claim is conditional at best. I therefore keep the reader's CONDITIONAL verdict unchanged and recommend the authors supply the missing validation and implementation details. I also note the introduction contains an explicit citation placeholder ('[some references that have used other controllers]'), which weakens the motivational argument but does not bear on the technical claim.","tokens_in":7138,"tokens_out":5361,"duration_ms":57982,"concrete_test":"Reproduce the exact protocol with a strict train/validation split over the impedance angle: train the HDP networks on a subset of R/X ratios (e.g., 0.02, 0.1, 1, 10) and evaluate on held-out ratios (e.g., 0.05, 0.5, 5), applying a step change in Pref/Qref and a local load disturbance. Report integrated squared tracking error and settling time for the HDP controller and for a PI controller tuned by a documented rule at each R/X value. If HDP does not outperform PI on held-out angles, the any-impedance and superiority claims fail.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is an empirical superiority/adaptivity claim, but Section IV provides no out-of-sample evidence. The HDP networks are trained in the same closed-loop plant and evaluated in the same simulation at only two line configurations (Table I: inductive and resistive). There is no held-out impedance angle, no setpoint or disturbance scenario during evaluation, and no numerical metric—only qualitative curves in Figures 7 and 8. The PI benchmark is also underspecified, so 'performs better' cannot be audited. In addition, the action-network update in Section III.B is written as ∆W_action = −α_a ζ ∂ζ/∂W_action with ζ = ∂J(k+1)/∂E(k); this is not the gradient of J with respect to W_action, which would require ∂J/∂W_action = (∂J/∂E)(∂E/∂W_action). If implemented literally, the action network is not minimizing the cost-to-go J. Together these issues mean the central claim rests on an unverified generalization mechanism and an under-specified comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a heuristic dynamic programming (HDP) controller, with critic and action neural networks, to replace the conventional integrator/droop controllers in a virtual synchronous generator (VSG) for grid-connected inverters. The authors argue that conventional VSG control, which relies on the inductive-grid assumption that active power depends on angle and reactive power on voltage magnitude, fails in resistive or semi-resistive grids. They derive the VSG power-flow equations, describe the HDP training procedure over a one-second horizon with random initial states and power setpoints, and compare the HDP controller with a conventional PI-based controller in two simulated cases: an inductive grid and a resistive grid. The paper concludes that the HDP controller performs better than the conventional controller, especially in the resistive case, and claims that the approach can adapt to any impedance angle.","tokens_in":7351,"tokens_out":3378,"duration_ms":35098,"significance":"If the central claim were established, the paper would offer an adaptive, model-free alternative to fixed-parameter VSG controllers in low-voltage resistive grids, where the standard P/Q decoupling assumption fails. The paper correctly identifies a real limitation of conventional VSG droop control and makes a reasonable conceptual case for using an adaptive critic design. The simulation setup covers two impedance regimes, and the use of standard VSG and HDP formulations gives the work a sound starting point. However, the evidence for the claimed superiority is currently qualitative only, the comparison baseline is not fully specified, the action-network update appears to be mis-derived, and the 'any impedance angle' claim is not tested. The significance is therefore conditional on substantial additional validation and correction.","major_comments":[{"comment":"The central claim that the HDP controller 'performs better' than the conventional controller is supported only by qualitative time-domain plots. No numerical metrics such as overshoot, settling time, steady-state error, or integrated squared error are reported, and there are no multiple runs, error bars, or statistical tests. The paper should provide quantitative comparison tables with clearly defined metrics and a statement of how many random initializations/experiments were used.","section":"Section IV, Figures 7-8"},{"comment":"The training and evaluation are performed in the same simulation model at only two fixed line configurations (inductive and resistive, Table I). There is no held-out impedance angle, no setpoint-change scenario, no disturbance test, and no robustness measure. The Abstract's claim that the controller 'can adapt to any impedance angle' is therefore unsupported by the presented evidence; at most, the two cases are consistent with that conjecture. The paper should validate on multiple R/X ratios and on conditions not seen during training.","section":"Section IV, training steps 1-4 and Table I"},{"comment":"The action-network weight update is written as ΔW_action = −α_a ζ ∂ζ/∂W_action with ζ = ∑_k ∂J(k+1)/∂E(k). This is not the gradient of the cost-to-go J with respect to the action weights; the correct backpropagation expression requires ∂J/∂W_action = (∂J/∂E)(∂E/∂W_action). As written, the update does not minimize the cost-to-go, so the optimality claim for the action network is not justified. Please correct the derivation or clarify the exact update rule implemented and how ∂J/∂E is computed.","section":"Section III.B, paragraphs on action network update"},{"comment":"The text states that the discount factor satisfies 0 < γ < 1, but Table I lists γ = 1. With an infinite-horizon cost and γ = 1, the cost-to-go J can diverge unless the system is asymptotically stable and the utility decays to zero; the paper does not address this. Additionally, the utility weights in Table I are given as [kP kQ kf] = [1 1 0], so the frequency error term is not actually used in the utility function despite being included in the definition. These inconsistencies should be resolved and the choices justified.","section":"Section III.A and Table I"},{"comment":"The PI baseline against which the HDP controller is compared is under-specified. No PI gains, tuning procedure, or exact controller structure are given, and the conventional controller described in Section II.A is an integrator with voltage droop rather than a standard PI. Without a precise definition of the baseline controller and its parameters, the claim that the HDP controller 'performs better than the conventional PIs' cannot be audited or reproduced.","section":"Section IV and Section II.A"}],"minor_comments":[{"comment":"The abstract contains a typo: 'an/or' should be 'and/or'.","section":"Abstract"},{"comment":"The text includes the placeholder '[some references that have used other controllers]', which must be replaced with actual citations.","section":"Section I, Introduction"},{"comment":"The term 'ADCs' appears where 'ACDs' (adaptive critic designs) is intended; please fix the abbreviation.","section":"Section III, paragraph 2"},{"comment":"The Bellman equation is written with summation index k over the same symbol as the time index, which is confusing; the right-hand side should use a dummy summation index and shifted time arguments, e.g., J(k) = ∑_{i=0}^∞ γ^i U(k+i+1).","section":"Section III.A, cost-to-go equation"},{"comment":"The resistive-grid subsection is labeled 'A. Resistive grid' but should be 'B. Resistive grid'.","section":"Section IV.B heading"},{"comment":"The phrase 'These coefficients can also be defined as the weight matrix in a normalized function' is unclear, as no normalization procedure is described.","section":"Section III.A, utility function"},{"comment":"The row 'ℽ 1' should be labeled as the discount factor γ, and the learning-rate row '[αc αc]  [1 1]' appears to conflate the critic and action learning rates; the notation should be clarified.","section":"Table I"},{"comment":"Reference [14] is missing publication year and publisher details; several references also lack page numbers.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an interesting practical problem and uses a plausible adaptive-critic framework, but the empirical comparison is not yet at the standard expected for a journal. The two main blockers are the unsupported 'any impedance angle' generalization and the apparently incorrect action-network update formula. Both are correctable: add quantitative metrics and held-out scenarios, and fix or clarify the update derivation. I would not reject because the core idea is sound and the deficiencies are of a kind that can be addressed in a thorough revision. However, if the authors are unable to provide additional validation experiments or a corrected derivation, the central claims would need to be substantially weakened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a legitimate extension — HDP applied to VSG voltage-magnitude control — and its motivation from P/Q coupling in resistive grids is sound. But the algorithm section as written has a real problem, and the empirical case is too thin to support the abstract's claims.\n\nWhat's genuinely there: the power-flow equations in Section II correctly show that in low-voltage/resistive grids, the conventional assumption that Q tracks voltage magnitude and P tracks angle breaks down. That is a real issue for VSG control. The idea of replacing the reactive-power PI with an HDP action-critic that takes both P and Q errors as inputs is sensible, and it is not in the cited HDP papers. The qualitative figures do show faster settling for the HDP in the resistive case.\n\nSoft spots, in decreasing order of severity:\n\n1. The action-network update in Section III.B is not a gradient of J. The paper defines ζ = Σ ∂J(k+1)/∂E(k) and then updates W_action by −α ζ ∂ζ/∂W_action, which minimizes ζ², not the cost-to-go J. If that equation is literal, the controller is not doing HDP. This needs a rewrite or a clarification.\n\n2. The comparison is entirely qualitative. No numerical metrics, no settling-time or overshoot values, no multiple runs. Training and evaluation use the same simulation model, and only two R/X cases are shown, so \"can adapt to any impedance angle\" is an overclaim. There is no held-out impedance angle or disturbance scenario.\n\n3. The PI baseline is underspecified — no gains, no tuning method — so \"better than PI\" cannot be audited.\n\n4. Minor but telling: the intro has a literal placeholder \"[some references that have used other controllers]\". That should have been resolved before submission.\n\n5. No code or data. Training details (learning rates, network initialization, number of episodes) are almost absent; the four-step procedure in Section IV could not be reproduced from the paper alone.\n\nWho is this for: people working on ADP/RL for power-electronic control, and VSG control in low-voltage microgrids. The idea is worth knowing, but the evidence is not.\n\nRecommendation: if I were an editor, I'd send it to review rather than desk reject, because the combination is new and the resistive-grid motivation is correct — but I'd expect heavy revision. The authors should fix the update rule, add metrics, report a range of R/X ratios, and tone down the generalization claim.","headline":"A credible VSG/HDP combination with a sound resistive-grid motivation, but the action-network update as written is wrong and the simulation evidence is too thin for the 'any impedance angle' claim.","tokens_in":7899,"tokens_out":3812,"would_cite":false,"duration_ms":37942,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a two-network heuristic dynamic programming controller, trained online, outperforms conventional PI voltage and reactive-power controllers for virtual synchronous generators, especially in resistive grids, and can…","keywords":["grid-connected inverter","heuristic dynamic programming","neural network","virtual synchronous generator","virtual inertia control","adaptive dynamic programming","reactive power control","resistive grid"],"falsifier":"Apply the same trained HDP networks to a grid with a 45-degree impedance angle (equal resistance and reactance) and to a step change in line resistance during operation; if settling time or reactive-power tracking is no better than the PI controller, or if the online weight updates fail to recover, the 'adapt to any impedance angle' claim is refuted.","tokens_in":6922,"feed_emoji":"⚡","tokens_out":5971,"duration_ms":56865,"temperature":0.7,"pith_summary":"This paper argues that the standard virtual synchronous generator (VSG) controller, which assumes the grid impedance is mostly inductive, breaks down in low-voltage resistive grids, and that a neural-network heuristic dynamic programming (HDP) controller can restore good tracking of active and reactive power there. The proposed controller replaces the conventional voltage and reactive-power PI loop with two small feedforward networks — an action network that sets the inverter voltage magnitude and a critic network that learns a cost-to-go value — trained together online. In simulation comparisons on inductive and resistive grids, the HDP controller reaches steady state faster than the PI controller in both cases and is reported to track power references better in the resistive case. If the claim holds, adaptive critic control would give inverter-based energy resources the frequency and voltage support of synchronous machines without retuning the controller for each unknown line impedance.","feed_headline":"Adaptive neural control beats PI in resistive-grid tests","feed_subtitle":"Heuristic dynamic programming lets virtual synchronous generators adapt where inductive-line assumptions fail.","key_machinery":"The load-bearing mechanism is the heuristic dynamic programming pair: an action network and a critic network. The critic network learns the cost-to-go $J(k)$ from the Bellman equation, with a discount factor $\\gamma$, by minimizing $[J(k) - \\gamma J(k+1) - U(k)]^2$ in gradient descent; the action network is trained through backpropagation to choose the inverter peak voltage $E(k)$ that minimizes the future cost. The utility function is $U(k) = \\sqrt{K_P e_P^2 + K_Q e_Q^2 + K_f e_f^2}$, built from active-power, reactive-power, and frequency errors. Because the action network takes both active and reactive power errors as inputs, it can capture the cross-coupling that appears when the line resistance is no longer negligible, which is the specific failure that motivates the paper.","core_discovery":"On the paper's own terms, the discovery is that a heuristic dynamic programming controller — two fully connected neural networks with two hidden layers of eight nodes each — can replace the integral/PI voltage-reactive power controller in a virtual synchronous generator and perform better than the conventional design under non-inductive line conditions. The paper derives power-flow equations showing that in inductive grids active power is approximately proportional to the power angle and reactive power to the voltage magnitude, so the conventional decoupled PI design works; in resistive grids that decoupling disappears, and the reactive power depends on both angle and voltage. The HDP action network receives active power, reactive power, frequency errors, and the inverter angle and outputs the voltage magnitude, while the critic network approximates the discounted cost-to-go $J(k)$ and is trained to satisfy the Bellman equation. Simulation results for the two fixed line configurations show the HDP reaching steady state faster in the inductive grid and tracking better in the resistive grid than the conventional PI controller, supporting the paper's conclusion that a well-trained HDP performs better than conventional PIs particularly in resistive grids.","pith_inferences":["The paper trains and evaluates at only two impedance angles (inductive and resistive); the natural next test is intermediate angles such as R/X = 1, where the coupling is strongest, to see whether the trained networks interpolate or need retraining.","Since the frequency error is present in the utility and input vectors but its coefficient $K_f$ is set to zero in the simulations, the same architecture could be exercised for simultaneous voltage and frequency control by activating that term.","A hardware-in-the-loop or field test with a step change in line impedance would separate genuine online adaptation from memorization of the two simulated operating points.","The random-initial-state training procedure suggests the controller may be robust to operating-point changes, but comparing it against a PI controller with adaptive gains under the same disturbances would give a stricter baseline."],"forward_implications":["For resistive-grid microgrids, the HDP controller would keep reactive power and voltage tracking without relying on the inductive-line approximation that invalidates conventional PI tuning.","The joint training of action and critic networks in the same cycle could shorten the adaptation time when the operating point changes.","Using both active- and reactive-power errors as network inputs removes the single-input single-output restriction that limits conventional PI voltage control in non-inductive grids.","The same control structure could be retrained rather than redesigned when line parameters change, because no analytic model of the impedance is needed for the neural network."],"supporting_citations":[{"why":"Introduces the virtual synchronous machine concept that the proposed controller is built on.","marker":"[3]"},{"why":"Establishes the equivalence between virtual synchronous machines and frequency droop control, grounding the VSG model used in the paper.","marker":"[4]"},{"why":"Defines adaptive critic designs, the class of controllers to which heuristic dynamic programming belongs.","marker":"[8]"},{"why":"Applies HDP to vector control of a grid-connected converter, the direct predecessor this paper extends to virtual inertia control.","marker":"[9]"},{"why":"Demonstrates direct HDP for power system stability, supporting HDP's suitability in power applications.","marker":"[10]"},{"why":"Shows nested-loop neural network vector control with HDP, providing the action/critic training pattern used here.","marker":"[11]"},{"why":"Supplies the inductive-grid assumption used to justify the conventional active/reactive power decoupling.","marker":"[12]"},{"why":"Also supports the inductive-line approximation in the standard power-flow equations that the paper argues fails in resistive grids.","marker":"[13]"},{"why":"Founds approximate dynamic programming for real-time control, the theoretical basis of the critic's cost-to-go learning.","marker":"[14]"}],"fun_headline_variants":["Neural HDP adapts VSG to resistive grids, beats PI","Heuristic dynamic programming outdoes PI when lines aren't inductive","Adaptive critic-action nets tune virtual inertia for any line","VSG control learns line impedance via two neural nets","Resistive grid? Neural HDP beats conventional PI in tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The controller's promised ability to adapt to any grid impedance rests on networks trained only in the same simulation at two fixed line configurations (inductive and resistive); if the training simply memorizes those two cases, the advantage over the PI controller would not carry over to other impedance angles.","fun_headline_variants_meta":{"raw":{"variants":["Neural HDP adapts VSG to resistive grids, beats PI","Heuristic dynamic programming outdoes PI when lines aren't inductive","Adaptive critic-action nets tune virtual inertia for any line","VSG control learns line impedance via two neural nets","Resistive grid? Neural HDP beats conventional PI in tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000713,"raw_usage":{"total_tokens":3169,"prompt_tokens":870,"completion_tokens":2299,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":2213}},"tokens_in":486,"tokens_out":2299,"duration_ms":16870,"temperature":1.0,"reasoning_tokens":2213,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:20:57.829705+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply the same trained HDP networks to a grid with a 45-degree impedance angle (equal resistance and reactance) and to a step change in line resistance during operation; if settling time or reactive-power tracking is no better than the PI controller, or if the online weight updates fail to recover, the 'adapt to any impedance angle' claim is refuted.","supporting_citations":[{"cited_title":"Adaptive Critic Designs,","cited_arxiv_id":null,"evidence_quote":"Defines adaptive critic designs, the class of controllers to which heuristic dynamic programming belongs."},{"cited_title":"Heuristic dynamic programming for neural network vector control of a grid-connected converter","cited_arxiv_id":null,"evidence_quote":"Applies HDP to vector control of a grid-connected converter, the direct predecessor this paper extends to virtual inertia control."},{"cited_title":"Direct heuristic dynamic programming method for power system stability enhancement,","cited_arxiv_id":null,"evidence_quote":"Demonstrates direct HDP for power system stability, supporting HDP's suitability in power applications."},{"cited_title":"Nested -loop neural network vector control of permanent magnet synchronous motors,","cited_arxiv_id":null,"evidence_quote":"Shows nested-loop neural network vector control with HDP, providing the action/critic training pattern used here."},{"cited_title":"Approximate dynamic programming for real -time control and neural modeling,","cited_arxiv_id":null,"evidence_quote":"Founds approximate dynamic programming for real-time control, the theoretical basis of the critic's cost-to-go learning."}],"review_version":1}