Pith. sign in

REVIEW 2 major objections 5 minor 22 references

A tractable formulation for multi-period linearized optimal power flow in presence of thermostatically controlled loads

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A change of variables turns hard stochastic control problems into convex programs.

desk verdict Clean MDP convexification, honest about prior work, but the TCL discretization has a factor-of-two diffusion error that should be corrected. read the letter →

arxiv 1908.09167 v1 pith:7YVWFOAB submitted 2019-08-24 eess.SY cs.SY

Emilio Benenati , Marcello Colombino , Emiliano Dall'Anese This is my paper · ORCID
classification eess.SYcs.SY MSC 90C4090C2593E20
keywords MarkovdecisionprocessesconvexoptimizationthermostaticallycontrolledloadsoptimalpowerflowrecedinghorizoncontrolFokker-Planckdiscretizationdistributedenergyresourcesstochastic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper establishes that a broad class of constrained finite-horizon Markov decision process (MDP) optimal control problems, which are generally nonconvex and intractable, can be recast as equivalent convex optimization problems through the change of variables $M^t := \Pi^t \mathrm{diag}(\rho^t)$. It then applies this result to an ensemble of thermostatically controlled loads (TCLs), proposing a discretization of the Fokker-Planck equation that yields a valid Markov-chain model under weaker assumptions than earlier work. Building on these pieces, the paper formulates a multi-period AC optimal power flow problem with TCLs and distributed energy resources as a convex program, and demonstrates in simulation that a receding-horizon controller can track a substation power reference and reduce PV curtailment. A sympathetic reader would care because the result turns a class of hard stochastic control problems into one solvable by standard convex solvers, with direct grid applications.

What carries the argument

The load-bearing object is the matrix $M^t = \Pi^t \mathrm{diag}(\rho^t)$, whose $(i,j)$ entry is the joint probability $\mathbb{P}(y(t+1)=i, y(t)=j)$. It linearizes the occupancy dynamics and converts the three allowed constraint forms of Definition 2.1 into convex constraints on $(\rho^t,M^t)$; the feasible set is $\mathcal{M}_N = \{(\rho,M)\in\mathbb{R}^N_{\ge 0}\times\mathbb{R}^{N\times N}_{\ge 0} : \mathbf{1}^\top M = \rho^\top\}$. The corresponding optimal Markov transition matrices are recovered by dividing column $j$ by $\rho^t_j$ when positive. A second mechanism is the asymmetric discretization of the Fokker-Planck equation, in which the density in each temperature bin is approximated by the value at the boundary toward which the system evolves, yielding a Metzler continuous-time generator whose Euler discretization is a valid stochastic matrix precisely when condition (18) holds, with no assumption on noise variance.

What would settle it

Run the proposed receding-horizon controller on a distribution feeder and compare the committed substation power and voltages against a full nonlinear AC power-flow simulation after each step: if the linear model's known offset pushes the actual substation power outside the tracking band while the linearized constraints report feasibility, the central guarantee that the OPF constraints enforce the grid limits would be false. For the MDP theorem itself, a counterexample would be a problem satisfying Assumptions 1 and 2 for which the optimal value of (7) is strictly smaller than that of (3).

Watch

Extended reading notes

Core claim

The central claim is Proposition 2.2: under mild convexity and feasibility assumptions (Assumptions 1 and 2), the nonlinear constrained MDP problem (3) is equivalent to the convex problem (7) after substituting $M^t = \Pi^t \mathrm{diag}(\rho^t)$. The entries $M^t_{(i,j)}$ are joint probabilities of being in state $j$ at time $t$ and state $i$ at $t+1$, so the bilinear dynamics $\rho^{t+1}=\Pi^t\rho^t$ become linear, $\rho^{t+1}=M^t\mathbf{1}$, and tractable constraints become convex in the new variables. The paper further claims that for TCL populations, an asymmetric discretization of the Fokker-Planck equations yields a genuine transition matrix without requiring large noise, and that the resulting finite-horizon TCL optimal control problem, embedded in a linearized AC OPF with DERs, is convex. It closes by showing numerical evidence of improved substation tracking and reduced curtailment on a distribution feeder.

Load-bearing premise

The whole application rests on the assumption that the linearized power-flow model (20)-(21) is accurate enough that the voltage and substation-power constraints in (26c)-(26d) are truly satisfied by the nonlinear AC network; the paper's own Figure 5 shows a persistent offset between the linear model and the nonlinear solution for substation power, so this holds only approximately and no offset-free mechanism is included.

Editorial extensions

If this is right

  • TCL switching probabilities and DER set-points can be decided in a single convex multi-period optimization, so standard convex solvers can replace bespoke stochastic-control algorithms.
  • A receding-horizon implementation can keep substation power near a reference during cloud-induced PV drops, reducing both tracking error and total PV curtailment compared with DER-only control.
  • The asymmetric discretization gives a Markov-chain TCL model that remains valid at zero process noise, provided the temperature and time grids satisfy the explicit condition (18).
  • The convex reformulation applies to any constrained MDP whose objective is convex and whose constraints take the three forms in Definition 2.1, not only to TCLs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Any population process with linear expectations in occupancy, such as fleets of batteries or electric vehicles, could plausibly use the same joint-probability lifting as long as costs and constraints are convex in occupancy and joint transition measures.
  • The constant offset between the linear model and the nonlinear AC solution shown in the paper's substation-power plot suggests that field deployment would need measurement feedback or integral action; the paper explicitly defers that, and adding such a loop is the natural next step.
  • The grid-spacing condition (18) could be used adaptively: a solver could coarsen temperature bins or refine the time step online to keep $\Pi^{\mathrm{nat}}$ stochastic without changing the convex structure.
  • Because the TCL model is built on expected counts, the controller commands switching probabilities rather than individual switches; the paper's simulations sample individual TCL trajectories, so the gap between expected and realized consumption (finite-population noise) is a testable robustness question.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a convex reformulation for a class of finite-horizon constrained Markov decision process (MDP) optimal control problems. The key result, Proposition 2.2, introduces the change of variables M_t = Π_t diag(ρ_t), which turns the bilinear dynamics and a class of 'tractable' constraints into a convex program; an optimal Π_t is then reconstructed from M_t via Eq. (8), with zero-probability columns handled by Assumption 2. The paper then constructs a Markov-chain model for a population of thermostatically controlled loads (TCLs) by an asymmetric discretization of the Fokker-Planck equation, and embeds this model in a multi-period linearized AC optimal power flow (OPF) problem with DER hardware constraints, voltage limits, substation power tracking, and comfort constraints (Problem (26)). A receding-horizon implementation is tested on a modified IEEE 37-node feeder with PV systems and three TCL populations, comparing the linear power-flow model against Matpower's nonlinear AC solution.

Significance. If correct, Proposition 2.2 provides a clean and useful convexification for constrained MDPs; the proof by reconstruction from M is sound, including the treatment of zero-probability states, and it extends the approach of [4] to generic convex constraints in a transparent way. The TCL-OPF application is timely: Problem (26) is convex and can be solved with standard solvers, which is a meaningful step beyond prior TCL formulations. The numerical study uses Matpower as an external benchmark and demonstrates qualitative benefits such as reduced PV curtailment and substation reference tracking. However, the TCL discretization in Section III-B contains internal inconsistencies in the diffusion coefficient and in the scaling of the switching matrix. Because that model is used in the numerical validation, the application claim is not fully supported as written, even though the convex MDP result itself and the high-level OPF formulation are unaffected by these errors.

major comments (2)
  1. [III-B, Eq. (16) and Eq. (18)] The diffusion coefficient in the discretization is a factor of two too large. The Fokker-Planck equation (11) and the integrated flux (13) have diffusion coefficient σ²/2, so the off-diagonal diffusive entries in Aon and Aoff should be σ²/(2Δx) and the diagonal entries should contain -σ²/Δx before the 1/Δx prefactor. The displayed matrices instead use σ²/Δx and -2σ²/Δx, which after the prefactor give σ²/Δx² and -2σ²/Δx². The same factor appears in Aswitch. Consequently, the Markov chain used in Section V has diffusion coefficient σ² rather than the σ²/2 of the stated SDE, and the stability condition (18) is the nonnegativity condition for the wrong diffusion coefficient. Please correct Eq. (16) and Eq. (18), or explicitly state that the discretized model intentionally uses a different noise level.
  2. [III-B, Eq. (16)] The definition A_nat := (1/Δx) blockdiag(Aon, Aoff) + Aswitch is dimensionally inconsistent as written: the displayed entries of Aswitch contain terms f(θ) and σ²/Δx that are not divided by Δx, whereas the block-diagonal terms are divided by Δx. With the displayed definitions the column sums of A_nat are not identically zero, contradicting the claim that 1ᵀ A_nat = 0ᵀ and hence the validity of Π_nat = I + Δt A_nat in P_N. Please clarify the intended scaling of Aswitch and the indexing of the ON/OFF blocks, in particular the role of N/2 when the number of ON states is k̄.
minor comments (5)
  1. [II, proof of Proposition 2.2] In the proof, the inequality labeled "Clearly f⋆₁ ≤ f⋆₂" should be "f⋆₂ ≤ f⋆₁"; the reverse inequality is established later in the proof. This is a typo and does not affect the conclusion.
  2. [V, cost function] The cost function expression 1/T ∑_{t=1}^{t} is not well defined; the upper summation index should be T or T−1. Additionally, γM is described as a cost on control action "at the PV systems" but it multiplies TCL switching probabilities, so the wording should be corrected.
  3. [V, numerical experiments] The TCL state evolution is said to be sampled once; for a stochastic system, a single sample path does not substantiate the claimed tracking and curtailment improvements. Please report the number of Monte Carlo runs or provide confidence intervals.
  4. [V, Figure 5] Figure 5 and the final paragraph of Section V acknowledge a constant offset between the linear model and the Matpower nonlinear solution. This means the substation-power constraint (26d) is only approximately enforced in closed loop; the limitation should be stated in the abstract or conclusion, not only in the figure caption.
  5. [Definition 2.1, Eq. (4c)] In the definition of the tractable constraint of type (4c), the summation index i collides with the constraint index i; please use a different dummy index such as ℓ for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the MDP convexification is a direct change of variables with a constructive equivalence proof, and the TCL/OPF application rests on external benchmarks and prior linearization models rather than on self-derived predictions.

full rationale

The paper's central theoretical claim, Proposition 2.2, is self-contained: the change of variables M_t = Π_t diag(ρ_t) is a bijection onto the set M_N, and the proof explicitly reconstructs feasible Π_t from optimal M_t using (8), including the zero-probability-column case handled by Assumption 2. No fitted parameter is renamed as a prediction; the cost weights in Section V are chosen by hand and are not used to fit the reported outcomes. The power-flow linearizations (20)-(21) are imported from prior external work [18]-[20] as modeling assumptions, not derived from the paper's own claims, and the numerical validation compares against Matpower's nonlinear AC power flow, an external benchmark. The only potentially problematic passage is the internal modeling discrepancy noted by the skeptic: the diffusion coefficient in the discretized matrices A_on and A_off in (16) appears to be a factor of two larger than the σ²/2 appearing in the Fokker-Planck equation (11). That is a correctness or consistency concern about the application model, not a circularity: it does not make any claim reduce to its own inputs by construction. No load-bearing argument reduces to a self-citation, and no uniqueness theorem is invoked to forbid alternatives. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 9 assumptions · 0 invented entities

No invented entities and no fitted constants; the central theorem is parameter-free. The application relies on standard modeling assumptions: Markovian dynamics, the Fokker-Planck thermal model, the asymmetric upwind discretization, condition (18), a linearized power-flow model, and an exact-penalty soft constraint. Simulation-specific weights and discretization steps are choices, not fitted data.

free parameters (3)
  • Cost weights gamma_P, gamma_Q, gamma_M, gamma_P0 = 3, 2, 1, 10^6
    Chosen by hand in Section V; they shape the numerical trade-off between curtailment, reactive power, switching, and substation tracking, so they influence the reported performance but are not fitted to data.
  • Discretization steps Delta_x and Delta_t = 0.1 C and 20 s
    The validity of the transition matrix rests on condition (18); these step sizes are chosen in Section V to satisfy it.
  • TCL model parameters (C, R, P, sigma, eta, theta_a, deadband, grid bounds) = see Table I
    Taken from typical residential heat-pump values [2]; not fitted, but the simulation results depend on them.
assumptions (9)
  • domain assumption The population of agents evolves as a Markov chain with transition matrix Pi_t (equation (1)).
    The entire MDP reformulation relies on the Markovian assumption, stated at the start of Section II.
  • standard math Cost f and constraints g or c_i are convex (Assumption 1 and Section IV).
    Convexity is required for the reformulated problem to be convex; it is an assumption on the problem class.
  • standard math Feasibility of input constraints (Assumption 2).
    Guarantees reconstruction of a feasible Pi in zero-probability states; the authors note it is easily tested.
  • domain assumption TCL temperature dynamics follow SDE (9) and Fokker-Planck equation (11).
    The aggregate model is built on this stochastic thermal model.
  • domain assumption The asymmetric discretization (14) approximates the density as constant over each bin and evaluated at the upwind edge.
    This is the modeling choice that yields transition matrix (17); if the approximation is poor, the Markov chain is not faithful to the original PDE.
  • domain assumption Condition (18) holds so that Pi_nat = I + Delta_t A_nat is a valid stochastic matrix.
    Requires diagonal elements nonnegative; with the chosen Delta_x, Delta_t and f1 > 0 it is satisfied in Section V.
  • domain assumption The grid voltage and substation power are linearly related to injections via (20)-(21).
    The AC OPF constraints (26c)-(26d) are valid only to the extent this linearization is accurate; Section V shows a constant offset.
  • domain assumption TCLs consume no reactive power.
    Simplifies the power-flow coupling; stated in Section IV.
  • standard math Exact penalty property of soft constraint with gamma_P0 high (result [22]).
    Used in Section V to replace the hard tracking constraint by a soft constraint; requires a sufficiently large weight.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A tractable formulation for multi-period linearized optimal power flow in presence of thermostatically controlled loads." pith.science (2026). https://pith.science/paper/7YVWFOAB

@misc{pith2026190809167,
  author       = {Pith},
  title        = {Pith review of: A tractable formulation for multi-period linearized optimal power flow in presence of thermostatically controlled loads},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7YVWFOAB}},
  note         = {Machine review of arXiv:1908.09167}
}
read the original abstract

This paper presents a convex reformulation of a nonlinear constrained optimization problem for Markov decision processes, and applies the technical findings to optimal control problems for an ensemble of thermostatically controlled loads (TCLs). The paper further explores the formulation and solution of a (linearized) AC optimal power flow problem when one or more ensembles of TCLs are connected to a power network. In particular, a receding horizon controller is proposed, to simultaneously compute the optimal set-points of distributed energy resources (DERs) in the grid and the optimal switching signal for the TCLs. This formulation takes into account hardware constraints of the DERs, operational constraints of the grid (e.g., voltage limits), comfort of the TCL users, and ancillary services provision at the substation. Numerical results are provided to verify the effectiveness of the proposed methodology.

Figures

Figures reproduced from arXiv: 1908.09167 by the authors.

Figure 1
Figure 1. The MDP representing a TCL evolution. The index of eac [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Maximum instantaneous active power available at the [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Advantages in using the TCL control. The first graph sh [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of linear and nonlinear simulation for th [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Comparison of the linear and nonlinear models for the [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages

  1. [4]

    Solution met hods for constrained markov decision process with continuous proba bility mod- ulation,

    M. Petrik, D. Subramanian, and J. Marecki, “Solution met hods for constrained markov decision process with continuous proba bility mod- ulation,” Uncertainty in Artificial Intelligence - Proceedings of the 29th Conference, UAI 2013 , 09 2013

    work page 2013

  2. [6]

    On the rang e of fea- sible power trajectories for a population of thermostatica lly controlled loads,

    D. Paccagnan, M. Kamgarpour, and J. Lygeros, “On the rang e of fea- sible power trajectories for a population of thermostatica lly controlled loads,” in 54th IEEE Conference on Decision and Control , 12 2015

    work page 2015

  3. [1]

    Achieving controllabi lity of electric loads,

    D. S. Callaway and I. A. Hiskens, “Achieving controllabi lity of electric loads,” Proceedings of the IEEE , vol. 99, no. 1, pp. 184–199, Jan 2011

    work page 2011

  4. [2]

    Using reside ntial electric loads for fast demand response: The potential reso urce and revenues, the costs, and policy recommendations,

    J. L. Mathieu, M. Dyson, and D. S. Callaway, “Using reside ntial electric loads for fast demand response: The potential reso urce and revenues, the costs, and policy recommendations,” in In Proceedings of the ACEEE Summer Study on Buildings , 2012. 8 10 12 14 160 0.2 0.4 0.6 0.8 Day time [h] E [MWh] Reduction in power curtailment 8 10 12 14 16 −1 −0.5 0 0...

    work page 2012

  5. [3]

    Optimal ensemble c ontrol of loads in distribution grids with network constraints,

    M. Chertkov, D. Deka, and Y . Dvorkin, “Optimal ensemble c ontrol of loads in distribution grids with network constraints,” in 2018 Power Systems Computation Conference (PSCC) . IEEE, 2018, pp. 1–7

    work page 2018

  6. [5]

    Electric load model synthes is by dif- fusion approximation of a high-order hybrid-state stochas tic system,

    R. Malhame and C.-Y . Chong, “Electric load model synthes is by dif- fusion approximation of a high-order hybrid-state stochas tic system,” IEEE Transactions on Automatic Control , vol. 30, pp. 854 – 860, 10 1985

    work page 1985

  7. [7]

    Dis- tributed control design for balancing the grid using flexibl e loads,

    Y . Chen, M. U. Hashmi, J. Mathias, A. Buˇ si´ c, and S. Meyn, “Dis- tributed control design for balancing the grid using flexibl e loads,” in Energy Markets and Responsive Grids . Springer, 2018, pp. 383–411

    work page 2018

  8. [8]

    Ancil lary service to the grid through control of fans in commercial bui lding hvac systems,

    H. Hao, Y . Lin, A. S. Kowli, P . Barooah, and S. Meyn, “Ancil lary service to the grid through control of fans in commercial bui lding hvac systems,” IEEE Transactions on smart grid , vol. 5, no. 4, pp. 2066–2074, 2014

    work page 2014

Show all 22 references
  1. [9]

    Tapping the energy storage potential in electric loads to deliver load following and regulation, with application to wind energy,

    D. S. Callaway, “Tapping the energy storage potential in electric loads to deliver load following and regulation, with application to wind energy,” Energy Conversion and Management, vol. 50, no. 5, pp. 1389 – 1400, 2009

    2009

  2. [10]

    State estima tion and control of electric loads to manage real-time energy imbala nce,

    J. L. Mathieu, S. Koch, and D. S. Callaway, “State estima tion and control of electric loads to manage real-time energy imbala nce,” IEEE Transactions on Power Systems, vol. 28, no. 1, pp. 430–440, Feb 2013

    2013

  3. [11]

    Distributed mpc of aggregate d het- erogeneous thermostatically controlled loads in smart gri d,

    M. Liu, Y . Shi, and X. Liu, “Distributed mpc of aggregate d het- erogeneous thermostatically controlled loads in smart gri d,” IEEE Transactions on Industrial Electronics , vol. 63, no. 2, pp. 1120–1129, Feb 2016

    2016

  4. [12]

    A c ontrol system framework for privacy preserving demand response of thermal inertial loads,

    A. Haider, X. Geng, G. Sharma, L. Xie, and P . R. Kumar, “A c ontrol system framework for privacy preserving demand response of thermal inertial loads,” in 2015 IEEE International Conference on Smart Grid Communications (SmartGridComm) , Nov 2015, pp. 181–186

    2015

  5. [13]

    Modeling and control of aggr egate air conditioning loads for robust renewable power management,

    S. Bashash and H. K. Fathy, “Modeling and control of aggr egate air conditioning loads for robust renewable power management, ” IEEE Transactions on Control Systems Technology, vol. 21, no. 4, pp. 1318– 1327, July 2013

    2013

  6. [14]

    Packetized en ergy manage- ment: Asynchronous and anonymous coordination of thermost atically controlled loads,

    M. Almassalkhi, J. Frolik, and P . Hines, “Packetized en ergy manage- ment: Asynchronous and anonymous coordination of thermost atically controlled loads,” in 2017 American Control Conference (ACC) , May 2017, pp. 1431–1437

    2017

  7. [15]

    Best-response pl an- ning of thermostatically controlled loads under power cons traints,

    F. De Nijs, M. Spaan, and M. De Weerdt, “Best-response pl an- ning of thermostatically controlled loads under power cons traints,” in Proceedings of the Twenty-Ninth AAAI Conference on Artifici al Intelligence, B. Bonet and S. Koenig, Eds., vol. 1. Association for the Advanceme...

    2015

  8. [16]

    Markov decision processes,

    C. C. White and D. J. White, “Markov decision processes, ” European Journal of Operational Research , vol. 39, no. 1, pp. 1 – 16, 1989

    1989

  9. [17]

    D. P . Bertsekas, Dynamic Programming and Optimal Control , 2nd ed. Athena Scientific, 2000

    2000

  10. [18]

    Load flow in multiphase distribution networks: Existence, uniqueness, non-singularity and linear models,

    A. Bernstein, C. Wang, E. Dall’Anese, J. Le Boudec, and C . Zhao, “Load flow in multiphase distribution networks: Existence, uniqueness, non-singularity and linear models,” IEEE Transactions on Power Systems, vol. 33, no. 6, pp. 5832–5843, Nov 2018

    2018

  11. [19]

    Linear power-flow mode ls in mul- tiphase distribution networks,

    A. Bernstein and E. Dall’Anese, “Linear power-flow mode ls in mul- tiphase distribution networks,” in 2017 IEEE PES Innovative Smart Grid Technologies Conference Europe (ISGT-Europe) , Sep. 2017, pp. 1–6

    2017

  12. [20]

    Fast power system analysi s via implicit linearization of the power flow manifold,

    S. Bolognani and F. D¨ orfler, “Fast power system analysi s via implicit linearization of the power flow manifold,” in in Proc. 53rd Annual Allerton Conference on Communication, Control, and Comput ing, 2015

    2015

  13. [21]

    Development of a high resoluti on, real time, distribution-level metering system and associated v isualization, modeling, and data analysis functions,

    J. Bank and J. Hambrick, “Development of a high resoluti on, real time, distribution-level metering system and associated v isualization, modeling, and data analysis functions,” Technical report, 5 2013

    2013

  14. [22]

    Necessary and sufficient conditions f or a penalty method to be exact,

    D. P . Bertsekas, “Necessary and sufficient conditions f or a penalty method to be exact,” Mathematical programming , vol. 9, no. 1, pp. 87–99, 1975

    1975

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.