End-to-End Learning of Correlated Operating Reserve Requirements in Security-Constrained Economic Dispatch

Haihao Lu; Hung-po Chao; Owen Shen; Patrick Jaillet

arxiv: 2604.05167 · v1 · submitted 2026-04-06 · 🧮 math.OC · cs.SY· eess.SY

End-to-End Learning of Correlated Operating Reserve Requirements in Security-Constrained Economic Dispatch

Owen Shen , Hung-po Chao , Haihao Lu , Patrick Jaillet This is my paper

Pith reviewed 2026-05-10 18:37 UTC · model grok-4.3

classification 🧮 math.OC cs.SYeess.SY

keywords security-constrained economic dispatchoperating reservesellipsoidal uncertainty setsrobust optimizationend-to-end learningconformal predictionrenewable forecast errors

0 comments

The pith

Choosing the shape of an ellipsoidal uncertainty set for renewable forecast errors lets security-constrained economic dispatch achieve lower costs while meeting a target coverage level.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper formulates the design of correlated operating reserves as a single optimization problem: pick the shape of an ellipsoid that describes possible forecast errors so that the resulting robust dispatch has the lowest possible cost, yet still covers the errors at the desired rate. By rewriting the coverage requirement as a radius that depends on the shape, the problem becomes differentiable and can be solved end-to-end using gradients from the dispatch solver itself. A four-way data split trains the shape on one portion, tunes it on another, calibrates coverage on a third, and tests on the last, producing both a consistent gradient signal and finite-sample coverage guarantees under exchangeability. On the IEEE 118-bus system the learned ellipsoid cuts dispatch cost by roughly 4.8 percent relative to a sample-covariance baseline while still meeting the coverage target.

Core claim

The central discovery is that the bilevel problem of choosing an ellipsoidal uncertainty set to minimize robust SCED cost subject to a coverage constraint can be reduced to a single-stage differentiable program by profiling the radius from the coverage requirement, after which KKT information from the inner dispatch solve supplies task-specific gradients; when combined with a four-way train-tune-calibrate-test split and split conformal calibration, the procedure yields both a consistent estimator for the smoothed objective and finite-sample marginal coverage under exchangeability.

What carries the argument

The profiled ellipsoidal uncertainty set whose radius is chosen to enforce the coverage constraint exactly, allowing the outer minimization of robust dispatch cost to be performed with gradients obtained from the inner SCED dual solution without differentiating through the solver.

If this is right

The learned static ellipsoid reduces dispatch cost by about 4.8 percent relative to the sample-covariance baseline while keeping empirical coverage above the target.
The same task gradient can be passed upstream to context-dependent encoders that map exogenous features to ellipsoid shapes.
The framework applies directly to a coupled SCED formulation that includes inter-zone transfer constraints on the IEEE 118-bus system.
Split conformal calibration after training delivers finite-sample marginal coverage guarantees under exchangeability without requiring knowledge of the error distribution.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If real-time forecast errors in power systems satisfy the exchangeability assumption used for calibration, the same four-way split could be deployed operationally to produce reserves that adapt to changing conditions while retaining statistical coverage.
The approach could be tested on other uncertainty representations, such as polyhedral sets, by deriving analogous profiling rules for their coverage constraints.
Lower dispatch costs from better-shaped reserve sets would reduce the total amount of flexible generation needed to integrate renewables, lowering system operating expenses on networks with high wind and solar penetration.

Load-bearing premise

The coverage requirement can be turned into a radius that depends only on the ellipsoid shape in a way that keeps the original robust guarantees intact, and the four-way data split produces finite-sample marginal coverage when the forecast errors are exchangeable.

What would settle it

Run the learned ellipsoid on a fresh test set of forecast errors drawn from the same distribution; if the empirical coverage drops below the target or the cost advantage over the sample-covariance baseline disappears, the claim is falsified.

read the original abstract

Operating reserve requirements in security-constrained economic dispatch (SCED) depend strongly on the assumed correlation structure of renewable forecast errors, yet that structure is usually specified exogenously rather than learned for the dispatch task itself. This paper formulates correlated reserve-set design as an end-to-end trainable robust optimization problem: choose the ellipsoidal uncertainty-set shape to minimize robust dispatch cost subject to a target coverage requirement. By profiling the coverage constraint into a shape-dependent radius, the original bilevel problem becomes a single-stage differentiable objective, and KKT/dual information from the SCED solve provides task gradients without differentiating through the solver. For unknown distributions, a four-way train/tune/calibrate/test split combines a smoothed quantile-sensitivity estimator for training with split conformal calibration for deployment, yielding finite-sample marginal coverage under exchangeability and a consistent gradient estimator for the smoothed objective. The same task gradient can also be passed upstream to context-dependent encoders, which we report as a secondary extension. The framework is evaluated on the IEEE~118-bus system with a coupled SCED formulation that includes inter-zone transfer constraints. The learned static ellipsoid reduces dispatch cost by about 4.8\% relative to the Sample Covariance baseline while maintaining empirical coverage above the target level.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper gives a clean profiled single-stage formulation plus KKT gradients for learning ellipsoidal reserves jointly with SCED, but the split-conformal coverage claim rests on exchangeability that renewable errors violate.

read the letter

The main takeaway is that this work turns correlated reserve design into an end-to-end differentiable problem by profiling the coverage constraint into a shape-dependent radius, then pulls task gradients from the KKT conditions of the SCED solve without differentiating through the optimizer. The four-way split with smoothed quantile training and split conformal calibration is the concrete mechanism they use to get a consistent gradient estimator and a finite-sample marginal coverage guarantee under exchangeability. That combination looks new relative to separate covariance estimation followed by fixed robust dispatch. On the IEEE 118-bus system they report a 4.8% dispatch cost reduction versus the sample-covariance baseline while keeping empirical coverage above target, which is a practical result worth noting. The formulation handles inter-zone transfer constraints and the static ellipsoid case is worked out clearly. The technical steps are coherent at the level of the abstract and the gradient estimator claim is standard implicit differentiation applied to this setting. The soft spot is the exchangeability assumption required for the conformal guarantee. Renewable forecast errors are time series with autocorrelation, diurnal cycles, and weather-driven non-stationarity, so random splits do not preserve the joint distribution. The marginal coverage that holds on the held-out test set therefore does not reliably carry over to deployment, which undercuts the safety interpretation of the cost savings. The paper states the guarantee under exchangeability but does not appear to test or correct for temporal dependence on the actual data. This is for people working on robust optimization in power systems or on end-to-end learning inside constrained dispatch problems. A reader who already knows implicit differentiation and conformal prediction will pick up the specific profiling trick and the SCED application quickly. The work is coherent enough and the experiments are on a standard test case, so it deserves a serious referee even though the coverage issue needs direct attention in revision.

Referee Report

2 major / 2 minor

Summary. The paper formulates correlated operating reserve design in SCED as an end-to-end trainable robust optimization problem: an ellipsoidal uncertainty set shape is chosen to minimize robust dispatch cost subject to a target coverage level. The bilevel problem is reduced to a single-stage differentiable objective by profiling the coverage constraint into a shape-dependent radius; task gradients are obtained via KKT conditions of the inner SCED without differentiating through the solver. A four-way train/tune/calibrate/test split uses a smoothed quantile-sensitivity estimator for training and split conformal prediction for calibration, delivering finite-sample marginal coverage under exchangeability. On the IEEE 118-bus system the learned static ellipsoid yields an approximately 4.8% dispatch-cost reduction relative to the sample-covariance baseline while preserving empirical coverage above the target.

Significance. If the coverage guarantees transfer to deployment, the framework offers a principled, task-specific alternative to exogenous or purely statistical reserve-set design, with potential efficiency gains in power-system operations. Strengths include the consistent gradient estimator obtained from KKT information, the finite-sample marginal coverage result via split conformal prediction, and the explicit four-way data split that separates training from calibration. These elements provide both optimization and statistical grounding that is uncommon in applied robust-optimization work.

major comments (2)

[Abstract and §3] Abstract and §3 (four-way split and split conformal calibration): the finite-sample marginal coverage guarantee is stated to hold under exchangeability. Renewable forecast errors on the IEEE 118-bus instance are time-series data with autocorrelation, diurnal cycles, and non-stationarity; exchangeability is therefore violated. The manuscript must either (a) replace the guarantee with a weaker but still valid statement (e.g., asymptotic coverage or block-conformal bounds) or (b) supply empirical evidence that the coverage remains above target on temporally contiguous test periods that respect the dependence structure. Without this, the 4.8% cost-reduction claim cannot be interpreted as a safe improvement over the sample-covariance baseline.
[Abstract and §2.2] Abstract and §2.2 (profiling step): the coverage constraint is profiled into a shape-dependent radius to obtain a single-stage differentiable objective. The manuscript must explicitly verify that this profiling preserves the original robust-optimization feasible set exactly (i.e., that the profiled radius is not an approximation that can shrink the uncertainty set below the target coverage level). Any approximation error would directly undermine the central cost-reduction claim.

minor comments (2)

Notation for the smoothed quantile-sensitivity estimator should be introduced with a short derivation or reference to the exact smoothing parameter schedule used in the experiments.
Figure captions for the IEEE 118-bus results should state the exact number of Monte-Carlo replications and the precise definition of “empirical coverage” (e.g., fraction of test scenarios inside the calibrated ellipsoid).

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful reading and constructive comments on our manuscript. We respond to each major comment below, indicating the revisions we will make.

read point-by-point responses

Referee: [Abstract and §3] Abstract and §3 (four-way split and split conformal calibration): the finite-sample marginal coverage guarantee is stated to hold under exchangeability. Renewable forecast errors on the IEEE 118-bus instance are time-series data with autocorrelation, diurnal cycles, and non-stationarity; exchangeability is therefore violated. The manuscript must either (a) replace the guarantee with a weaker but still valid statement (e.g., asymptotic coverage or block-conformal bounds) or (b) supply empirical evidence that the coverage remains above target on temporally contiguous test periods that respect the dependence structure. Without this, the 4.8% cost-reduction claim cannot be interpreted as a safe improvement over the sample-covariance baseline.

Authors: We agree that the finite-sample marginal coverage guarantee relies on exchangeability, which is violated by the temporal dependence in renewable forecast errors. To address this, we will add experiments in the revised manuscript using temporally contiguous train/test splits that respect the autocorrelation and non-stationarity. These will confirm that empirical coverage stays above the target, thereby supporting the reported cost reduction under realistic conditions. We will also update the abstract and §3 to explicitly state that the theoretical guarantee assumes exchangeability while the empirical results hold more generally. This is a partial revision focused on adding the requested empirical validation. revision: partial
Referee: [Abstract and §2.2] Abstract and §2.2 (profiling step): the coverage constraint is profiled into a shape-dependent radius to obtain a single-stage differentiable objective. The manuscript must explicitly verify that this profiling preserves the original robust-optimization feasible set exactly (i.e., that the profiled radius is not an approximation that can shrink the uncertainty set below the target coverage level). Any approximation error would directly undermine the central cost-reduction claim.

Authors: The profiling step preserves the feasible set exactly. For a fixed ellipsoid shape, the radius is defined as the smallest value such that the coverage constraint is satisfied at or above the target level. Since the robust dispatch cost is monotonically non-decreasing in the radius, any optimal solution to the bilevel problem occurs at equality in the coverage constraint. Hence, minimizing cost over shapes using the profiled radius is equivalent to the original formulation. During training we employ a smoothed quantile estimator for differentiability, but the deployed model uses the exact radius obtained via split conformal prediction. We will add a short proposition and proof sketch in §2.2 to formalize this equivalence. No change to the central claims is needed. revision: yes

Circularity Check

0 steps flagged

No significant circularity in the derivation chain

full rationale

The paper's central derivation profiles the coverage constraint into a shape-dependent radius to convert the bilevel problem into a single-stage differentiable objective, then applies standard KKT-based implicit differentiation for task gradients and split conformal prediction on a four-way data split for finite-sample marginal coverage under exchangeability. These steps rely on established mathematical reformulations and conformal prediction properties rather than redefining outputs in terms of inputs. The reported 4.8% cost reduction is an empirical outcome on held-out test data after separate training, tuning, and calibration phases, not equivalent to the fitted parameters or assumptions by construction. No self-definitional reductions, fitted inputs renamed as predictions, or load-bearing self-citations appear in the provided derivation.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The framework rests on exchangeability for the coverage guarantee and on the validity of the ellipsoidal representation plus the profiling step for differentiability; no free parameters or invented entities are introduced beyond standard robust optimization constructs.

axioms (1)

domain assumption Exchangeability of the data samples for finite-sample marginal coverage via split conformal calibration
Invoked to obtain the coverage guarantee after the four-way train/tune/calibrate/test split.

pith-pipeline@v0.9.0 · 5532 in / 1229 out tokens · 25432 ms · 2026-05-10T18:37:09.937242+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

By profiling the coverage constraint into a shape-dependent radius, the original bilevel problem becomes a single-stage differentiable objective, and KKT/dual information from the SCED solve provides task gradients
IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

split conformal calibration for deployment, yielding finite-sample marginal coverage under exchangeability

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages

[1]

16 Dimitris Bertsimas, David B

doi: 10.1287/opre.1030.0065. 16 Dimitris Bertsimas, David B. Brown, and Constantine Caramanis. Theory and applications of robust optimization.SIAM Review, 53(3):464–501, 2011. doi: 10.1137/080734510. Dimitris Bertsimas, Eugene Litvinov, Xu Andy Sun, Jinye Zhao, and Tongxin Zheng. Adaptive robust optimization for the security-constrained unit commitment pr...

work page doi:10.1287/opre.1030.0065 2011
[2]

Frédéric and Shapiro, Alexander , year =

doi: 10.1007/978-1-4612-1394-9. Abhilash Reddy Chenreddy and Erick Delage. End-to-end conditional robust optimization. In Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence, volume 244 ofProceedings of Machine Learning Research, pages 736–748. PMLR, 2024. URLhttps: //proceedings.mlr.press/v244/chenreddy24a.html. Frank H. Clar...

work page doi:10.1007/978-1-4612-1394-9 2024

[1] [1]

16 Dimitris Bertsimas, David B

doi: 10.1287/opre.1030.0065. 16 Dimitris Bertsimas, David B. Brown, and Constantine Caramanis. Theory and applications of robust optimization.SIAM Review, 53(3):464–501, 2011. doi: 10.1137/080734510. Dimitris Bertsimas, Eugene Litvinov, Xu Andy Sun, Jinye Zhao, and Tongxin Zheng. Adaptive robust optimization for the security-constrained unit commitment pr...

work page doi:10.1287/opre.1030.0065 2011

[2] [2]

Frédéric and Shapiro, Alexander , year =

doi: 10.1007/978-1-4612-1394-9. Abhilash Reddy Chenreddy and Erick Delage. End-to-end conditional robust optimization. In Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence, volume 244 ofProceedings of Machine Learning Research, pages 736–748. PMLR, 2024. URLhttps: //proceedings.mlr.press/v244/chenreddy24a.html. Frank H. Clar...

work page doi:10.1007/978-1-4612-1394-9 2024