Pith. sign in

REVIEW 3 major objections 4 minor 17 references

MLMC-based Resource Adequacy Assessment with Active Learning Trained Surrogate Models

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read This paper establishes that active-learning-trained surrogate models, with training time counted in the budget, make multilevel Monte Carlo resource adequacy assessment faster than random-sampled surrogates at comparable cost.

desk verdict Useful speed-metric fix and an honest AL-in-MLMC study, but the headline efficiency gain is modest and lacks uncertainty quantification. read the letter →

arxiv 2505.20930 v2 pith:WYKKVWBU submitted 2025-05-27 cs.LG

classification cs.LG
keywords multilevelMonteCarloactivelearningvote-by-committeeresourceadequacysurrogatemodellossofloadexpectationexpectedenergynotservedrandomforest
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multilevel Monte Carlo (MLMC) can make power-system reliability estimation cheaper by pairing a fast surrogate model with a small number of expensive exact simulations, but the time spent labeling and training the surrogate is real and has been overlooked in earlier comparisons. This paper argues that once that training time is counted, the choice of which scenarios to label becomes as important as how many are labeled. It introduces a training-time-adjusted speed metric for MLMC and a vote-by-committee active learning loop that labels daily margin traces where an ensemble of random forest predictors disagrees most. On a single-node resource adequacy case study, the active-learning surrogates achieve higher estimator speeds for both Loss of Load Expectation (LOLE) and Expected Energy Not Served (EENS) than random-sampled surrogates trained for a comparable amount of time.

What carries the argument

The central objects are the training-time-corrected MLMC speed metric $s = q^2 / \left(\sum_l \sigma_{Y_l}\sqrt{\tau_l}\right)^2$ and the vote-by-committee active learning loop built on random forests. The random forest acts as its own committee: for each candidate daily margin trace, the standard deviation of ENS predictions across its 100 decision trees is computed, and the 91 traces with the largest disagreement are labeled with the exact model and added to the training set. The speed metric converts training time into a first-class cost, allowing AL-trained and random-trained surrogates to be compared on equal footing under a fixed budget.

What would settle it

Run the same AL-trained surrogate MLMC procedure on a system where loss-of-load events are produced by rare multi-day ramps while the ENS disagreement query keeps selecting ordinary low-margin days; if the LOLE speed at comparable training time is no better than a random-surrogate baseline, the central assumption is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the effective performance of surrogate-based MLMC under a fixed total time budget is $1/c^2 = s\,(t - t_{\text{train}})$, so training time directly discounts the asymptotic MLMC speed $s$; therefore a surrogate that trains faster from fewer, better-chosen labels can beat a larger randomly trained surrogate. The paper shows this with vote-by-committee active learning: a random forest's disagreement, measured as the standard deviation of ENS predictions across its trees, selects the 91 most uncertain daily traces at each of up to 20 rounds. In the case study, 10 AL rounds produce LOLE speed 1.462 and EENS speed 0.526, compared with 1.158 and 0.473 for a randomly trained surrogate using 7300 days at comparable training time, and 20 AL rounds reach 2.009 and 0.891. MLMC remains unbiased because the exact model is used at the top level to correct whatever bias the daily-margin surrogate introduces.

Load-bearing premise

The method assumes that the daily scenarios where the surrogate ensemble most disagrees about energy-not-served are the same scenarios whose labels most reduce the error of the yearly loss-of-load estimate, so that expensive simulations are spent where they help the MLMC estimator most.

Editorial extensions

If this is right

  • Surrogate-based MLMC comparisons should be reported with the training-time-aware metric $1/c^2 = s(t - t_{\text{train}})$, because otherwise the apparent speedup from larger training sets can be illusory within a realistic budget.
  • For a target coefficient of variation below the break-even threshold, adding more active-learning rounds is computationally worthwhile, while beyond that threshold the extra training time is not repaid by faster simulation.
  • Active learning with vote-by-committee random forests yields higher surrogate accuracy and higher correlation with the exact model than random sampling at equal training time, which directly increases the covariance between adjacent MLMC levels and reduces estimator variance.
  • The daily-margin surrogate's aggregation error is corrected by the exact model inside the MLMC estimator, so users can safely substitute the fast surrogate for the expensive yearly simulation without introducing bias into the final estimate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the disagreement query is defined on ENS predictions, so in systems where loss-of-load events are rare and localized, the selected daily traces may not be the ones that most reduce the variance of the MLMC difference estimator; the speed advantage is therefore likely to be system-dependent.
  • Editorial inference: the same training-time-adjusted speed comparison could be applied to any learned lower-level model in MLMC, such as weather emulators or reduced-order physics models, not only power-system adequacy.
  • Editorial inference: a natural extension is to feed the exact-model evaluations produced during the MLMC run itself back into the training set, effectively recycling labels that the current algorithm discards.
  • Editorial inference: coupling active learning with importance sampling over the scenario distribution could focus labeling on the rare high-impact events that dominate LOLE and EENS variance, a direction the paper names but does not test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes combining multilevel Monte Carlo (MLMC) with active learning (AL) trained random-forest surrogate models for resource adequacy assessment (LOLE and EENS). The MLMC framework is standard; the paper's stated contribution is a speed metric that accounts for surrogate training time and a vote-by-committee active learning strategy to reduce the number of expensive exact-simulation labels. A single-node case study compares AL-trained surrogates with randomly trained surrogates at similar training times and reports that AL yields higher MLMC speed, e.g., LOLE speed 1.462 vs. 1.158 for AL-10 vs. random-7300. The authors provide code and data via Zenodo.

Significance. If the reported speed advantage is robust, the paper addresses a practical gap: surrogate-based MLMC for resource adequacy often ignores labeling cost, and active learning is a plausible way to reduce that cost. The derivation of the training-time correction in Eqs. (9)-(10) is coherent, and the empirical comparison is made against the exact simulator rather than against the surrogate itself. The break-even analysis is a useful practical tool. The main limitations are that the quantitative evidence rests on a single-node test system, the speed values in Table I are point estimates without uncertainty quantification, and the active learning query criterion is based on ENS variance while LOLE is also a target metric. These limitations currently prevent the abstract's general claim from being fully established.

major comments (3)
  1. [Section IV-B and Table I] The statement that 'All simulations were repeated for 10 times' does not specify whether the 10 repetitions cover the entire active-learning pipeline (initial set sampling, pool construction, query selection, retraining) or only the final MLMC evaluation with a fixed surrogate. Table I reports averaged LOLE/EENS values but gives no standard errors or confidence intervals for the speed values. The central comparison AL-10 vs. random-7300 (LOLE speed 1.462 vs. 1.158; EENS speed 0.526 vs. 0.473) is only 26% and 11% better, respectively, and without uncertainty quantification on these ratios the advantage may be within seed or Monte Carlo noise. Please clarify the replication structure and report standard errors or bootstrap intervals for the speed metrics.
  2. [Section V and abstract] The abstract claims that active learning in combination with MLMC can result in a 'substantial reduction variance' within a computational budget, but the evidence is limited to a single-node system with one storage dispatch policy and one set of AL hyperparameters (pool size 3650, batch size 91, 100 trees, rounds 5/10/20). The speed gain may depend on system size, dispatch complexity, or these hyperparameter choices. At minimum, the authors should add a second system configuration or a sensitivity analysis of AL parameters to show that the qualitative conclusion is not specific to the tested case.
  3. [Sections III and IV-B, Eq. (5)] The vote-by-committee criterion selects samples with high variance in the ENS random forest's decision-tree predictions, yet Table I reports speed improvements for both LOLE and EENS. The paper does not demonstrate that ENS-based disagreement is informative for LOL predictions. Because the MLMC variance reduction in Eq. (5) depends on the covariance between surrogate and exact outputs for each metric separately, the LOLE speed gain could be incidental to the ENS-based query. Please report how the ENS-variance query affects the LOLE surrogate's covariance, or compare with an LOL-variance query strategy.
minor comments (4)
  1. [Abstract] The phrase 'substantial reduction variance' should be 'substantial variance reduction' or 'substantial reduction in variance.'
  2. [Figures 2 and 3] The subfigures labeled (a) and (b) lack explicit axis labels and captions in the text; adding them would help readers verify that the RMSE and correlation are computed on the stated daily and yearly test sets.
  3. [Eqs. (7)-(9)] Equation (9) is an algebraic rearrangement of the definition of s and the identity 1/c^2 = q^2/σ^2_hat; presenting it as a 'training cost' subtraction may overstate its conceptual novelty, though the relative comparison in Eq. (10) is a useful practical reformulation.
  4. [Section IV-B] The text says 'green and orange boxes' in Fig. 1, but the figure appears to use colored outlines; please align the description with the actual figure rendering.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor definitional self-citation in the speed metric; main AL-vs-random claim is externally benchmarked and not circular.

  1. self definitional [Section II-C, Eq. (9), building on Eq. (8) from [11]]
    "Thus, the speed formulation in Eq. (7) is modified as: 1/c2 = q2/((t − ttrain) · σ2) * t − q2/((t − ttrain) · σ2) * ttrain = s · (t − ttrain) (9)"

    With t = tsim + ttrain, the bracketed coefficient is exactly the speed s defined in Eq. (8), which is taken from the authors' prior work [11]. Equation (9) is therefore a pure algebraic rearrangement of Eq. (7)/(8): it re-labels the previous speed metric and subtracts s·ttrain as a 'training cost'. The concluding statement that training time reduces effective speed is true by construction of the metric, not by empirical derivation. This is a minor, acknowledged adaptation of the authors' own definition, and it does not bear the main empirical comparison in Table I, which is benchmarked against the exact simulator.

full rationale

The central claim that active-learning-trained surrogates improve MLMC efficiency relative to randomly trained surrogates within a comparable training budget is supported by Table I, where LOLE and EENS estimates are produced by MLMC combining surrogate models with the exact simulation model. The benchmark is the exact simulator, not a target defined by the method, so the AL-vs-random comparison is self-contained and externally grounded. The only reduction by construction is the speed metric: Eq. (9) is algebraically identical to the speed definition from the authors' earlier work [11] after substituting tsim = t − ttrain, so the statement that training time lowers effective speed is definitional rather than a new empirical result. This is a minor self-citation that is not load-bearing for the central comparison. The vote-by-committee active-learning query and the daily-to-yearly aggregation in Eq. (12) are stated modeling assumptions evaluated on independent random test sets, not fitted outputs renamed as predictions. Concerns about missing error bars on the speed ratios in Table I are statistical robustness issues, not circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central empirical comparison rests on standard MLMC theory from [11] and on several user-chosen algorithm settings. No new physical entities or mediators are postulated. The main threat is the assumption that the ENS-variance query criterion transfers to MLMC variance reduction across both LOL and ENS estimators.

free parameters (5)
  • Initial labeled set size = 730 daily margin traces
    Chosen by hand to initialize the active learning loop; affects surrogate quality and training time, but the paper does not tune it.
  • Active learning pool size = 3650 daily margin traces per round
    Chosen by hand; larger pools would increase query selection overhead.
  • Query batch size = 91 daily margin traces per AL round
    Chosen by hand; determines how many exact simulations are requested per iteration.
  • Number of AL rounds = 5, 10, and 20 explored
    Stopping criterion for active learning; the paper scans these values rather than deriving an optimal stopping rule.
  • Random forest tree count = 100 decision trees
    Basic setting chosen without hyperparameter tuning; the paper notes that tuning could improve results.
assumptions (5)
  • standard math MLMC telescoping sum identity and variance allocation formulas (Eqs. 2-4) are valid.
    Taken from [11]; the paper relies on unbiasedness and the variance formula without re-deriving them.
  • domain assumption Yearly system behavior is well approximated by summing daily surrogate predictions.
    Eq. (12) aggregates 365 daily predictions; the footnote acknowledges daily-trace bias but assumes the MLMC correction handles it.
  • domain assumption Total time is the additive sum of training time and simulation time.
    Eq. (9) treats t = t_train + t_sim; this ignores parallel execution, shared I/O costs, and other interactions.
  • domain assumption Disagreement among random forest trees is a valid informativeness measure for reducing MLMC estimator variance.
    Section IV-B selects queries using ENS prediction standard deviation across trees; this proxy need not align with the variance of MLMC level differences.
  • domain assumption The single-node test system is representative of larger resource adequacy problems.
    The case study uses one system with 12 thermal generators and 27 storage units; generalization to large-scale systems is listed as future work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MLMC-based Resource Adequacy Assessment with Active Learning Trained Surrogate Models." pith.science (2026). https://pith.science/paper/WYKKVWBU

@misc{pith2026250520930,
  author       = {Pith},
  title        = {Pith review of: MLMC-based Resource Adequacy Assessment with Active Learning Trained Surrogate Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WYKKVWBU}},
  note         = {Machine review of arXiv:2505.20930}
}
read the original abstract

Multilevel Monte Carlo (MLMC) is a flexible and effective variance reduction technique for accelerating reliability assessments of complex power system. Recently, data-driven surrogate models have been proposed as lower-level models in the MLMC framework due to their high correlation and negligible execution time once trained. However, in resource adequacy assessments, pre-labeled datasets are typically unavailable. For large-scale systems, the efficiency gains from surrogate models are often offset by the substantial time required for labeling training data. Therefore, this paper introduces a speed metric that accounts for training time in evaluating MLMC efficiency. Considering the total time budget is limited, a vote-by-committee active learning approach is proposed to reduce the required labeling calls. A case study demonstrates that, within a given computational budget, active learning in combination with MLMC can result in a substantial reduction variance.

Figures

Figures reproduced from arXiv: 2505.20930 by the authors.

Figure 1
Figure 1. MLMC framework for resource adequacy assessment with AL trained surrogate models. The action flow is marked by black arrows; green and orange [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Performance of surrogate LOL and ENS estimators as function of [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance of surrogate LOL and ENS estimators as function of [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 17 canonical work pages

  1. [1]

    Uncertainty characterization for generation adequacy assessments – including an application to the recent european energy crisis,

    M. Spilger, D. Schneider, and C. Weber, “Uncertainty characterization for generation adequacy assessments – including an application to the recent european energy crisis,” Energy Economics , vol. 144, p. 108 304, Apr. 1, 2025

  2. [2]

    Clarifying the inter- pretation and use of the LOLE resource adequacy metric,

    G. Stephen, S. H. Tindemans, J. Fazio, et al. , “Clarifying the inter- pretation and use of the LOLE resource adequacy metric,” in 2022 17th International Conference on Probabilistic Methods Applied to Power Systems (PMAPS), Jun. 12, 2022, pp. 1–4

  3. [3]

    Minimizing unserved energy using heterogeneous storage units,

    M. P. Evans, S. H. Tindemans, and D. Angeli, “Minimizing unserved energy using heterogeneous storage units,” IEEE Transactions on Power Systems, vol. 34, no. 5, pp. 3647–3656, Sep. 2019

  4. [4]

    Assessment of the ca- pacity credit of renewables and storage in multi-area power systems,

    G. Liu, M. Vrakopoulou, and P. Mancarella, “Assessment of the ca- pacity credit of renewables and storage in multi-area power systems,” IEEE Transactions on Power Systems, vol. 36, no. 3, pp. 2334–2344, May 2021

  5. [5]

    Surrogate model uncertainty quantification for reliability-based design optimization,

    M. Li and Z. Wang, “Surrogate model uncertainty quantification for reliability-based design optimization,” Reliability Engineering & System Safety, Complex Systems RAMS Optimization: Methods and Applications, vol. 192, p. 106 432, Dec. 2019

  6. [6]

    An intelligent operational reliability assessment approach considering sample imbalance,

    M. Gao, J. Yu, S. Kamel, and Z. Yang, “An intelligent operational reliability assessment approach considering sample imbalance,” in 2023 IEEE 7th Conference on Energy Internet and Energy System Integration (EI2), Dec. 2023, pp. 4729–4734

  7. [7]

    Multilevel monte carlo for reliability theory,

    L. J. Aslett, T. Nagapetyan, and S. J. V ollmer, “Multilevel monte carlo for reliability theory,” Reliability Engineering & System Safety , vol. 165, pp. 188–196, Sep. 2017

  8. [8]

    Multilevel monte carlo with surrogate models for resource adequacy assessment,

    E. Sharifnia and S. H. Tindemans, “Multilevel monte carlo with surrogate models for resource adequacy assessment,” in 2022 17th International Conference on Probabilistic Methods Applied to Power Systems (PMAPS), Jun. 2022, pp. 1–6

Show all 17 references
  1. [9]

    Deep-learning-enhanced static risk- oriented security assessment under uncertainty,

    A. Masoumi and M. Korkali, “Deep-learning-enhanced static risk- oriented security assessment under uncertainty,” in 2024 IEEE Power & Energy Society General Meeting (PESGM) , Jul. 2024, pp. 1–5

  2. [10]

    Artificial intelligence and design of experiments for resource adequacy assessment in power systems,

    J. Priesmann, J. M ¨unch, M. Tillmanns, et al., “Artificial intelligence and design of experiments for resource adequacy assessment in power systems,” Energy Strategy Reviews, vol. 53, p. 101 368, May 2024

  3. [11]

    Accelerating system adequacy assess- ment using the multilevel monte carlo approach,

    S. Tindemans and G. Strbac, “Accelerating system adequacy assess- ment using the multilevel monte carlo approach,” Electric Power Systems Research, vol. 189, p. 106 740, Dec. 1, 2020

  4. [12]

    Surrogate model assisted multi- criteria operation evaluation of community integrated energy sys- tems,

    J. Jiang, H. Yu, G. Song, et al. , “Surrogate model assisted multi- criteria operation evaluation of community integrated energy sys- tems,” Sustainable Energy Technologies and Assessments , vol. 53, p. 102 656, Oct. 1, 2022

  5. [13]

    Active learning concerning sampling cost for enhancing AI-enabled building energy system modeling,

    A. Li, F. Xiao, Z. Xiao, et al., “Active learning concerning sampling cost for enhancing AI-enabled building energy system modeling,” Advances in Applied Energy , vol. 16, p. 100 189, Dec. 2024

  6. [14]

    Active learning literature survey,

    B. Settles, “Active learning literature survey,” University of Wisconsin-Madison Department of Computer Sciences, Technical Report, 2009

  7. [15]

    Building energy optimiza- tion using surrogate model and active sampling,

    K. Bamdad, M. E. Cholette, and J. Bell, “Building energy optimiza- tion using surrogate model and active sampling,” Journal of Building Performance Simulation, vol. 13, no. 6, pp. 760–776, Nov. 1, 2020

  8. [16]

    Active learning-based machine learning approach for enhancing environmental sustainability in green building energy con- sumption,

    S. Mahmood, H. Sun, A. Ali Alhussan, A. Iqbal, and E.-S. M. El-kenawy, “Active learning-based machine learning approach for enhancing environmental sustainability in green building energy con- sumption,” Scientific Reports, vol. 14, no. 1, p. 19 894, Aug. 27, 2024

  9. [17]

    Predicting energy consumption in multiple buildings using machine learning for improving energy efficiency and sustainability,

    A.-D. Pham, N.-T. Ngo, T. T. Ha Truong, N.-T. Huynh, and N.-S. Truong, “Predicting energy consumption in multiple buildings using machine learning for improving energy efficiency and sustainability,” Journal of Cleaner Production , vol. 260, p. 121 082, Jul. 1, 2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.