Ranking probabilistic forecasting models with different loss functions

Bartosz Uniejewski; Tomasz Serafin

REVIEW 3 major objections 6 minor 20 references

Ranking probabilistic forecasting models with different loss functions

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Ranking forecast models by the coverage of their traded quantiles maximizes battery-storage profit.

desk verdict A plausible but under-supported empirical claim: trading-hour coverage ranks probabilistic price forecasts better than global pinball, but a window-length contradiction and missing uncertainty metrics need fixing before the result is trustworthy. read the letter →

arxiv 2411.17743 v1 pith:EBCXWUJE submitted 2024-11-24 econ.EM

Tomasz Serafin , Bartosz Uniejewski This is my paper · ORCID

classification econ.EM MSC 62M2091B84

keywords probabilisticforecastingelectricitypricepinballlossempiricalcoveragequantileforecastsmodelrankingbatterystoragetradingday-aheadmarket

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks which way of scoring probabilistic electricity-price forecasts, when used to pick the best forecasting model, leads to the largest profits from a battery-storage trading strategy. It introduces six ranking metrics built from pinball loss and empirical coverage, then tests them on German day-ahead prices from 2015 to 2023. The central finding is that ranking models by the coverage of the quantile forecasts that are actually used in the trading hours—the bid at the low-price hour and the offer at the high-price hour—beats all other metrics across nearly every prediction-interval level. This matters because it connects statistical forecast evaluation directly to economic decisions: the metric that rewards the right behavior in the trading strategy is the one that earns the most money.

What carries the argument

The key machinery is a set of six statistical performance metrics derived from pinball loss and empirical coverage, applied in a 30-day rolling window to select the best of nine probabilistic forecasting models for a battery-storage trading strategy. The strategy buys 1 MWh at the hour with the lowest predicted median price and sells at the highest, with bid and offer prices set at the $\alpha$-level prediction interval bounds, using a 2.5 MWh battery with 90% round-trip efficiency. The decisive metric, 'SP Coverage hours', checks exactly what the trader cares about: for each day, whether the realized price is below the upper quantile of the prediction interval at the selected buying hour and above the lower quantile at the selling hour. By ranking models according to this targeted coverage instead of coverage over all hours or average pinball across all quantiles, the selection process rewards models whose intervals are reliable precisely where the trading decision is made.

What would settle it

Resample the out-of-sample period with bootstrap and recompute the average per-trade profit for each ranking metric many times; if SP Coverage hours is not the top earner in the large majority of resamples, or if its margin over the runner-up overlaps with zero, then the 'clear winner' claim fails. Alternatively, repeating the comparison on a different day-ahead market and finding that another metric wins would refute the claimed superiority.

Watch

Extended reading notes

Core claim

The paper's main claim is that, for the battery-storage trading strategy considered, ranking probabilistic forecasting models by 'SP Coverage hours'—an empirical-coverage metric that scores a day as a hit only when the realized price falls below the bid quantile at the buying hour and above the offer quantile at the selling hour—delivers the highest average profit per trade out-of-sample. This result holds across almost all values of the prediction-interval parameter $\alpha$ from 50% to 98%. The paper further reports that all pinball-loss-based metrics perform similarly to each other, with average pinball across all hours and quantiles doing best among them, and that coverage averaged over all 24 hours is more variable. The conclusion is that the statistical metric most aligned with the economic objective—covering the quantiles that are actually used for bids and offers—is the one that identifies the best forecasting model.

Load-bearing premise

The paper assumes that a single out-of-sample path—German day-ahead prices from 2015 to 2023, one battery-storage trading strategy, and one set of nine forecast models—is enough to establish which ranking metric is best, and reports the winning metric without confidence intervals or significance tests.

Editorial extensions

If this is right

A market participant using probabilistic day-ahead forecasts could switch from pinball-based model selection to trading-hour coverage and expect higher per-trade profits with no change to the forecasting models or the trading strategy.
Among pinball-loss-based metrics, the choice is less consequential, since they deliver similar profits for most prediction-interval levels.
Coverage averaged over all hours is a workable but more volatile criterion, which suggests that the information contained in the specific trading hours drives the economic gain.
The ranking metric can be computed from the same quantile forecasts already used for bidding, so the approach is operationally cheap to implement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An extension the paper leaves implicit is to test the same ranking metrics on other day-ahead markets (for example, Nordic or North American) to see whether the trading-hour coverage advantage is a stable property or an artifact of German price dynamics.
A bootstrap or subsample analysis of Figure 5.1 would tell whether the gap between SP Coverage hours and the next-best metric is statistically meaningful; the paper reports no uncertainty quantification.
The principle behind the winning metric generalizes naturally to any quantile-based trading rule: evaluate forecasts on the quantiles actually used in the decision, not on the full predictive distribution.
A separate testable extension is to vary the battery size, efficiency, or the choice of trading hours; if the advantage of trading-hour coverage depends on those design choices, the paper's practical recommendation becomes conditional.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper. First, it does something genuinely useful: it compares statistical ranking metrics for probabilistic electricity price forecasts against the economic outcome (battery-storage trading profit) in a genuine out-of-sample rolling design. The SP Coverage hours metric is new and sensible—it checks whether the quantiles used for the actual bid and offer cover the realized prices. Second, the headline claim that this metric 'exhibits superior economic performance' is plausible but not yet proven, because the paper contains a load-bearing internal inconsistency and no statistical uncertainty quantification.

The experimental setup is a real strength. Nine forecasting models, a full year-by-year rolling calibration, and a realistic BESS trading strategy with battery state constraints. The out-of-sample period runs 2015–2023 on German day-ahead prices. That is a fair test bed, and the economic evaluation is not circular: profit is computed from actual accepted bids, and the ranking metrics are computed independently.

The soft spots are concrete. The most serious is the window-length contradiction. Section 4.1 says a 30-day rolling window is used to compute the statistical-performance metrics, and the equation sums from d=i to i+30. Section 4.2 says the third rolling window is 182 days. Figure 4.1's caption says 30 days. The manuscript never resolves this. It is not cosmetic: the window length determines which model gets selected each day, and different window lengths could rank the nine models differently. The 'clear winner' in Figure 5.1 is the output of one undisclosed choice. No sensitivity analysis over the window length is reported. This must be settled before the central claim can be evaluated.

Second, Figure 5.1 shows average profit differences across metrics and α values with no confidence intervals, significance tests, or multiple-comparison control. With a single out-of-sample path, those gaps could be noise. The stress-test note is right on both counts.

Third, the unlimited-bids benchmark is described in Section 3.2.1 but never shown in the results. It would anchor the economic value of the quantile-based strategy. Minor: several citations are missing placeholders (e.g., '?'), which is sloppy but fixable.

Who is this for? Researchers working on probabilistic electricity price forecasting, model ranking, and trading applications. It is a modest extension of Uniejewski (2024), not a breakthrough. If the window-length issue is resolved and uncertainty is addressed, the paper would be a solid applied contribution. As is, I would not take the headline result at face value, but it deserves a serious referee rather than a desk reject. Send it to review, and ask the authors to fix the inconsistency and add significance checks.

Referee Report

3 major / 6 minor

Summary. The paper compares statistical performance metrics for ranking probabilistic electricity price forecasting models, with the goal of identifying which ranking metric leads to the highest economic profit in a battery-storage trading strategy. Nine probabilistic forecasting models (built on a common point forecast) are ranked by six metrics derived from pinball loss and empirical coverage, including a metric computed only at the trading hours. A rolling out-of-sample selection exercise on German day-ahead prices (2015–2023) is used: each metric selects one model per day, and the selected model's forecasts drive the battery bid/offer decisions. Average per-trade profit is then compared across metrics for prediction-interval levels α = 50%–98%. The central claim is that ranking models by the coverage of quantile forecasts at the trading hours ('SP Coverage hours') yields the highest average profit.

Significance. If the central claim is robust, the paper makes a useful empirical contribution to the literature linking statistical forecast evaluation and economic value in electricity trading. The experimental design is a genuine rolling out-of-sample model selection exercise, rather than an in-sample fit, and the economic evaluation is transparently described. The comparison of several candidate metrics on a common set of forecasting models is a sensible way to study metric choice. However, the strength of the headline finding is currently difficult to assess because of an unresolved internal inconsistency in the rolling-window length, the absence of any uncertainty quantification for the profit differences in Figure 5.1, and the fact that the 'SP Coverage hours' metric is constructed almost exactly from the acceptance conditions that generate profit. The paper therefore needs additional work before its main conclusion can be accepted as stated.

major comments (3)

[§4.1 vs §4.2 and Figure 4.1] The rolling-window length for computing the statistical performance metrics is stated inconsistently: §4.1 writes 'a 30-day rolling window' and the displayed sum runs from d=i to i+30, and the caption of Figure 4.1 says the third window is 30 days, while §4.2 states that the third rolling window for statistical performance is 182 days. This is not a purely cosmetic discrepancy, because the window length determines which forecasting model is selected each day, and the resulting ranking of metrics in Figure 5.1 depends on this choice. The authors must (i) state which length was actually used, (ii) correct the conflicting statements, and (iii) provide at least a sensitivity analysis over plausible window lengths, since the 'clear winner' status of SP Coverage hours may be specific to the reported implementation.
[§5, Figure 5.1] The central comparison is presented without confidence intervals, significance tests, or any form of multiple-comparison control across the many values of α examined. The data are a single out-of-sample path (one market, one trading strategy, one calendar period), so the profit differences between metrics could be due to sampling variation. The authors should report uncertainty (e.g., block bootstrap or subperiod analysis) and, at minimum, discuss the magnitude of the profit gaps relative to their variability. Without this, the claim that SP Coverage hours 'exhibits a superior economic performance' is not statistically supported.
[§4.1 and §3.2.1] The SP Coverage hours metric is defined as the joint event that the bid at hour h1 is accepted (actual price below the upper PI bound) and the offer at hour h2 is accepted (actual price above the lower PI bound). These are exactly the acceptance conditions in step (3) of the trading strategy that generate the profitable round-trip trade. Consequently, ranking models by historical SP Coverage hours is directly aligned with the frequency of profitable order pairs, and the 'superior economic performance' of this metric is at least partly mechanical rather than a finding about the generic value of coverage-based evaluation. The paper should explicitly discuss this endogeneity between the metric definition and the trading strategy, and should show whether the conclusion survives when the metric is not engineered to the strategy's acceptance conditions.

minor comments (6)

[§4.1, displayed equation for SP Coverage hours] The second condition in the SP Coverage hours definition appears to have the wrong hour subscript: the text reads 'P_{d,h2} > \hat P^{(1−α)/2}_{d,h1}', which should probably be '\hat P^{(1−α)/2}_{d,h2}'.
[§3.2.1, step (2)] The sentence 'The prices \hat U^α_{d,h1} and \hat L^α_{d,h1} are marked with black dots' should refer to \hat L^α_{d,h2} rather than \hat L^α_{d,h1}.
[§2.2.2] The conformal prediction formula has a stray bracket after 'for q > 50%]'.
[§2.2.2 and §2.2.3] Two citations appear as unresolved '?' placeholders: the calibration-window reference in Conformal Prediction and the literature reference for Johnson's distribution. These should be completed.
[§5] The word 'avarage' should be 'average'.
[§3.2.1 and §5] The 'unlimited-bids benchmark' is described in Section 3.2.1 but never reported in the Results. Even if the paper's ranking claim is about relative performance across metrics, the benchmark is needed to position the economic magnitudes and to assess whether any statistical metric adds value over a naive price-taking strategy.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity is present: the metric-ranking comparison is out-of-sample, and the SP Coverage hours metric is defined from price–quantile comparisons rather than from realized profit.

full rationale

The paper's central claim is an empirical, out-of-sample comparison. For each day, statistical metrics (pinball loss components and empirical coverage variants) are computed over a rolling window; the top-ranked model's forecasts are then used for next-day battery trading, and economic performance is measured by realized per-trade profit. The headline metric SP Coverage hours is defined in Section 4.1 as an indicator that the actual price lies inside the bid/offer quantiles at the two trading hours; this is structurally aligned with the both-orders-accepted event in the Section 3.2.1 strategy, but it is not a function of realized profit, and a model with high coverage can still earn low profit if price spreads are small or only one leg clears. The result is therefore not forced by definition. The self-citations to Uniejewski (2024) supply the SQR estimator and the trading strategy, but both are fully specified in Sections 2.2.5 and 3.2.1, so the citations are not load-bearing, and no uniqueness claim is imported. No fitted parameter is relabeled as a prediction. The manuscript contains an unresolved internal inconsistency (Section 4.1 states a 30-day rolling window while Section 4.2 states 182 days for the same metric-calculation window) and several missing citation placeholders; these are reproducibility and completeness concerns, not circularity, and they do not raise the circularity score.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central comparison rests on several hand-chosen design parameters (alpha, rolling window, calibration lengths, battery parameters) and domain assumptions about the representativeness of one market and the realism of the trading simulation. No new physical entities are introduced; the new metrics are scoring functions, not entities.

free parameters (4)

Prediction interval level alpha = 50% to 98% in steps
The trading strategy's bid and offer prices are set at quantiles (1±alpha)/2; the comparison across metrics is made for each alpha, and results vary with alpha. Chosen by hand as a robustness scan.
Statistical performance rolling window length = 30 days (Section 4.1) or 182 days (Section 4.2, inconsistent)
The model ranking uses the average of the metric over the last 30 days (as stated in Section 4.1 and Figure 4.1); Section 4.2 contradicts this with 182 days. The ranking and resulting profits depend on this window.
Calibration window lengths = 364 days for point forecasts, 182 days for probabilistic forecasts
These windows are chosen by hand to fit the dataset; results may depend on them.
Battery parameters = 2.5 MWh capacity, 90% round-trip efficiency, 1 MWh transaction limit
Inherited from Uniejewski (2024) and fixed across all simulations; not fitted here but central to the economic evaluation.

assumptions (5)

standard math The point forecast model errors epsilon_d,h are iid (Section 2.1).
Used to justify quantile estimates from historical errors; standard but unverified.
domain assumption The German day-ahead market is a representative environment for comparing forecast evaluation metrics.
All conclusions are drawn from a single market; Section 4.2 describes the dataset but gives no justification of generalizability.
domain assumption The trading strategy's battery state dynamics and bid/offer acceptance rules correctly model a real BESS operator.
Section 3.2.1 defines the strategy; the profit calculation depends on this model being realistic.
domain assumption A 30-day (or 182-day) rolling average of each metric is a sufficient statistic for selecting the best forecasting model for the next day.
Section 4.1 makes this selection rule; no sensitivity analysis for the window length is reported.
domain assumption No transaction costs beyond round-trip efficiency.
Trading profit is computed gross of fees; real markets may erode the advantage.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Ranking probabilistic forecasting models with different loss functions." pith.science (2026). https://pith.science/paper/EBCXWUJE

@misc{pith2026241117743,
  author       = {Pith},
  title        = {Pith review of: Ranking probabilistic forecasting models with different loss functions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EBCXWUJE}},
  note         = {Machine review of arXiv:2411.17743}
}

read the original abstract

In this study, we introduced various statistical performance metrics, based on the pinball loss and the empirical coverage, for the ranking of probabilistic forecasting models. We tested the ability of the proposed metrics to determine the top performing forecasting model and investigated the use of which metric corresponds to the highest average per-trade profit in the out-of-sample period. Our findings show that for the considered trading strategy, ranking the forecasting models according to the coverage of quantile forecasts used in the trading hours exhibits a superior economic performance.

Figures

Figures reproduced from arXiv: 2411.17743 by the authors.

**Figure 3.1.** Illustration of the quantile-based bidding str [PITH_FULL_IMAGE:figures/full_fig_p012_3_1.png] view at source ↗

**Figure 4.1.** Dataset from the German day-ahead electricity m [PITH_FULL_IMAGE:figures/full_fig_p016_4_1.png] view at source ↗

**Figure 5.** shows the average profits from trading based on the [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗

Figures from the paper (1 more)

**Figure 5.1.** Figure 5.1: Comparison of average profits from the trading st [PITH_FULL_IMAGE:figures/full_fig_p018_5_1.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 16 canonical work pages

[1]

, year 1993

author Chatfield, C. , year 1993 . title Calculating interval forecasts . journal Journal of Business & Economic Statistics volume 11 , pages 121--135

work page 1993
[2]

, author Guerre, E

author Fernandes, M. , author Guerre, E. , author Horta, E. , year 2021 . title Smoothing quantile regressions . journal Journal of Business & Economic Statistics volume 39 , pages 338--357

2021
[3]

, author Raftery, A

author Gneiting, T. , author Raftery, A. , year 2007 . title Strictly proper scoring rules, prediction, and estimation . journal Journal of the American Statistical Association volume 102 , pages 359--378

2007
[4]

, author Lichtendahl Jr, K.C

author Grushka-Cockayne, Y. , author Lichtendahl Jr, K.C. , author Jose, V.R.R. , author Winkler, R.L. , year 2017 . title Quantile evaluation, sensitivity to bracketing, and sharing business payoffs . journal Operations Research volume 65 , pages 712--728

work page 2017
[5]

, author Pinson, P

author Hong, T. , author Pinson, P. , author Fan, S. , author Zareipour, H. , author Troccoli, A. , author Hyndman, R.J. , year 2016 . title Probabilistic energy forecasting: G lobal E nergy F orecasting C ompetition 2014 and beyond . journal International Journal of Forecasting volume 32 , pages 896--913

work page 2016
[6]

, year 1949

author Johnson, N.L. , year 1949 . title Systems of frequency curves generated by methods of translation . journal Biometrika volume 36 , pages 149--176

work page 1949
[7]

, author Ziel, F

author Kath, C. , author Ziel, F. , year 2018 . title The value of forecasts: Quantifying the economic gains of accurate quarter-hourly electricity price forecasts . journal Energy Economics volume 76 , pages 411--423

work page 2018
[8]

, author Ziel, F

author Kath, C. , author Ziel, F. , year 2021 . title Conformal prediction interval estimation and applications to day-ahead and intraday power markets . journal International Journal of Forecasting volume 37 , pages 777--799

work page 2021

Show all 20 references

[9]

, year 2005

author Koenker, R.W. , year 2005 . title Quantile Regression . publisher Cambridge University Press

2005
[10]

, author Serafin, T

author Maciejowska, K. , author Serafin, T. , author Uniejewski, B. , year 2024 . title Probabilistic forecasting with a hybrid factor-qra approach: Application to electricity trading . journal Electric Power Systems Research volume 234 , pages 110541 . :https://doi.org/10.101...

2024
[11]

, author Uniejewski, B

author Maciejowska, K. , author Uniejewski, B. , author Weron, R. , year 2023 . title Forecasting electricity prices , in: booktitle Oxford Research Encyclopedia of Economics and Finance . publisher Oxford University Press . :10.1093/acrefore/9780190625979.013.667

2023
[12]

, author Narajewski, M

author Marcjasz, G. , author Narajewski, M. , author Weron, R. , author Ziel, F. , year 2023 . title Distributional neural networks for electricity price forecasting . journal Energy Economics volume 125 , pages 106843

2023
[13]

, author Uniejewski, B

author Marcjasz, G. , author Uniejewski, B. , author Weron, R. , year 2020 . title Probabilistic electricity price forecasting with NARX networks: Combine point or probabilistic forecasts? journal International Journal of Forecasting volume 36 , pages 466--479

2020
[14]

, author Weron, R

author Nowotarski, J. , author Weron, R. , year 2015 . title Computing electricity spot price prediction intervals using quantile regression and forecast averaging . journal Computational Statistics volume 30 , pages 791--803

2015
[15]

, author Weron, R

author Serafin, T. , author Weron, R. , year 2024 . title Loss functions in regression models: Impact on profits and risk in day-ahead electricity trading . journal WORking papers in Management Science (WORMS), WORMS/24/03, volume Department of Operations Research and Business...

2024
[16]

, year 2024

author Uniejewski, B. , year 2024 . title Smoothing quantile regression averaging: A new approach to probabilistic forecasting of electricity prices . https://arxiv.org/abs/2302.00411, http://arxiv.org/abs/2302.00411 arXiv:2302.00411

2024 arXiv
[17]

, author Weron, R

author Uniejewski, B. , author Weron, R. , year 2021 . title Regularized quantile regression averaging for probabilistic electricity price forecasting . journal Energy Economics volume 95 , pages 105121

2021
[18]

, author Zhang, N

author Wang, Y. , author Zhang, N. , author Tan, Y. , author Hong, T. , author Kirschen, D. , author Kang, C. , year 2019 . title Combining probabilistic load forecasts . journal IEEE Transactions on Smart Grid volume 10 , pages 3664--3674

2019
[19]

, author Petropoulos, F

author Yardley, E. , author Petropoulos, F. , year 2021 . title Beyond error measures to the utility and cost of the forecasts . journal Foresight volume Q4 , pages 36--45

2021
[20]

, author Weron, R

author Ziel, F. , author Weron, R. , year 2018 . title Day-ahead electricity price forecasting with high-dimensional structures: U nivariate vs. multivariate modeling frameworks . journal Energy Economics volume 70 , pages 396--420

2018

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

, year 1993

author Chatfield, C. , year 1993 . title Calculating interval forecasts . journal Journal of Business & Economic Statistics volume 11 , pages 121--135

work page 1993

[2] [2]

, author Guerre, E

author Fernandes, M. , author Guerre, E. , author Horta, E. , year 2021 . title Smoothing quantile regressions . journal Journal of Business & Economic Statistics volume 39 , pages 338--357

2021

[3] [3]

, author Raftery, A

author Gneiting, T. , author Raftery, A. , year 2007 . title Strictly proper scoring rules, prediction, and estimation . journal Journal of the American Statistical Association volume 102 , pages 359--378

2007

[4] [4]

, author Lichtendahl Jr, K.C

author Grushka-Cockayne, Y. , author Lichtendahl Jr, K.C. , author Jose, V.R.R. , author Winkler, R.L. , year 2017 . title Quantile evaluation, sensitivity to bracketing, and sharing business payoffs . journal Operations Research volume 65 , pages 712--728

work page 2017

[5] [5]

, author Pinson, P

author Hong, T. , author Pinson, P. , author Fan, S. , author Zareipour, H. , author Troccoli, A. , author Hyndman, R.J. , year 2016 . title Probabilistic energy forecasting: G lobal E nergy F orecasting C ompetition 2014 and beyond . journal International Journal of Forecasting volume 32 , pages 896--913

work page 2016

[6] [6]

, year 1949

author Johnson, N.L. , year 1949 . title Systems of frequency curves generated by methods of translation . journal Biometrika volume 36 , pages 149--176

work page 1949

[7] [7]

, author Ziel, F

author Kath, C. , author Ziel, F. , year 2018 . title The value of forecasts: Quantifying the economic gains of accurate quarter-hourly electricity price forecasts . journal Energy Economics volume 76 , pages 411--423

work page 2018

[8] [8]

, author Ziel, F

author Kath, C. , author Ziel, F. , year 2021 . title Conformal prediction interval estimation and applications to day-ahead and intraday power markets . journal International Journal of Forecasting volume 37 , pages 777--799

work page 2021