An Interval-Score ROC Curve for Assessment, Calibration and Ensembling of Probabilistic Forecasts

Flavio Bobba; Giuseppe De Nicolao; Marco Capelletti; Simone Milanesi

REVIEW 3 major objections 6 minor 13 references

An Interval-Score ROC Curve for Assessment, Calibration and Ensembling of Probabilistic Forecasts

T0 review · 3 major / 6 minor · reviewed 2026-07-31 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The true data-generating process draws a Pareto-optimal, convex curve of interval sharpness versus error, and that geometry alone calibrates and ensembles probabilistic forecasts.

desk verdict Solid geometric toolkit for interval-score trade-offs; math core is clean, hull-as-ensemble is the soft spot. read the letter →

arxiv 2607.28178 v1 pith:GF4G35RI submitted 2026-07-30 stat.ME

Simone Milanesi , Marco Capelletti , Flavio Bobba , Giuseppe De Nicolao This is my paper · ORCID

classification stat.ME MSC 62M2062C0590C29

keywords probabilisticforecastingintervalscoreROCcurvecalibrationParetooptimalityconvexhullensemblingsharpnessproperscoringrules

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Probabilistic forecasts are usually ranked by a single proper score, which hides the trade-off between how tight the intervals are and how often they miss. This paper replaces that scalar with the IS–ROC curve: the path traced in the plane of mean interval width versus mean absolute miss distance as you vary how tight the prediction is. It proves that the curve belonging to the true data-generating process is both Pareto optimal and convex, so nothing can sit strictly better than it. From that geometry the authors build a tangent-based calibration map and a convex-hull ensemble that stitches competing forecasters into one efficient frontier. A reader who cares about forecast comparison, recalibration, or model combination gets a single visual object that does all three without collapsing the trade-off first.

What carries the argument

The IS–ROC curve: the parametric plot (mean sharpness(β), mean absolute distance(β)) obtained by sweeping the tightness parameter β of a tunable interval predictor. Its geometry carries Pareto comparison, tangent calibration, and convex-hull ensembling.

What would settle it

Construct a data-generating process and a competing tunable interval predictor whose empirical (mean sharpness, mean absolute distance) points lie strictly below and left of the oracle IS–ROC curve on a large sample, or show that no mixing of the segment endpoints can achieve the intermediate performance claimed by the convexification lemma.

Watch

Extended reading notes

Core claim

The Interval-Score ROC (IS–ROC) curve of the data-generating process is the Pareto frontier in the mean-sharpness / mean-absolute-distance plane and is always convex. No other tunable interval predictor can place an operating point strictly southwest of it, and every point on a convexification segment is realizable by a randomized mix of the endpoint predictors. That geometric fact turns calibration into finding the supporting line of slope −α/2 and turns ensembling into taking the lower convex hull of competing curves.

Load-bearing premise

That randomly mixing two fixed interval predictors is an acceptable way to build a real ensemble point on a hull segment, even when the resulting intervals may need an extra fix to stay nested across tightness levels.

Editorial extensions

If this is right

Forecasts can be ranked and diagnosed by whether their IS–ROC curves dominate or cross, without first choosing a single coverage level or scalar score.
Any dominant convex IS–ROC curve can be calibrated by reading off the tangency point for each Interval Score slope −α/2, recovering an adjusted tightness map g(α).
Intersecting or non-convex curves can be replaced by their lower convex hull, yielding an ensemble whose operating points are either original or randomized mixes of endpoints.
Calibration leaves the curve’s shape unchanged and only reparameterizes β, so two predictors that share an IS–ROC curve are equivalent up to that map.
The same workflow extends to building progressively richer predictors: start covariate-free, add covariates, recalibrate, then convexify into an ensemble.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Conditional IS–ROC curves (one per covariate stratum) could let the same geometry drive regime-specific calibration and ensembling before the pieces are recombined.
Because the oracle curve is convex and Pareto, any systematic non-convexity or interior location of a fitted curve is a direct visual signature of misspecification or of unused covariates.
The step-calibration that appears on hull segments implies flat stretches in the recovered predictive CDF; that may be a useful diagnostic for when the ensemble is only interpolating rather than learning new structure.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing worth knowing: this is a real methods contribution, not a rebrand. They take the Interval Score, open it into the (mean sharpness, mean absolute distance) plane over a tightness parameter, and show that the oracle curve is Pareto optimal and convex. That geometry then drives tangent calibration and convex-hull comparison/ensembling. The ROC analogy is earned, not decorative.

What works. Theorems 2.9–2.12 sit on standard propriety of the interval/pinball score plus a clean randomization lemma for segments. The i.i.d. derivative proof (dMAD/dS = −β/2) checks out. Non-uniqueness for symmetric same-median laws is stated honestly, with the parallel to ordinary ROC. The user guide and the two synthetic examples make the workflow usable: dominance, overlap, intersection, step calibration on flat hull pieces. Citation pattern is appropriate (Gneiting–Raftery, Gneiting–Balabdaoui–Raftery, Christoffersen, Askanazi et al., classical ROC). Circularity is low; they are not fitting the frontier to the demos.

Soft spots, in proportion. Validation is synthetic only and there is no code. More load-bearing: Lemma A.1 gives a Bernoulli mix for a single operating point. Turning the full convex hull into a nested Tunable Interval Predictor across tightness is not fully specified. The authors flag that convexification can break nesting and wave at isotonic repair without developing it. Step calibration leaves g undefined on flat segments. So the hull is a strong diagnostic envelope and a pointwise IS selector; calling it a deployable nested ensemble is a step ahead of the write-up. That is a gap, not a collapse of the core claims.

Who it is for: people who already live in proper scoring rules and want a graphical post-processing layer for comparison, recalibration, and light model combination. Subfield tooling with practice value, not a field reorg. I would send it to peer review. A serious referee should push on nesting/operability of hull ensembles and ask for at least one real-data demo with shared code. Engage if you care about forecast evaluation geometry; skip if you only want new scoring rules or large empirical bake-offs.

Referee Report

3 major / 6 minor

Summary. The paper introduces the Interval-Score ROC (IS–ROC) curve: the parametric path of a Tunable Interval Predictor (TIP) in the (mean sharpness, mean absolute distance) plane as tightness β varies. It proves that the curve induced by the data-generating process is Pareto optimal (Thm 2.10, via Interval Score propriety) and convex (Thm 2.11 for the covariate-free smooth case; Thm 2.12 in general via a randomization/convexification lemma). From this geometry it derives a tangent-based calibration map g(α) that chooses the IS-optimal tightness for each nominal level α, a convex-hull procedure for non-convex or intersecting curves, and an ensemble strategy that selects or mixes operating points on the global lower envelope. A practical workflow and two synthetic examples illustrate dominance, non-uniqueness for symmetric equal-median laws, recalibration, and hull-based ensembling.

Significance. If the claims hold operationally, the work supplies a genuine multi-objective diagnostic that sits between scalar proper scores (CRPS/WIS/IS) and binary interval tests (Christoffersen), with a clean geometric link from Interval Score level sets to calibration and model combination. The Pareto and convexity theorems are load-bearing and appear correctly derived from standard propriety plus elementary expectation identities; the explicit derivative chain dMAD/dS = −β/2 in the smooth covariate-free case is a useful closed-form characterization. The ROC analogy is apt and the tangent calibration idea is natural. Credit is due for a coherent geometric program and for stating non-uniqueness (Thm 2.13) rather than overclaiming identification. The main open value is whether convex-hull points are constructible as nested TIPs in deployment, not only as a diagnostic envelope.

major comments (3)

[§3.4, Lemma A.1, Def. 2.1] §3.4 and Lemma A.1: The ensemble claim is only partially supported. Lemma A.1 constructs a single operating point by Bernoulli mixing of two fixed intervals. A TIP (Def. 2.1) requires a nested family across all β. The manuscript notes that convexification can break nesting and only gestures at an undeveloped isotonic repair. Without a specified, validated construction that restores nesting (or a clear restriction that the hull is a diagnostic envelope and a pointwise IS selector, not a deployable nested forecaster), the claim of “ensemble forecasting through convexification” overreaches the delivered method.
[§3.3, Eq. (1)–(2)] §3.3 step calibration: On linear hull segments the map g is left undefined at the segment slope α̂ and becomes a step elsewhere. When the calibrated curve is turned into a predictive distribution via (2), flat segments become atoms/plateaus. The paper should state precisely what probabilistic object is returned for α = α̂ (e.g., any endpoint, a prescribed mixture, or refusal to output a unique law) and whether the resulting object remains a valid, nested TIP. As written, “geometry directly yields a calibrated probabilistic ensemble” is not fully specified on the segments that the method itself introduces.
[§5] §5: Evidence is limited to two synthetic, low-dimensional examples. The central applied claims—comparison, calibration recovery (Ex. 1), and hull ensembling (Ex. 2)—need at least one real forecasting task (e.g., energy, epidemiology, or finance as motivated in §1.1) with proper scoring comparison against CRPS/WIS baselines and a check that any nesting repair does not degrade the envelope. Without that, the operational advantage over existing tools remains conjectural.

minor comments (6)

[§2.2] Terminology drifts between PIT, TIP, and “PIT” in places (e.g., end of §2.2: “achievable by the PIT”). Standardize on TIP for the predictor and reserve PIT for the probability integral transform discussed in §1.3.
[Figure 1] Figure 1 is described in text but the three schematic panels are not fully self-explanatory in the manuscript copy; add axis labels and a one-line caption distinguishing dominant-convex / dominant-nonconvex / no-dominant cases.
[Remark 2.14] Remark 2.14’s triangular exclusion region is useful; a short proof or pointer that the right derivative at s=0 equals −1/2 for the oracle would help readers.
[Appendix B.5] Appendix B.5 tables are helpful; clarify whether MAD expressions are population quantities under Y∼F or under the DGP G when F≠G (calibration examples mix both).
[§1.5] Related work: briefly position against quantile calibration / isotonic distributional calibration literature beyond Gneiting–Ranjan pooling, since Eq. (2) is a form of quantile-level recalibration.
Minor typos: “possibily”, “acovariate-free”, “IS—ROC” vs “IS–ROC” hyphenation inconsistency, “Forthecomputation” spacing in §4.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: oracle Pareto/convexity and calibration follow from Interval Score propriety and elementary identities, not from self-fitting or self-citation.

full rationale

The load-bearing chain is external and definitional in the honest sense, not circular. Theorem 2.9 invokes the known strict propriety of the Interval Score (pinball decomposition; Gneiting & Raftery), which is independent of this paper. Theorem 2.10 is a short contradiction: any pointwise southwest improvement in (MS, MAD) would strictly improve IS at the matching α, contradicting propriety. Convexity (Thm 2.11 by differentiation under regularity; Thm 2.12 by contradiction via Lemma A.1’s Bernoulli mixture of two fixed interval predictors) uses only expectation linearity and the already-established Pareto property of the DGP curve. Calibration g(α) := arg min_β IS(α,β) and the tangent construction are operational definitions of an optimal tightness map on a given curve, not quantities fitted to data and then re-presented as predictions. Convex-hull ensembling likewise constructs admissible operating points from mixtures (Lemma A.1); it does not smuggle a target frontier back into the premises. Citations are to standard external literature (Gneiting, Fawcett, Christoffersen, etc.); there is no load-bearing self-citation, uniqueness theorem imported from the authors, or ansatz-via-prior-work. Synthetic examples illustrate the geometry rather than tune free constants that redefine the claim. Residual concerns (nesting repair after convexification, step calibration on flat segments) are constructibility/operability gaps, not circular reductions of the claimed theorems to their inputs.

Assumptions & free parameters 0 free parameters · 5 assumptions · 2 invented entities

The load-bearing content is standard probability and proper-scoring-rule theory plus the paper’s definitional framing of tunable central intervals. No physical constants or data-fitted scales underwrite the theorems. The main modeling commitments are central (median-centered) intervals, the MS–MAD pair as the two axes, and admissibility of randomized mixtures for hull segments.

assumptions (5)

domain assumption The Interval Score is a proper scoring rule; the oracle attains the minimal IS among all interval forecasts at each nominal level (Gneiting & Raftery).
Invoked as Theorem 2.9 and used to prove Pareto optimality of the oracle IS-ROC (Theorem 2.10).
domain assumption Prediction intervals are central intervals from a quantile function: l_β = q_{β/2}, u_β = q_{1−β/2}, nested in β, merging at the median when β=1.
Definition 2.1 and Remark 2.3; the entire IS-ROC geometry is built on this family, not arbitrary asymmetric intervals.
standard math Expectations defining MS(β) and MAD(β) exist; sample means on {(x_i,y_i)} consistently estimate them for comparison and calibration.
Section 2.2 estimators and §4 workflow; needed for empirical curves to track population IS-ROC.
ad hoc to paper A Bernoulli mixture of two fixed interval predictors is an admissible tunable interval strategy whose (MS, MAD) lies on the segment joining the endpoints.
Lemma A.1 underpins convexification, non-convex calibration, and ensembling (§3.3–3.4, Theorem 2.12).
standard math For the strict convexity proof without covariates, G, S(β), MAD(β) are strictly monotonic and differentiable and MAD(S) is twice differentiable.
Hypothesis of Theorem 2.11; the general case drops this via the hull argument.

invented entities (2)

IS-ROC Curve
purpose: Parametric curve β ↦ (MS(β), MAD(β)) representing the full tightness trade-off of a tunable interval predictor.
Core object of the paper (Definition 2.4); methodological construct with geometric theorems, not an external physical entity.
Tunable Interval Predictor (TIP)
purpose: Formalize interval maps with a tightness knob, nesting, and median merger so families of intervals can be scored as curves.
Definition 2.1; packaging device for the framework rather than an empirically distinct object.

how reviews work

0 comments

Cite this review

Pith. "Pith review of An Interval-Score ROC Curve for Assessment, Calibration and Ensembling of Probabilistic Forecasts." pith.science (2026). https://pith.science/paper/GF4G35RI

@misc{pith2026260728178,
  author       = {Pith},
  title        = {Pith review of: An Interval-Score ROC Curve for Assessment, Calibration and Ensembling of Probabilistic Forecasts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GF4G35RI}},
  note         = {Machine review of arXiv:2607.28178}
}

read the original abstract

Probabilistic forecast evaluation is inherently multi-objective, yet existing proper scoring rules reduce predictive performance to a single scalar value, potentially obscuring the trade-off between forecast concentration and predictive accuracy. We introduce the Interval-Score Receiver Operating Characteristic (IS-ROC) Curve, a graphical framework that represents the complete family of interval forecasts generated by varying prediction tightness. We show that the IS-ROC Curve induced by the data generating process is Pareto optimal and convex, providing a geometric characterization of the optimal forecasting frontier. Building on these properties, we propose a geometry-based calibration procedure based on tangent optimization and convexification, together with an ensemble strategy that combines competing forecasters through convex hull construction. Finally, we provide a practical workflow and numerical examples illustrating forecast comparison, calibration, and ensemble construction within the proposed framework.

Figures

Figures reproduced from arXiv: 2607.28178 by the authors.

**Figure 2.** Example 1. Comparison of the DGP and the three probabilistic forecasters. The [PITH_FULL_IMAGE:figures/full_fig_p020_2.png] view at source ↗

**Figure 3.** Example 1. Comparison of the IS–ROC Curves. The left panel compares the [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: Example 1. The left panel illustrates the calibration function, which maps [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]

**Figure 5.** Figure 5: Example 2. Probability density functions corresponding to [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]

**Figure 6.** Figure 6: Example 2. IS–ROC Comparison. The left panel reports the original IS–ROC [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]

**Figure 7.** Figure 7: Example 2. Calibration process on the MAD–S plane for intersecting IS–ROC [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]

**Figure 8.** Figure 8: Example 2. Step calibration function associated with the convexified frontier. [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]

**Figure 9.** Figure 9: Example 2. Calibrated cumulative distribution functions corresponding to the [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]

**Figure 10.** Figure 10: Example of a linear IS–ROC Curve. The left panel shows the probability [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]

**Figure 11.** Figure 11: The quasi-convex IS–ROC Curve associated to the misspecified predictors is [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]

**Figure 12.** Figure 12: Two predictive distributions with the same median producing identical [PITH_FULL_IMAGE:figures/full_fig_p028_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references

[1]

Evaluating epidemic forecasts in an interval format.PLoS computational biology, 17(2):e1008618, 2021

Johannes Bracher, Evan L Ray, Tilmann Gneiting, and Nicholas G Reich. Evaluating epidemic forecasts in an interval format.PLoS computational biology, 17(2):e1008618, 2021

2021
[2]

Review on probabilistic forecasting of wind power generation.Renewable and Sustainable Energy Reviews, 32:255–270, 2014

Yao Zhang, Jianxue Wang, and Xifan Wang. Review on probabilistic forecasting of wind power generation.Renewable and Sustainable Energy Reviews, 32:255–270, 2014

2014
[3]

Vulnerable growth

Tobias Adrian, Nina Boyarchenko, and Domenico Giannone. Vulnerable growth. American Economic Review, 109(4):1263–1289, 2019

2019
[4]

A Philip Dawid. Present position and potential developments: Some personal views statistical theory the prequential approach.Journal of the Royal Statistical Society: Series A (General), 147(2):278–290, 1984

1984
[5]

Probabilistic forecasts, calibration and sharpness.Journal of the Royal Statistical Society Series B: Statistical Methodology, 69(2):243–268, 2007

Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E Raftery. Probabilistic forecasts, calibration and sharpness.Journal of the Royal Statistical Society Series B: Statistical Methodology, 69(2):243–268, 2007

2007
[6]

Strictly proper scoring rules, prediction, and estimation.Journal of the American statistical Association, 102(477):359–378, 2007

Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation.Journal of the American statistical Association, 102(477):359–378, 2007

2007
[7]

Forecast scoring and calibration.Lecture notes

Ryan Tibshirani. Forecast scoring and calibration.Lecture notes. URL: https://www. stat. berkeley. edu/˜ ryantibs/statlearn-s23/lectures/calibration. pdf, 2023

2023
[8]

An introduction to roc analysis.Pattern recognition letters, 27(8):861– 874, 2006

Tom Fawcett. An introduction to roc analysis.Pattern recognition letters, 27(8):861– 874, 2006

2006

Show all 13 references

[9]

Receiver operating characteristic (roc) movies, universal roc (uroc) curves, and coefficient of predictive ability (cpa).Machine Learning, 111(8):2769–2797, 2022

Tilmann Gneiting and Eva-Maria Walz. Receiver operating characteristic (roc) movies, universal roc (uroc) curves, and coefficient of predictive ability (cpa).Machine Learning, 111(8):2769–2797, 2022

2022
[10]

Evaluating interval forecasts.International economic review, pages 841–862, 1998

Peter F Christoffersen. Evaluating interval forecasts.International economic review, pages 841–862, 1998

1998
[11]

On the comparison of interval forecasts.Journal of Time Series Analysis, 39(6):953–965, 2018

Ross Askanazi, Francis X Diebold, Frank Schorfheide, and Minchul Shin. On the comparison of interval forecasts.Journal of Time Series Analysis, 39(6):953–965, 2018

2018
[12]

Combining predictive distributions

Tilmann Gneiting and Roopesh Ranjan. Combining predictive distributions. 2013

2013
[13]

Analysis and visualization of classifier performance with nonuniform class and cost distributions

Foster Provost and Tom Fawcett. Analysis and visualization of classifier performance with nonuniform class and cost distributions. InProceedings of AAAI-97 Workshop on AI Approaches to Fraud Detection & Risk Management, pages 57–63, 1997. 31

1997

Pith tools

Reviewed July 31, 2026 · model on record in the stance chip above.

[1] [1]

Evaluating epidemic forecasts in an interval format.PLoS computational biology, 17(2):e1008618, 2021

Johannes Bracher, Evan L Ray, Tilmann Gneiting, and Nicholas G Reich. Evaluating epidemic forecasts in an interval format.PLoS computational biology, 17(2):e1008618, 2021

2021

[2] [2]

Review on probabilistic forecasting of wind power generation.Renewable and Sustainable Energy Reviews, 32:255–270, 2014

Yao Zhang, Jianxue Wang, and Xifan Wang. Review on probabilistic forecasting of wind power generation.Renewable and Sustainable Energy Reviews, 32:255–270, 2014

2014

[3] [3]

Vulnerable growth

Tobias Adrian, Nina Boyarchenko, and Domenico Giannone. Vulnerable growth. American Economic Review, 109(4):1263–1289, 2019

2019

[4] [4]

A Philip Dawid. Present position and potential developments: Some personal views statistical theory the prequential approach.Journal of the Royal Statistical Society: Series A (General), 147(2):278–290, 1984

1984

[5] [5]

Probabilistic forecasts, calibration and sharpness.Journal of the Royal Statistical Society Series B: Statistical Methodology, 69(2):243–268, 2007

Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E Raftery. Probabilistic forecasts, calibration and sharpness.Journal of the Royal Statistical Society Series B: Statistical Methodology, 69(2):243–268, 2007

2007

[6] [6]

Strictly proper scoring rules, prediction, and estimation.Journal of the American statistical Association, 102(477):359–378, 2007

Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation.Journal of the American statistical Association, 102(477):359–378, 2007

2007

[7] [7]

Forecast scoring and calibration.Lecture notes

Ryan Tibshirani. Forecast scoring and calibration.Lecture notes. URL: https://www. stat. berkeley. edu/˜ ryantibs/statlearn-s23/lectures/calibration. pdf, 2023

2023

[8] [8]

An introduction to roc analysis.Pattern recognition letters, 27(8):861– 874, 2006

Tom Fawcett. An introduction to roc analysis.Pattern recognition letters, 27(8):861– 874, 2006

2006