REVIEW 3 major objections 4 minor 43 references
Efficient pooling of predictions via kernel embeddings
T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Estimating the weights that combine probabilistic forecasts optimally is a convex quadratic program whenever accuracy is scored by a kernel score.
desk verdict Proposition 2 is correct and the order-statistic pooling idea is genuinely new, but the paper's practical scope is mainly discrete predictive distributions, and the case study would benefit from uncertainty quantification. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the kernel mean embedding $\mu_F = \mathbb E_X[k(X,\cdot)]$ of a probability distribution $F$ into the reproducing kernel Hilbert space $\mathcal H_k$ of a positive definite kernel $k$. The identity that carries the argument is the kernel-score-as-squared-MMD formula $S_k(F,y)=\tfrac12\|\mu_F-k(y,\cdot)\|_{\mathcal H_k}^2$; combined with linearity of the embedding and convexity of squared norms, it turns the empirical score of a mixture into $w^\top A w + 2c^\top w$, with $A_{jl}=\sum_i \alpha_i \mathbb E[k(X_{j,i},X_{l,i})]$ and $c_j=-\sum_i \alpha_i\mathbb E[k(X_{j,i},y_i)]$. For discrete predictive distributions, these expectations reduce to finite averages of kernel evaluations, which is what makes the quadratic program easy to assemble for arbitrary kernels and domains.
What would settle it
Take a small synthetic set of discrete forecasts and observations, fix the energy kernel, and grid-search the simplex of weights for the lowest empirical CRPS; if any grid point beats the quadratic-program solution, the quadratic claim in Proposition 2 fails.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is Proposition 2: for any kernel score $S_k$ derived from a positive definite kernel $k$ on an outcome space $\mathcal Y$, the empirical score $\hat S_n(w)$ of a linear pool $\sum_j w_j F_{j,i}$ is a quadratic function of the weight vector $w$, and minimizing it subject to $w_j\ge 0$, $\sum_j w_j=1$ is a convex quadratic programming problem. The argument runs through kernel mean embeddings: with $\mu_F$ the embedding of a forecast $F$, the kernel score equals $\tfrac12\|\mu_F-\mu_{\delta_y}\|_{\mathcal H_k}^2$, and the embedding of the linear pool is the same linear combination of embeddings. Consequently the score matrix $A$ and linear term $c$ in the quadratic program are just kernel evaluations averaged over training data. The paper further claims that the same mechanism applies to weighted combinations of point predictions and, on the real line, to weighted combinations of order statistics of discrete predictive distributions, which can simultaneously recalibrate the pooled forecast; and it reports that in an application to operational wind speed forecasts the order-statistic weighting improves CRPS by 10–18% over the optimized linear pool and 20–30% over equal weighting.
Load-bearing premise
For the method to be computationally efficient, each component prediction must be a discrete predictive distribution (or a parametric family with a closed-form kernel embedding), because the quadratic program is assembled from kernel evaluations between sample members and observations.
Editorial extensions
If this is right
- If Proposition 2 is right, optimal kernel-score pooling is as easy as solving a quadratic program of dimension equal to the number of component forecasts, on any domain — Euclidean, functional, graph, or spatio-temporal — that admits a positive definite kernel.
- For discrete predictive distributions (ensemble forecasts, MCMC output, generative model samples), the entries of the quadratic program are plain kernel sums, so the method needs no closed-form embedding formulas.
- Weighting order statistics of discrete forecasts is a recalibrating non-linear combination that still solves the same kind of quadratic program, and it can reproduce any target distribution whose support is covered by the components (Proposition 6).
- In the wind-speed case study, order-statistic pooling improved CRPS by 10–18% over optimized linear pooling and 20–30% over equal weighting; roughly 10% improvements persisted after member-by-member post-processing.
- Weighted or quantile-focused kernel scores also fit the same quadratic framework, so preferences about which outcome regions matter most can be encoded through the choice of kernel.
Reading between the lines
- An implication the authors leave implicit: because the optimization is a convex quadratic program, the same weight-estimation step could be embedded in larger pipelines — online updating as new training pairs arrive, or joint estimation with kernel hyperparameters — without leaving the convex regime.
- A testable extension: the order-statistic weighting scheme could be applied to multivariate discrete forecasts by ranking sample members with a fixed order on $\mathbb R^d$; the paper notes this for future work, and the quadratic machinery would carry over directly once an order is chosen.
- The connection to MMD suggests a natural diagnostic: the optimal objective value equals the training-average kernel score, so the same code that estimates weights also yields a measure of how much pooling improved forecast accuracy.
- The case-study finding that control members received the lowest weight despite having the best point MSE suggests the method identifies which ensemble members contribute uncertainty information rather than point accuracy; that pattern may generalize beyond weather forecasting.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript develops an RKHS-based framework for estimating weights in probabilistic forecast combination. It shows that for any kernel score, the empirical score of a linear pool is a convex quadratic function of the weights (Proposition 2), reducing weight estimation to a convex quadratic programming problem. The framework is extended to pooling discrete predictive distributions, to pooling point predictions, and to an order-statistic pooling scheme that acts as a flexible recalibration device. Proposition 6 establishes a universality property for the generalized pool. The methods are applied to wind-speed ensemble forecasts from three operational models, where the order-statistic pool improves CRPS relative to the linear pool.
Significance. The main theoretical result is elegant and useful: it connects proper scoring rules, kernel mean embeddings, and forecast combination in a way that makes optimal kernel-score pooling computationally tractable. The proof of Proposition 2 is straightforward but the formulation as a convex QP is valuable, and the order-statistic extension is practically attractive. The case study illustrates the potential of the approach. The paper would be more convincing if the scope of the 'arbitrary outcome domains' efficiency claim were stated precisely and if the empirical improvements were accompanied by uncertainty quantification. With those revisions, this should be a solid contribution to the forecasting and statistical machine learning literature.
major comments (3)
- [Abstract; §4.1] The abstract and introduction claim 'efficient implementation ... on arbitrary outcome domains' without qualification. Section 4.1 correctly notes that the construction is trivial only when the predictive distributions are discrete (Equation 13) or when closed-form kernel mean embeddings are available; for continuous parametric families and common kernels, forming A and c generally requires numerical integration or Monte Carlo approximation. Please qualify the abstract and introduction to state that the efficiency guarantee applies directly to discrete predictive distributions, or specify conditions under which embeddings are analytically available.
- [§5.2-5.3, Figures 5-7] The reported improvements (10-18% over LP Discrete and 20-30% over equal-weight LP) are presented as point estimates without confidence intervals or significance tests. The weights are estimated per station and, to judge from Figures 3 and 5, also per lead time; for LP Ordered this amounts to up to 83 fitted weights per station-lead-time pair. The paper should explicitly state whether weights are estimated separately for each lead time and should quantify sampling uncertainty (for example, with a block bootstrap by station or by year) to rule out overfitting. This is important because the 'substantial improvements' claim in the abstract rests on these numbers.
- [Introduction; §4.2, Remark 7] The introduction states that the key result 'also holds for alternative combination strategies,' which is easy to read as applying to the general weight-function formulation in Equation 16. Remark 7 correctly notes that the general weight functions cannot generally be optimized via Proposition 2 unless the predictive distributions are discrete or the weight functions are supported on finitely many points. The wording should be adjusted so that the reader understands that only the discrete/order-statistic specialization inherits the QP efficiency.
minor comments (4)
- [§2.2, Equation (9)] The notation µδy in Equation (9) is typographically ambiguous; please write µ_{δ_y}.
- [Example 3] The displayed formula for A_{j,l} should show the denominator M_j M_l explicitly; as typeset, it is easy to misread the expression as missing the division.
- [§5.1] Please state whether the 1030 days already exclude missing dates and how missing dates are handled when constructing the QP matrices A and c.
- [§4.2, Example 5] A short remark on uniqueness of the QP solution when A is singular (e.g., duplicated order statistics) would be helpful, since the order-statistic formulation can produce repeated points.
Circularity Check
No circular derivation: Proposition 2 is direct algebra from kernel embeddings; the case study is out-of-sample; imported self-citations are peripheral and independently checkable.
full rationale
The central claim, Proposition 2, is a direct algebraic consequence of the kernel mean embedding representation: the empirical kernel score of a linear pool is written as a quadratic form w^T A w + 2 c^T w, with A in equation 10 a weighted sum of Gram matrices and therefore positive semidefinite. No fitted parameter is renamed as a prediction; the weights are decision variables optimized on training data and evaluated on a held-out year. Proposition 6 constructs weight functions explicitly as w_j = f_0 / f_+ from a target density, which is a representation/existence statement rather than an empirical prediction, so building the weights from f_0 is not circular. The paper also honestly flags the practical requirement of computable kernel mean embeddings for continuous parametric families in Section 4.1, so the abstract's "arbitrary outcome domains" phrasing is an over-broad implementation claim rather than a circularity. Self-citations, including Allen et al. (2023), Steinwart & Ziegel (2021), and Ziegel et al. (2024), support peripheral facts such as weighted kernel scores being kernel scores and characteristic kernels giving strict propriety; these are published, externally checkable mathematical results and are not the load-bearing step in the QP reduction. Overall, no load-bearing circular step was found; the score of 1 reflects only the minor gap between the broad abstract wording and the discrete-distribution implementation caveat, not circular reasoning.
Assumptions & free parameters
free parameters (1)
- Combination weights w_j (or w_{j,m} for point and order-statistic pooling) =
Estimated per station, lead time, and method; full vectors not reported
assumptions (5)
- standard math Positive definite kernel k on Y has a reproducing kernel Hilbert space with the reproducing property, and expectations defining mean embeddings are finite.
- standard math Kernel score S_k in equation 4 is a proper scoring rule and equals half the squared MMD between forecast and observation Dirac.
- domain assumption Component predictions F_{j,i} are fixed distributions with known or computable kernel mean embeddings; in the application they are discrete empirical measures with equal member weights.
- domain assumption For the case study, the two-year training period is representative of the test year, with no major model changes.
- domain assumption Universality Proposition 6 assumes the target distribution's support is contained in the union of component supports and all distributions are absolutely continuous.
Cite this review
Pith. "Pith review of Efficient pooling of predictions via kernel embeddings." pith.science (2026). https://pith.science/paper/RLO5J5PK
@misc{pith2026241116246,
author = {Pith},
title = {Pith review of: Efficient pooling of predictions via kernel embeddings},
year = {2026},
howpublished = {\url{https://pith.science/paper/RLO5J5PK}},
note = {Machine review of arXiv:2411.16246}
}
read the original abstract
Probabilistic predictions are probability distributions over the set of possible outcomes. Such predictions quantify the uncertainty in the outcome, making them essential for effective decision making. By combining multiple predictions, the information sources used to generate the predictions are pooled, often resulting in a more informative forecast. Probabilistic predictions are typically combined by linearly pooling the individual predictive distributions; this encompasses several ensemble learning techniques, for example. The weights assigned to each prediction can be estimated based on their past performance, allowing more accurate predictions to receive a higher weight. This can be achieved by finding the weights that optimise a proper scoring rule over some training data. By embedding predictions into a Reproducing Kernel Hilbert Space (RKHS), we illustrate that estimating the linear pool weights that optimise kernel-based scoring rules is a convex quadratic optimisation problem. This permits an efficient implementation of the linear pool when optimally combining predictions on arbitrary outcome domains. This result also holds for other combination strategies, and we additionally study a flexible generalisation of the linear pool that overcomes some of its theoretical limitations, whilst allowing an efficient implementation within the RKHS framework. These approaches are compared in an application to operational wind speed forecasts, where this generalisation is found to offer substantial improvements upon the traditional linear pool.
Figures
Figures from the paper (10 more)
Reference graph
Works this paper leans on
-
[1]
Evaluating forecasts for high-impact events using transformed kernel scores
Sam Allen, David Ginsbourger, and Johanna Ziegel. Evaluating forecasts for high-impact events using transformed kernel scores. SIAM/ASA Journal on Uncertainty Quantification, 11: 0 906--940, 2023
work page 2023
-
[2]
Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American Mathematical Society, 68: 0 337--404, 1950
work page 1950
-
[3]
Harmonic analysis on semigroups, volume 100
Christian Berg, Jens Peter Reus Christensen, and Paul Ressel. Harmonic analysis on semigroups, volume 100. Springer-Verlag New York, 1984
work page 1984
-
[4]
Jonathan Berrisch and Florian Ziel. CRPS learning. Journal of Econometrics, 237: 0 105221, 2023
work page 2023
-
[5]
Bagging predictors
Leo Breiman. Bagging predictors. Machine Learning, 24: 0 123--140, 1996
1996
-
[6]
Random forests
Leo Breiman. Random forests. Machine Learning, 45: 0 5--32, 2001
2001
-
[7]
Verification of forecasts expressed in terms of probability
Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78: 0 1--3, 1950
work page 1950
-
[8]
Ensemble forecasting and the need for calibration
Roberto Buizza. Ensemble forecasting and the need for calibration. In Statistical postprocessing of ensemble forecasts, pp.\ 15--48. Elsevier, 2018
work page 2018
Show all 43 references
-
[9]
Combining probability distributions from experts in risk analysis
Robert T Clemen and Robert L Winkler. Combining probability distributions from experts in risk analysis. Risk Analysis, 19: 0 187--203, 1999
1999
-
[10]
The geometry of proper scoring rules
A Philip Dawid. The geometry of proper scoring rules. Annals of the Institute of Statistical Mathematics, 59: 0 77--93, 2007
2007
-
[11]
Coherent combination of experts' opinions
Alexander P Dawid, Morris H DeGroot, Julia Mortera, Roger Cooke, Simon French, Christian Genest, MJ Schervish, Dennis V Lindley, KJ McConway, and Robert L Winkler. Coherent combination of experts' opinions. Test, 4: 0 263--313, 1995
1995
-
[12]
Ensemble learning
Thomas G Dietterich. Ensemble learning. The Handbook of Brain Theory and Neural Networks, 2 0 (1): 0 110--125, 2002
2002
-
[13]
Experiments with a new boosting algorithm
Yoav Freund and Robert E Schapire. Experiments with a new boosting algorithm. In International Conference on Machine Learning, volume 96, pp.\ 148--156. Citeseer, 1996
1996
-
[14]
Kernel measures of conditional dependence
Kenji Fukumizu, Arthur Gretton, Xiaohai Sun, and Bernhard Sch \"o lkopf. Kernel measures of conditional dependence. Advances in Neural Information Processing Systems, 20, 2007
2007
-
[15]
Pooling operators with the marginalization property
Christian Genest. Pooling operators with the marginalization property. Canadian Journal of Statistics, 12: 0 153--163, 1984
1984
-
[16]
Strictly proper scoring rules, prediction, and estimation
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102: 0 359--378, 2007
2007
-
[17]
Comparing density forecasts using threshold-and quantile-weighted scoring rules
Tilmann Gneiting and Roopesh Ranjan. Comparing density forecasts using threshold-and quantile-weighted scoring rules. Journal of Business & Economic Statistics, 29: 0 411--422, 2011
2011
-
[18]
Combining predictive distributions
Tilmann Gneiting and Roopesh Ranjan. Combining predictive distributions. Electronic Journal of Statistics, 7: 0 1747--1782, 2013
2013
-
[19]
The continuous ranked probability score for circular variables and its application to mesoscale forecast ensemble verification
Eric P Grimit, Tilmann Gneiting, Veronica J Berrocal, and Nicholas A Johnson. The continuous ranked probability score for circular variables and its application to mesoscale forecast ensemble verification. Quarterly Journal of the Royal Meteorological Society, 132: 0 2925--2942, 2006
2006
-
[20]
Combining density forecasts
Stephen G Hall and James Mitchell. Combining density forecasts. International Journal of Forecasting, 23: 0 1--13, 2007
2007
-
[21]
The elements of statistical learning: data mining, inference, and prediction
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction. Springer, 2 edition, 2017
2017
-
[22]
Probability judgments for continuous quantities: Linear combinations and calibration
Stephen C Hora. Probability judgments for continuous quantities: Linear combinations and calibration. Management Science, 50: 0 597--604, 2004
2004
-
[23]
kernlab - an S4 package for kernel methods in R
Alexandros Karatzoglou, Alexandros Smola, Kurt Hornik, and Achim Zeileis. kernlab - an S4 package for kernel methods in R . Journal of Statistical Software, 11: 0 1--20, 2004
2004
-
[24]
Seamless multimodel postprocessing for air temperature forecasts in complex topography
Regula Keller, Jan Rajczak, Jonas Bhend, Christoph Spirig, Stephan Hemri, Mark A Liniger, and Heini Wernli. Seamless multimodel postprocessing for air temperature forecasts in complex topography. Weather and Forecasting, 36: 0 1031--1042, 2021
2021
-
[25]
A survey on graph kernels
Nils M Kriege, Fredrik D Johansson, and Christopher Morris. A survey on graph kernels. Applied Network Science, 5: 0 1--42, 2020
2020
-
[26]
Ensemble forecasting
Martin Leutbecher and Tim N Palmer. Ensemble forecasting. Journal of Computational Physics, 227: 0 3515--3539, 2008
2008
-
[27]
Is it better to average probabilities or quantiles? Management Science, 59: 0 1594--1611, 2013
Kenneth C Lichtendahl Jr., Yael Grushka-Cockayne, and Robert L Winkler. Is it better to average probabilities or quantiles? Management Science, 59: 0 1594--1611, 2013
2013
-
[28]
Scoring rules for continuous probability distributions
James E Matheson and Robert L Winkler. Scoring rules for continuous probability distributions. Management Science, 22: 0 1087--1096, 1976
1976
-
[29]
Kernel mean embedding of distributions: A review and beyond
Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, Bernhard Sch \"o lkopf, et al. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends in Machine Learning , 10: 0 1--141, 2017
2017
-
[30]
Combining density forecasts using focused scoring rules
Anne Opschoor, Dick Van Dijk, and Michel van der Wel. Combining density forecasts using focused scoring rules. Journal of Applied Econometrics, 32: 0 1298--1313, 2017
2017
-
[31]
Variogram-based proper scoring rules for probabilistic forecasts of multivariate quantities
Michael Scheuerer and Thomas M Hamill. Variogram-based proper scoring rules for probabilistic forecasts of multivariate quantities. Monthly Weather Review, 143: 0 1321--1334, 2015
2015
-
[32]
o lkopf, Alexander Smola, and Klaus-Robert M \
Bernhard Sch \"o lkopf, Alexander Smola, and Klaus-Robert M \"u ller. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 10: 0 1299--1319, 1998
1998
-
[33]
Equivalence of distance-based and RKHS -based statistics in hypothesis testing
Dino Sejdinovic, Bharath Sriperumbudur, Arthur Gretton, and Kenji Fukumizu. Equivalence of distance-based and RKHS -based statistics in hypothesis testing. The Annals of Statistics, pp.\ 2263--2291, 2013
2013
-
[34]
Strictly proper kernel scores and characteristic kernels on compact spaces
Ingo Steinwart and Johanna F Ziegel. Strictly proper kernel scores and characteristic kernels on compact spaces. Applied and Computational Harmonic Analysis, 51: 0 510--542, 2021
2021
-
[35]
The opinion pool
Mervyn Stone. The opinion pool. The Annals of Mathematical Statistics, pp.\ 1339--1342, 1961
1961
-
[36]
Online learning with the continuous ranked probability score for ensemble forecasting
Jean Thorey, Vivien Mallet, and Paul Baudin. Online learning with the continuous ranked probability score for ensemble forecasting. Quarterly Journal of the Royal Meteorological Society, 143: 0 521--529, 2017
2017
-
[37]
Ensemble post-processing using member-by-member approaches: theoretical aspects
Bert Van Schaeybroeck and St \'e phane Vannitsem. Ensemble post-processing using member-by-member approaches: theoretical aspects. Quarterly Journal of the Royal Meteorological Society, 141: 0 807--818, 2015
2015
-
[38]
Forecast combinations: An over 50-year review
Xiaoqian Wang, Rob J Hyndman, Feng Li, and Yanfei Kang. Forecast combinations: An over 50-year review. International Journal of Forecasting, 39: 0 1518--1547, 2023
2023
-
[39]
Statistical methods for post-processing ensemble weather forecasts
Robin Williams. Statistical methods for post-processing ensemble weather forecasts. PhD thesis, University of Exeter, 2016
2016
-
[40]
Probability forecasts and their combination: A research perspective
Robert L Winkler, Yael Grushka-Cockayne, Kenneth C Lichtendahl Jr, and Victor Richmond R Jose. Probability forecasts and their combination: A research perspective. Decision Analysis, 16: 0 239--260, 2019
2019
-
[41]
A kernel two-sample test for functional data
George Wynne and Andrew B Duncan. A kernel two-sample test for functional data. Journal of Machine Learning Research, 23: 0 1--51, 2022
2022
-
[42]
Characteristic kernels on Hilbert spaces , Banach spaces , and on sets of measures
Johanna Ziegel, David Ginsbourger, and Lutz D \"u mbgen. Characteristic kernels on Hilbert spaces , Banach spaces , and on sets of measures. Bernoulli, 30: 0 1441--1457, 2024
2024
-
[43]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.