Uncertainty quantification via conformal prediction in data assimilation
Pith reviewed 2026-06-26 05:38 UTC · model grok-4.3
The pith
Conformal prediction generates uncertainty intervals for data assimilation that achieve target coverage in a one-dimensional shallow water model.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
In the one-dimensional modified shallow water model, standard CP, normalized CP, and conformalized quantile regression each produce prediction sets whose average empirical coverage meets or approaches the chosen confidence level, with normalized CP and CQR typically yielding shorter intervals and lower interval score loss than standard CP while all three variants can supply perturbations that are assimilated alongside or in place of ensemble perturbations.
What carries the argument
Conformal prediction, which wraps any point predictor to return a set of outcomes guaranteed to contain the true value at a user-specified probability under exchangeability.
If this is right
- CP variants can be inserted into existing ensemble-based assimilation systems without retraining the underlying model.
- Normalized CP and conformalized quantile regression tend to produce tighter intervals than standard CP at the same coverage level.
- CP-derived perturbations can replace or augment ensemble spread in the assimilation step.
- The relative strengths of each CP variant depend on the chosen score metric and the variability of the underlying forecast errors.
Where Pith is reading between the lines
- The approach could be combined with ensemble methods to produce hybrid uncertainty estimates that inherit both finite-sample guarantees and flow-dependent spread.
- If coverage holds in higher-dimensional models, CP might allow smaller ensemble sizes while still meeting reliability requirements for probabilistic forecasts.
- The method's performance under model error or non-exchangeable conditions remains open for direct testing.
Load-bearing premise
The one-dimensional modified shallow water model sufficiently represents the challenges of uncertainty quantification in data assimilation for numerical weather prediction.
What would settle it
Repeating the experiments in a three-dimensional or operational atmospheric model and observing that any CP variant's empirical coverage falls more than a few percentage points below its nominal level across multiple assimilation cycles.
read the original abstract
Quantifying the evolution of uncertainty is critical to both probabilistic forecasting and data assimilation in numerical weather prediction. In this study, we investigate the applicability of conformal prediction (CP), a recent machine learning (ML) method, to quantify uncertainty in a controlled, idealized setting. We use the one dimensional modified shallow water model, designed to mimic the convective process. CP provides a set of possible outcomes with a chosen confidence level. Here, we compare and evaluate the average empirical coverage, the average interval length, miss low, miss high and average interval score loss (AISL) for three variants of CP, namely a) Standard CP, b) Normalized CP and c) Conformalized Quantile Regression. We further compare these CP-based uncertainty estimates with traditional ensemble-based measures such as standard deviation intervals and ensemble spread. In addition, we investigate the integration of CP-derived uncertainty within the data assimilation cycle through CP perturbations. Our results highlight the strengths and limitations of each approach, providing insight into the effectiveness of CP to complement common ensemble-based uncertainty quantification in simplified atmospheric models.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper examines conformal prediction (CP) for uncertainty quantification in data assimilation, using a one-dimensional modified shallow water model to mimic convective processes. It evaluates three CP variants (standard CP, normalized CP, conformalized quantile regression) against ensemble-based methods (standard deviation intervals, ensemble spread) via metrics including empirical coverage, interval length, miss rates, and average interval score loss (AISL). It also tests integration of CP perturbations into the data assimilation cycle and concludes that the results highlight strengths and limitations of each approach for simplified atmospheric models.
Significance. If the empirical comparisons hold, the work offers concrete, scoped insight into CP as a complement to ensemble UQ in idealized DA settings. Credit is due for explicit scoping to the 1D model, use of standard CP evaluation metrics, and direct comparison to ensemble spread/std rather than overclaiming transfer to operational NWP.
minor comments (3)
- The abstract and introduction should explicitly state the number of ensemble members, the exact form of the modified shallow water equations, and the data assimilation scheme (e.g., EnKF variant) used, as these choices directly affect the reported coverage and AISL values.
- Figure captions and axis labels for the coverage and interval-length plots should include the exact confidence level (e.g., 90 %) and the number of test realizations, to allow readers to assess statistical variability without consulting the main text.
- The section describing CP perturbations in the DA cycle should clarify whether the conformal sets are used to replace or augment the ensemble perturbations and whether any recalibration of the assimilation weights is performed.
Simulated Author's Rebuttal
We thank the referee for the careful reading and positive assessment of our manuscript on conformal prediction for uncertainty quantification in idealized data assimilation. The recommendation for minor revision is noted. No specific major comments were raised in the report.
Circularity Check
No significant circularity detected
full rationale
The paper applies standard conformal prediction variants (Standard CP, Normalized CP, Conformalized Quantile Regression) and compares them directly to ensemble spread/std using fixed, externally defined metrics (empirical coverage, interval length, miss rates, AISL) on a 1D modified shallow water model. No load-bearing derivation, fitted parameter, or self-citation chain reduces any claimed result to its own inputs by construction; the evaluation is a straightforward experimental comparison scoped to the idealized setting without internal redefinition or renaming of known results.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
M. W\". A simple dynamical model of cumulus convection for data assimilation research , journal =
-
[2]
Analysis Scheme in the Ensemble K alman Filter
Burgers, Gerrit and van Leeuwen, Peter Jan and Evensen, Geir. Analysis Scheme in the Ensemble K alman Filter. Mon. Wea. Rev. 1998
1998
-
[3]
Evensen, Geir , title =. Ocean Dynamics , year =. doi:10.1007/s10236-003-0036-9 , url =
-
[4]
G. Evensen. Data Assimilation: The Ensemble Kalman Filter. 2009
2009
-
[5]
G. Evensen. Sequential data assimilation with a nonliear quasi-geostrophic model using M onte C arlo methods to forecast error statistics. Journal of Geophysical Research. 1994
1994
-
[6]
and Janji\'c, Tijana and Craig, George C
Haslehner, M. and Janji\'c, Tijana and Craig, George C. , title =. Q J R Meteorol Soc , volume =. doi:10.1002/qj.2757 , pages =
-
[7]
Proceedings of the 34th International Conference on Machine Learning , pages =
On Calibration of Modern Neural Networks , author =. Proceedings of the 34th International Conference on Machine Learning , pages =. 2017 , editor =
2017
-
[8]
Price, Ilan and Sanchez-Gonzalez, Alvaro and Alet, Ferran and Andersson, Tom R. and El-Kadi, Andrew and Masters, Dominic and Ewalds, Timo and Stott, Jacklynn and Mohamed, Shakir and Battaglia, Peter and Lam, Remi and Willson, Matthew , title =. Nature , year =. doi:10.1038/s41586-024-08252-9 , url =
-
[9]
Tijana Janji\'c and Dennis McLaughlin and Stephen E. Cohn and Martin Verlaan. Conservation of Mass and Preservation of Positivity with Ensemble-Type Kalman Filter Algorithms. Monthly Weather Review. 2014. doi:10.1175/MWR-D-13-00056.1
-
[10]
Janji\'c, T. and Ruckstuhl, Y. and Toint, P. L. , journal =. doi:https://doi.org/10.1002/qj.4004 , url =. https://rmets.onlinelibrary.wiley.com/doi/pdf/10.1002/qj.4004 , year =
-
[11]
Geophysical Research Letters , title =
Tijana Janji. Geophysical Research Letters , title =. 2021 , number =. doi:10.1029/2021gl094962 , volume =
-
[12]
Quarterly Journal of the Royal Meteorological Society , volume =
Gustafsson, N. and Janji\'c, T. and Schraff, C. and Leuenberger, D. and Weissman, M. and Reich, H. and Brousseau, P. and Montmerle, T. and Wattrelot, E. and BuaÂnek, A. and Mile, M. and Hamdi, R. and Lindskog, M. and Barkmeijer, J. and Dahlbom, M. and Macpherson, B. and Ballard, S. and Inverarity, G. and Carley, J. and Alexander, C. and Dowell, D. and Li...
-
[13]
Quarterly Journal of the Royal Meteorological Society , volume=
Comparison of uncertainty quantification methods for cloud simulation , author=. Quarterly Journal of the Royal Meteorological Society , volume=. 2023 , publisher=
2023
-
[14]
Ruckstuhl, Y. and T. Janji\'c , title =. Q J R Meteorol Soc , volume =. 2018 , doi=
2018
-
[15]
and Janji\'c, T
Ruckstuhl, Y. and Janji\'c, T. and Rasp, S. , doi =. Training a convolutional neural network to conserve mass in data assimilation , url =. Nonlinear Processes in Geophysics , number =
-
[16]
osner, S. and Dembczynski, K. and Haasenritter, J. and Hirsch, O. and Donner-Banzhoff, N. and H\
Senge, R. and B\"osner, S. and Dembczynski, K. and Haasenritter, J. and Hirsch, O. and Donner-Banzhoff, N. and H\"ullermeier, E. , title=. Information Sciences , volume=
-
[18]
and Destercke, S
H\"ullermeier, E. and Destercke, S. and Shaker, M.H. , title=. Proc.\ UAI, 38th Conference on Uncertainty in Artificial Intelligence , address=
-
[19]
Quarterly Journal of the Royal Meteorological Society , volume =
Palmer, Tim , title =. Quarterly Journal of the Royal Meteorological Society , volume =. doi:https://doi.org/10.1002/qj.3383 , abstract =
-
[20]
Quarterly Journal of the Royal Meteorological Society , year=
Skill and relative economic value of the ECMWF ensemble prediction system , author=. Quarterly Journal of the Royal Meteorological Society , year=
-
[21]
Kuleshov and N
V. Kuleshov and N. Fenner and S. Ermon , year=2018, title=. Proc. ICML, 35th International Conference on Machine Learning , pages=
2018
-
[22]
Pearce and A
T. Pearce and A. Brintrup and M. Zaki and A. Neely , title=. Proc.\ ICML, Int.\ Conf.\ on Machine Learning , pages=
-
[23]
H\"ullermeier and W.\ Waegeman , title=
E. H\"ullermeier and W.\ Waegeman , title=. Machine Learning , volume=110, number=3, pages=
-
[24]
Shaker and E
M.H. Shaker and E. H\"ullermeier , title =. CoRR , volume =. 2021 , url =
2021
-
[25]
Evaluating Robustness of Predictive Uncertainty Estimation: Are
A. Evaluating Robustness of Predictive Uncertainty Estimation: Are. Proc.\. 2021 , pages =
2021
-
[26]
ugner and S. G\
B. Charpentier and D. Z\"ugner and S. G\"unnemann , title=. Proc.\ NeurIPS, 33rd Neural Information Processing Systems , volume=
-
[27]
Malinin and M
A. Malinin and M. Gales , booktitle =. Predictive Uncertainty Estimation via Prior Networks , pages =
-
[28]
Proc.\ NeurIPS, 31st Conference on Neural Information Processing Systems , address=
Evidential Deep Learning to Quantify Classification Uncertainty , author=. Proc.\ NeurIPS, 31st Conference on Neural Information Processing Systems , address=
-
[29]
Proc.\ ICML, 35th International Conference on Machine Learning , pages=
S.\ Depeweg and J.M.\ Hernandez-Lobato and F.\ Doshi-Velez and S.\ Udluft , title=. Proc.\ ICML, 35th International Conference on Machine Learning , pages=
-
[30]
Silva Filho and H
T. Silva Filho and H. Song and M. Perell. Classifier Calibration:. CoRR , volume =. 2021 , url =
2021
-
[31]
Proedrou and I
K. Proedrou and I. Nouretdinov and V. Vovk and A. Gammerman , title=. Proc.\ European Conf.\ on Machine Learning, ECML , pages=
-
[32]
Tools in Artificial Intelligence , volume=18, number=2, pages=
H.\ Papadopoulos , title=. Tools in Artificial Intelligence , volume=18, number=2, pages=
-
[33]
Papadopoulos and K
H. Papadopoulos and K. Proedrou and V. Vovk and A. Gammerman , title=. Proc.\ ECML, European Conference on Machine Learning , pages=
-
[34]
Song and T
H. Song and T. Diethe and M. Kull and P. Flach , year=2019, title=. Proc.\ ICML, 36th International Conference on Machine Learning , page=
2019
-
[35]
Sahoo and S
R. Sahoo and S. Zhao and A. Chen and S. Ermon , year=2021, title=. Threshold , volume=3, number=
2021
-
[36]
Int.\ Journal of Neural Systems , volume=14, number=2, pages=
M.\ Seeger , title=. Int.\ Journal of Neural Systems , volume=14, number=2, pages=
-
[37]
Bayesian Neural Networks: An Introduction and Survey , ISBN=
Goan, Ethan and Fookes, Clinton , year=. Bayesian Neural Networks: An Introduction and Survey , ISBN=. doi:10.1007/978-3-030-42553-1_3 , booktitle=
-
[38]
Blei and A
D.M. Blei and A. Kucukelbir and J.D. McAuliffe , title=. Journal of the American Statistical Association , volume=112, number=518, pages=
-
[39]
Breiman , title=
L. Breiman , title=. Machine Learning , volume=45, number=1, pages=
-
[40]
Lakshminarayanan and A
B. Lakshminarayanan and A. Pritzel and C. Blundell , title=. Proc.\ NeurIPS, 31st Conf.\ on Neural Information Processing Systems , address=
-
[41]
Proc.\ ICANN 2016, Part II , series=
Analysis of dropout learning regarded as ensemble learning , author=. Proc.\ ICANN 2016, Part II , series=
2016
-
[42]
Koenker and K.F
R. Koenker and K.F. Hallock , title=. Journal of Economic Perspectives , volume=15, number=4, pages=
-
[43]
Proc.\ ICML, 33rd Int.\ Conf.\ on Machine Learning , address=
Y.\ Gal and Z.\ Ghahramani , title=. Proc.\ ICML, 33rd Int.\ Conf.\ on Machine Learning , address=
-
[44]
Nature Reviews Physics , year=2021, volume=3, number=6, pages=
Physics-informed machine learning , author=. Nature Reviews Physics , year=2021, volume=3, number=6, pages=
2021
-
[45]
Yang and X
L. Yang and X. Meng and G.E. Karniadakis , title=. Journal of Computational Physics , volume=415, pages=
-
[46]
L. von R. Informed Machine Learning --. CoRR , volume =. 2019 , url =
2019
-
[47]
Raissi and P
M. Raissi and P. Perdikaris and G.E. Karniadakis , title=. Journal of Computational Physics , volume=378, pages=
-
[48]
Lagaris and A
I.E. Lagaris and A. Likas and D.I. Fotiadis , title=. IEEE Transactions on Neural Networks , volume=9, pages=
-
[49]
2006 , publisher=
Gaussian Processes for Machine Learning , author=. 2006 , publisher=
2006
-
[50]
2019 , publisher=
Automated Machine Learning: Methods, Systems, Challenges , author=. 2019 , publisher=
2019
-
[51]
Journal of Artificial intelligence Research , year=2022, note=
A Survey of Methods for Automated Algorithm Configuration , author=. Journal of Artificial intelligence Research , year=2022, note=
2022
-
[52]
Proc.\ ICML 2017, 34th Int.\ Conf.\ on Machine Learning , editor=
Statistical Inference for Incomplete Ranking Data: The Case of Rank-Dependent Coarsening , author=. Proc.\ ICML 2017, 34th Int.\ Conf.\ on Machine Learning , editor=
2017
-
[53]
Elsken and J.H
T. Elsken and J.H. Metzen and F. Hutter , title=. Journal of Machine Learning Research , volume=20, number=55, pages=
-
[54]
H\"ullermeier , title =
A.\ Shaker and E. H\"ullermeier , title =. Neurocomputing , year =
-
[55]
Heldt and E
A.\ Shaker and W. Heldt and E. H\"ullermeier , title=. Proc.\ ECML/PKDD, European Conf.\ on Machine Learning and Knowledge Discovery in Databases , address=
-
[56]
Wever and A
M. Wever and A. Tornede and F. Mohr and E. H\"ullermeier , title=. IEEE Transaction on Pattern Analysis and Machine Intelligence , doi=
-
[57]
Mohr and M
F. Mohr and M. Wever and A. Tornede and E. H\"ullermeier , title=. IEEE Transaction on Pattern Analysis and Machine Intelligence , volume=43, number=9, pages=
-
[58]
Waegeman and K
W. Waegeman and K. Dembczynski and E. H\"ullermeier , title=. Data Mining and Knowledge Discovery , volume=
-
[59]
Philosophical Transactions of the Royal Society A , doi=
Deep learning for post-processing ensemble weather forecasts , author=. Philosophical Transactions of the Royal Society A , doi=
-
[60]
CoRR , author=
On the Difficulty of Epistemic Uncertainty Quantification in Machine Learning:. CoRR , author=
-
[61]
V.L. Nguyen and M.H. Shaker and E. H\"ullermeier , title=. doi:10.1007/s10994-021-06003-9 , journal=
-
[62]
H\"ullermeier , title =
F.\ Mohr and M.\ Wever and E. H\"ullermeier , title =. Machine Learning , volume =
-
[63]
H\"ullermeier , title=
A.\ Shaker and E. H\"ullermeier , title=. Int.\ Journal of Applied Mathematics and Computer Science , volume=24, number=1, pages=
-
[64]
Shaker and E
A. Shaker and E. H\"ullermeier , title=. Evolving Systems , volume=3, pages=
-
[65]
Beringer and E
J. Beringer and E. H\"ullermeier , title=. Data and Knowledge Engineering , publisher=
-
[66]
Lienen and N
J. Lienen and N. Nommensen and R. Ewerth and E. H\"ullermeier , title=. Proc.\ ACML, Asian Conference on Machine Learning , year=2021, pages=
2021
-
[67]
Proc.\ NIPS, Advances in Neural Information Processing Systems , pages=
A.\ Kendall and Y.\ Gal , title=. Proc.\ NIPS, Advances in Neural Information Processing Systems , pages=
-
[68]
osner and K.\ Dembczynski and J.\ Haasenritter and O.\ Hirsch and N.\ Donner-Banzhoff and E. H\
R.\ Senge and S.\ B\"osner and K.\ Dembczynski and J.\ Haasenritter and O.\ Hirsch and N.\ Donner-Banzhoff and E. H\"ullermeier , title=. Information Sciences , volume=
-
[69]
H\"ullermeier , title=
E. H\"ullermeier , title=. Int.\ Journal of Approximate Reasoning , volume=55, number=7, pages=
-
[70]
Mathematics of Climate and Weather Forecasting , volume =
Stochastic Galerkin method for cloud simulation , DOI =. Mathematics of Climate and Weather Forecasting , volume =. 2019 , pages =
2019
-
[71]
Conformal Prediction for Reliable Machine Learning: Theory, Adaptations and Applications , editor=
-
[72]
Wilhelm, Jannik and Mohr, Susanna and Punge, Heinz JÃŒrgen and MÃŒhr, Bernhard and Schmidberger, Manuel and Daniell, James E. and Bedka, Kristopher M. and Kunz, Michael , title =. Weather , volume =. doi:https://doi.org/10.1002/wea.3886 , year =
-
[73]
Chaos: An Interdisciplinary Journal of Nonlinear Science , volume =
Wikner,Alexander and Pathak,Jaideep and Hunt,Brian and Girvan,Michelle and Arcomano,Troy and Szunyogh,Istvan and Pomerance,Andrew and Ott,Edward , title =. Chaos: An Interdisciplinary Journal of Nonlinear Science , volume =. 2020 , doi =
2020
-
[74]
2022 , eprint=
A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification , author=. 2022 , eprint=
2022
-
[75]
2022 , publisher=
Algorithmic Learning in a Random World , author=. 2022 , publisher=
2022
-
[76]
Proceedings of the IASTED International Conference on Artificial Intelligence and Applications (AIA 2008) , year=
Normalized nonconformity measures for regression conformal prediction , author=. Proceedings of the IASTED International Conference on Artificial Intelligence and Applications (AIA 2008) , year=
2008
-
[77]
Advances in neural information processing systems , volume=
Conformalized quantile regression , author=. Advances in neural information processing systems , volume=
-
[78]
Ashesh Chattopadhyay and Ebrahim Nabizadeh and Eviatar Bach and Pedram Hassanzadeh , keywords =. Deep learning-enhanced ensemble-based data assimilation for high-dimensional nonlinear dynamical systems , journal =. 2023 , issn =. doi:https://doi.org/10.1016/j.jcp.2023.111918 , url =
-
[79]
Journal of the American Statistical Association , year=
Strictly Proper Scoring Rules, Prediction, and Estimation , author=. Journal of the American Statistical Association , year=
-
[80]
Quarterly Journal of the Royal Meteorological Society , volume =
Gleiter, Tabea and Janjić, Tijana and Chen, Nan , title =. Quarterly Journal of the Royal Meteorological Society , volume =. doi:https://doi.org/10.1002/qj.4245 , url =. https://rmets.onlinelibrary.wiley.com/doi/pdf/10.1002/qj.4245 , year =
-
[81]
Legler, S. and Janjić, T. , title =. Quarterly Journal of the Royal Meteorological Society , volume =. doi:https://doi.org/10.1002/qj.4235 , url =. https://rmets.onlinelibrary.wiley.com/doi/pdf/10.1002/qj.4235 , year =
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.