REVIEW 5 major objections 5 minor 24 references
Boosting weather forecast via generative superensemble
T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read GenEPS, a plug-and-play generative framework, turns deterministic AI weather forecasts into an ensemble superensemble that reaches a 10-day 500 hPa geopotential ACC of 0.679, exceeding the ECMWF ensemble's 0.646.
desk verdict A genuinely new plug-and-play ensemble wrapper for deterministic ML weather models, but the headline skill gain over ECMWF ENS is not yet credible because the verification target is the same ERA5 used to train the generative prior. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is an unconditional diffusion prior over atmospheric states, trained patch-wise on ERA5 (patches of 360×720 to keep one Rossby wave cycle). It is used via SDEdit: a deterministic forecast field $x_1$ is pushed a fraction $t_0 = 0.4$ along the forward diffusion SDE, adding noise, then denoised with the learned score by simulating the reverse SDE, yielding samples from the posterior $P(x_0 \mid x_1)$. This generative state matching (GSM) step is what perturbs initial conditions, corrects intermediate states, and lets trajectories switch between Pangu and Fengwu at each forecast step. A two-stage ensemble-of-expert-denoisers training — one model over all diffusion times, fine-tuned over the interval $[0.6, 1]$ — sharpens the posterior for the chosen $t_0$ without extra inference cost.
What would settle it
Compare the day-10 500 hPa geopotential forecasts of GenEPS and ECMWF ENS against independent radiosonde observations rather than ERA5; if the reported ACC gap (0.679 vs 0.646) narrows to zero or reverses, the apparent skill gain is an artifact of matching the verification dataset.
Extended reading notes
Core claim
The central claim is that sampling three distinct sources of uncertainty — initial condition, model stochasticity, and model formulation — is enough to convert deterministic AI weather models into a forecast system that outperforms a leading operational ensemble. The mechanism is generative state matching: a diffusion prior learned on historical atmospheric states is used, through SDEdit, to replace a forecast field's initial and intermediate states with posterior samples that retain the field's large-scale information while restoring consistency with the historical distribution. This decoupling of states from model-specific bias allows cross-model continuation of trajectories, so two or three models behave like many. The resulting superensemble mean reaches ACC 0.679 at day 10 for Z500 versus 0.646 for ECMWF ENS, and the paper reports better CRPS in early lead times, lower error than ERA5 on station wind observations at day 10, better heat-wave and tropical-cyclone representation than any deterministic AI model, and energy spectra closer to ERA5.
Load-bearing premise
The deciding premise is that pulling every forecast state toward a probability model trained on 2018–2022 ERA5 weather analysis makes the states genuinely more accurate, rather than merely closer to the same dataset that is also used to score the forecasts.
Editorial extensions
If this is right
- Any deterministic AI forecast model gains ensemble capability by adding GSM as a plug-in, with no retraining of the backbone model.
- With only two or three backbone models, cross-model trajectory switching generates novel model behaviors, letting a small pool sample model-formulation uncertainty more densely.
- The superensemble mean beats ECMWF ENS in 10-day Z500 ACC (0.679 vs 0.646; 0.683 with the ECMWF mean included) and in early-lead CRPS, while using far fewer computational resources.
- Ensemble members represent extreme events better than deterministic AI forecasts, with best-member F1 near that of ECMWF ENS for the 2023 North China heat wave and a mean track error of 133 km for Typhoon Doksuri.
Reading between the lines
- Because the same ERA5 prior is both the training target and the verification reference, a neutral test against independent observations (e.g., radiosondes for variables other than 10-m wind) would separate genuine skill from regression to the ERA5 climatology.
- The plug-and-play property implies that any future deterministic model upgrade inherits the ensemble machinery, so the superensemble's skill could track the best backbone model rather than requiring a bespoke probabilistic training pipeline.
- The same state-matching correction could in principle be applied to a physics-based deterministic forecast such as IFS HRES; the paper does not test this, but the decoupling argument suggests it would also benefit from ensemble generation.
- Longer or larger training data for the diffusion prior would likely sharpen extreme-event representation, which the paper names as its main limitation.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces GenEPS, a post-processing framework that trains an unconditional diffusion model on ERA5 as a prior and uses SDEdit-based generative state matching to convert deterministic data-driven weather forecasts into ensembles. The framework also permits switching between different forecast models during integration and combines single-model ensembles, cross-model ensembles, and deterministic forecasts into a 'superensemble.' The authors report a day-10 500 hPa geopotential ACC of 0.679 for GenEPS versus 0.646 for ECMWF ENS, rising to 0.683 when the ECMWF ensemble mean is included, along with improved CRPS, spread behavior, spectral properties, and case studies of a heat wave and Typhoon Doksuri. The central claim is that a plug-and-play generative post-processing system can exceed a major operational ensemble system in medium-range skill.
Significance. If the central claim is established, this would be a practically significant result: it would show that a relatively lightweight, model-agnostic generative post-processing layer can add ensemble capability and improve skill over deterministic machine-learning forecast models. The paper's strengths include the simple and general formulation, the inclusion of an independent station-based evaluation, and a candid discussion of limitations in Section 7. However, the headline result is not yet established because the evaluation is entangled with the training data, lacks statistical rigor, and omits the closest machine-learning ensemble baselines. These issues are addressable with additional experiments, so the work is worth pursuing in revision.
major comments (5)
- [Section 8.1, 8.4, 8.7 and Figure 3c] The headline ACC comparison is potentially conflated with the training/verification data overlap. The diffusion prior is trained on ERA5 for 2018-2022, while ACC, RMSE, and CRPS are verified against ERA5 with a 39-year climatology. SDEdit at t0=0.4 substantially perturbs the forecast field and regenerates it from that ERA5-trained prior, so the reported 0.033 ACC advantage over ECMWF ENS could partly reflect regression toward the verification distribution rather than additional predictive information. I ask for (i) verification of the headline metric against observations or a reanalysis not used in training (e.g., radiosonde Z500 or JRA-55), and (ii) a control experiment in which the same SDEdit protocol is applied with a prior trained on a different dataset or period, or in which non-generative perturbations with similar spatial spectra are used at the same t0.
- [Section 4 and Section 8.1] The skill comparison rests on point estimates from 72 initialization dates with no confidence intervals or significance tests. The day-10 ACC gap of 0.033 could easily arise from sampling variability. Please provide bootstrap confidence intervals for the ACC/RMSE/CRPS differences and a season-by-season breakdown. In addition, the evaluation protocol for the ECMWF ENS baseline is not described; please confirm that the comparison is controlled, i.e., the same dates, the same 1.5-degree grid, the same climatology, and the same scoring code are used for all systems.
- [Section 4 and references [16,17]] GenCast and Fuxi-ENS are cited but never compared against. Because these are the closest machine-learning ensemble baselines, and at least GenCast has been reported to be competitive with or better than ECMWF ENS, the claim that GenEPS provides 'state-of-the-art deterministic and probabilistic weather forecasting skills' is unsupported. A quantitative comparison, or a clear justification for omitting these baselines, is needed before the headline claim can be accepted.
- [Section 3 and Figure 3d] The incremental contribution of the generative components is not isolated. The paper does not compare GenEPS with a simple multi-model average of the deterministic Pangu, Fengwu, and Fuxi forecasts, nor with ensembles generated by adding noise directly to initial conditions. Without such ablations, it is unclear whether the reported improvements come from GSM/SDEdit and cross-model switching or merely from ensemble averaging and multi-model mixing. In addition, the sensitivity of the results to key hyperparameters t0 (Section 8.4), the fine-tuning interval (Section 8.5), and the patch size (Section 8.3) is not reported.
- [Section 5] The only independent observational validation, 10-meter wind speed at 2,168 Chinese stations, shows GenEPS (MAE 0.99 m/s) slightly worse than ERA5 (0.98 m/s), with no significance test for the differences among models. The paper attributes this to inherited ERA5 biases, but that does not alter the fact that the independent evidence does not demonstrate superiority over ERA5 or ECMWF. This should be reconciled with the headline ACC claim, for example by adding station-based verification for upper-air variables such as Z500 or temperature.
minor comments (5)
- [Section 6.2] There is a duplicated and incomplete passage: 'presents a comprehensive analysis of various forecasting models in predicting the track and structure of a tropical cyclone.' This sentence fragment should be removed or rewritten.
- [Section 8.2] The text contains unresolved citation placeholders '(cite)' in the description of the forward and reverse SDEs; these should be replaced with proper references.
- [Section 6.2] The 133-km mean track error is attributed to both GenEPS and Pangu ENS in different sentences; please clarify which product achieved this value and how the track ensemble mean is defined.
- [Section 4 and Figure 3a] The composition of the 240 GenEPS members is not defined in the main text; please specify how many members come from Pangu, Fengwu, Fuxi, and the cross-model ensemble, and how the ensemble mean is computed.
- [General and Section 8.4] There are several typographical issues, including 'median-range' in the keywords, 'choiced' for 'chosen' in Section 8.4, and a possible typo in Equation (6) where an extraneous 'r' appears; please proofread the manuscript.
Circularity Check
The headline ACC gain is verified against ERA5, the same reanalysis used to train the GenEPS diffusion prior; the claimed spectral consistency is likewise a direct training objective.
-
fitted input called prediction
[Abstract; Sections 8.1, 8.4, and 8.7.]
"GenEPS achieves an Anomaly Correlation Coefficient (ACC) of 0.679 for 500hPa geopotential (Z500), exceeding the ECMWF Ensemble Prediction System's (ENS) ACC of 0.646. [...] We posit that ERA5 most accurately represents the objective atmospheric state PDF. [...] The model training utilized data spanning 2018-2022, with 2023 data reserved for testing. [...] The ACC quantifies the spatial correlation between ensemble mean forecast anomalies and ERA5 anomalies relative to climatology mean."
The GenEPS diffusion prior is fit to ERA5 (2018-2022), and SDEdit edits forecasts by adding noise to t0=0.4 and sampling from the reverse SDE learned on that ERA5 distribution. The headline ACC (0.679 vs 0.646) is computed by correlating the edited ensemble-mean forecast anomaly with the ERA5 anomaly (Eq. 6), the same reanalysis used to train the prior. The 'prediction' is thus partly a measure of how closely the output matches its own training distribution, especially since ERA5 is asserted to define the objective atmospheric state PDF. The only independent validation, 10-m wind over 2,168 stations, shows GenEPS at 0.99 m/s MAE merely approaching ERA5's 0.98 m/s, so the ERA5-based superiority over ECMWF ENS is not independently confirmed.
-
fitted input called prediction
[Abstract; Section 8.4 (Figure 7 caption).]
"GenEPS also enhances extreme event representation and produces energy spectra more consistent with ERA5 reanalysis. [...] A conceptual interpretation of the Generalized Score Matching (GSM) framework: atmospheric states are perturbed through forward SDE, followed by trained reverse SDE that align the atmospheric state characteristics with ERA5."
The spectral-consistency result is the direct output of the training objective: the reverse SDE is explicitly trained to align atmospheric state characteristics with ERA5. Reporting that the edited fields have energy spectra closer to ERA5 therefore restates what the model was fitted to do rather than providing an independent verification of forecast quality. The same training/verification overlap that affects the ACC comparison is present in this claimed benefit.
full rationale
GenEPS is not circular in the narrow sense that Eq. 6 equals a fitted parameter, nor does it rely on a load-bearing self-citation chain or an imported uniqueness theorem. The diffusion prior is trained on 2018-2022 ERA5 and evaluated on 2023 ERA5, and the multi-model superensemble uses genuinely out-of-sample deterministic forecasts, so some independent content exists. However, the central headline claims share a training/verification overlap: the unconditional prior is explicitly fit to ERA5, SDEdit is used to force forecast fields toward that ERA5-learned distribution, and both the headline ACC (0.679 vs 0.646) and the spectral-consistency result are measured against ERA5. The independent station-based wind evaluation shows parity with ERA5 (0.99 vs 0.98 m/s) rather than the superiority claimed from the ERA5-verified ACC. With only 72 initialization dates and no confidence interval, the 0.033 ACC gap over ECMWF ENS is not robustly separated from regression toward the verification/training distribution. Score 6 reflects this partial circularity: the ACC 'prediction' is partly a restatement of the fit, but the superensemble construction and the station evaluation provide independent content that prevents a higher score.
Assumptions & free parameters
free parameters (3)
- t0 (SDEdit perturbation time) =
0.4
- Fine-tuning interval for expert denoisers =
t in [0.6, 1]
- Patch size for score matching =
360 x 720 grid points
assumptions (6)
- domain assumption ERA5 reanalysis is the objective atmospheric state probability density function.
- ad hoc to paper Local patch score functions capture sufficient global dynamics because ERA5 is generated by NWP from local processes and patches span one Rossby wave.
- domain assumption SDEdit with an unconditional prior yields the correct posterior P(x0|x1) for forecast states at all lead times.
- ad hoc to paper GSM decouples intermediate forecast states from model-specific systematic biases, making states transferable between different deterministic models.
- domain assumption Five years of ERA5 (2018-2022) adequately represents the atmospheric state distribution, including rare extremes in 2023.
- domain assumption The ECMWF ENS ACC and RMSE values used as benchmarks are computed on identical dates, grid, and climatology.
Cite this review
Pith. "Pith review of Boosting weather forecast via generative superensemble." pith.science (2026). https://pith.science/paper/4ZUCONFP
@misc{pith2026241208377,
author = {Pith},
title = {Pith review of: Boosting weather forecast via generative superensemble},
year = {2026},
howpublished = {\url{https://pith.science/paper/4ZUCONFP}},
note = {Machine review of arXiv:2412.08377}
}
read the original abstract
Accurate weather forecasting is essential for socioeconomic activities. While data-driven forecasting demonstrates superior predictive capabilities over traditional Numerical Weather Prediction (NWP) with reduced computational demands, its deterministic nature and limited advantages over physics-based ensemble predictions restrict operational applications. We introduce the generative ensemble prediction system (GenEPS) framework to address these limitations by randomizing and mitigating both random errors and systematic biases. GenEPS provides a plug-and-play ensemble forecasting capability for deterministic models to eliminate random errors, while incorporating cross-model integration for cross-model ensembles to address systematic biases. The framework culminates in a super-ensemble approach utilizing all available data-driven models to further minimize systematic biases. GenEPS achieves an Anomaly Correlation Coefficient (ACC) of 0.679 for 500hPa geopotential (Z500), exceeding the ECMWF Ensemble Prediction System's (ENS) ACC of 0.646. Integration of the ECMWF ensemble mean further improves the ACC to 0.683. The framework also enhances extreme event representation and produces energy spectra more consistent with ERA5 reanalysis. GenEPS establishes a new paradigm in ensemble forecasting by enabling the integration of multiple data-driven models into a high-performing super-ensemble system.
Reference graph
Works this paper leans on
-
[1]
Advances in Science and Research 10(1), 65–70 (2013)
Perrels, A., Frei, T., Espejo, F., Jamin, L., Thomalla, A.: Socio-economic benefits of weather and climate services in europe. Advances in Science and Research 10(1), 65–70 (2013)
work page 2013
-
[2]
Monthly Weather Review 141(9), 3142–3153 (2013)
Magnusson, L., K¨ all´ en, E.: Factors influencing skill improvements in the ecmwf forecasting system. Monthly Weather Review 141(9), 3142–3153 (2013)
work page 2013
-
[3]
Nature 525(7567), 47–55 (2015)
Bauer, P., Thorpe, A., Brunet, G.: The quiet revolution of numerical weather prediction. Nature 525(7567), 47–55 (2015)
2015
-
[4]
Nature 619(7970), 533–538 (2023)
Bi, K., Xie, L., Zhang, H., Chen, X., Gu, X., Tian, Q.: Accurate medium-range global weather forecasting with 3d neural networks. Nature 619(7970), 533–538 (2023)
2023
-
[5]
arXiv preprint arXiv:2304.02948 (2023)
Chen, K., Han, T., Gong, J., Bai, L., Ling, F., Luo, J.-J., Chen, X., Ma, L., Zhang, T., Su, R., et al.: Fengwu: Pushing the skillful global medium-range weather forecast beyond 10 days lead. arXiv preprint arXiv:2304.02948 (2023)
arXiv 2023
-
[6]
npj Climate and Atmospheric Science 6(1), 190 (2023)
Chen, L., Zhong, X., Zhang, F., Cheng, Y., Xu, Y., Qi, Y., Li, H.: Fuxi: A cas- cade machine learning forecasting system for 15-day global weather forecast. npj Climate and Atmospheric Science 6(1), 190 (2023)
2023
-
[7]
Science 382(6677), 1416–1421 (2023)
Lam, R., Sanchez-Gonzalez, A., Willson, M., Wirnsberger, P., Fortunato, M., Alet, F., Ravuri, S., Ewalds, T., Eaton-Rosen, Z., Hu, W., et al.: Learning skillful medium-range global weather forecasting. Science 382(6677), 1416–1421 (2023)
2023
-
[8]
Wolfram, S., Gad-el-Hak, M.: A new kind of science. Appl. Mech. Rev. 56(2), 18–19 (2003)
work page 2003
Show all 24 references
-
[9]
Annual review of fluid mechanics 52(1), 477–508 (2020)
Brunton, S.L., Noack, B.R., Koumoutsakos, P.: Machine learning for fluid mechanics. Annual review of fluid mechanics 52(1), 477–508 (2020)
2020
-
[10]
Journal of Advances in Modeling Earth Systems 16(6), 2023–004019 (2024)
Rasp, S., Hoyer, S., Merose, A., Langmore, I., Battaglia, P., Russell, T., Sanchez- Gonzalez, A., Yang, V., Carver, R., Agrawal, S., et al.: Weatherbench 2: A benchmark for the next generation of data-driven global weather models. Journal of Advances in Modeling Earth Systems ...
2024
-
[11]
Quarterly Journal of the Royal Meteorological Society: A journal of the atmospheric sciences, applied meteorology and physical oceanography 128(581), 747–774 (2002)
Palmer, T.N.: The economic value of ensemble forecasts as a tool for risk assess- ment: From days to decades. Quarterly Journal of the Royal Meteorological Society: A journal of the atmospheric sciences, applied meteorology and physical oceanography 128(581), 747–774 (2002)
2002
-
[12]
25 years of ensemble forecasting
Buizza, R.: Introduction to the special issue on “25 years of ensemble forecasting”, QJ Roy. Meteor. Soc., 145, 1–11 (2019)
2019
-
[13]
Science 285(5433), 1548–1550 (1999)
Krishnamurti, T., Kishtawal, C.M., LaRow, T.E., Bachiochi, D.R., Zhang, Z., 20 Williford, C.E., Gadgil, S., Surendran, S.: Improved weather and seasonal climate forecasts from multimodel superensemble. Science 285(5433), 1548–1550 (1999)
1999
-
[14]
basic concept
Hagedorn, R., Doblas-Reyes, F.J., Palmer, T.N.: The rationale behind the suc- cess of multi-model ensembles in seasonal forecasting—i. basic concept. Tellus A: Dynamic Meteorology and Oceanography 57(3), 219–233 (2005)
2005
-
[15]
Journal of Advances in Modeling Earth Systems 14(3), 2021–002766 (2022)
Pan, B., Anderson, G.J., Goncalves, A., Lucas, D.D., Bonfils, C.J., Lee, J.: Improving seasonal forecast using probabilistic deep learning. Journal of Advances in Modeling Earth Systems 14(3), 2021–002766 (2022)
2022
-
[16]
arXiv preprint arXiv:2312.15796 (2023)
Price, I., Sanchez-Gonzalez, A., Alet, F., Andersson, T.R., El-Kadi, A., Mas- ters, D., Ewalds, T., Stott, J., Mohamed, S., Battaglia, P., et al.: Gencast: Diffusion-based ensemble forecasting for medium-range weather. arXiv preprint arXiv:2312.15796 (2023)
2023 arXiv
-
[17]
arXiv preprint arXiv:2405.05925 (2024)
Zhong, X., Chen, L., Li, H., Liu, J., Fan, X., Feng, J., Dai, K., Luo, J.-J., Wu, J., Lu, B.: Fuxi-ens: A machine learning model for medium-range ensemble weather forecasting. arXiv preprint arXiv:2405.05925 (2024)
2024 arXiv
-
[18]
In: International Confer- ence on Machine Learning, pp
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsuper- vised learning using nonequilibrium thermodynamics. In: International Confer- ence on Machine Learning, pp. 2256–2265 (2015). PMLR
2015
-
[19]
Advances in neural information processing systems 33, 6840–6851 (2020)
Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840–6851 (2020)
2020
-
[20]
In: International Conference on Learning Representations
Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-based generative modeling through stochastic differential equations. In: International Conference on Learning Representations
-
[21]
arXiv preprint arXiv:2108.01073 (2021)
Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., Ermon, S.: Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073 (2021)
2021 arXiv
-
[22]
Advances in neural information processing systems 36 (2024)
Wang, Z., Jiang, Y., Zheng, H., Wang, P., He, P., Wang, Z., Chen, W., Zhou, M., et al.: Patch diffusion: Faster and more data-efficient training of diffusion models. Advances in neural information processing systems 36 (2024)
2024
-
[23]
arXiv preprint arXiv:2211.01324 (2022)
Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song, J., Zhang, Q., Kreis, K., Aittala, M., Aila, T., Laine, S., et al.: ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 (2022)
2022 arXiv
-
[24]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976–11986 (2022) 21
2022
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.