REVIEW 4 major objections 5 minor 19 references
CRPS-LAM shows that limited-area probabilistic forecasts can be sampled in one forward pass per ensemble member, matching diffusion-model error and improving long-lead calibration.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 10:26 UTC pith:UTQGH5KR
load-bearing objection Useful regional adaptation of CRPS-based ensemble forecasting with a real speedup, but the joint-distribution claim is supported only by marginals and a heuristic. the 4 major comments →
CRPS-LAM: Probabilistic Regional Weather Forecasting with Continuous Ranked Probability Score
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that matching marginal distributions per grid point and variable—summed into a fair CRPS loss—is sufficient, when combined with one shared latent noise vector and a convolutional weight-sharing backbone, to generate physically plausible regional ensemble forecasts with diffusion-level error. Ensemble members are produced by sampling a 32-dimensional Gaussian noise vector, passing it through a linear layer into conditional normalization layers, and running the CNN/GNN backbone once; the same network is rolled out autoregressively to 57 h lead times. The paper reports that this scheme yields sampling speeds on par with deterministic models, about 39x faster than the diffus
What carries the argument
The fair CRPS estimator: for each grid point and variable, the loss is the average absolute deviation between ensemble members and the observation minus half the average pairwise deviation among members, making the training objective an unbiased proper scoring rule. Stochasticity is carried by one 32-dimensional noise vector, transformed by a linear layer and injected through conditional normalization layers, so every output field in a member is a function of the same draw. The hybrid CNN/GNN backbone shares weights across spatial locations, which the paper argues promotes spatially coherent fields even though the loss itself is per-location and per-variable.
Load-bearing premise
The load-bearing premise is that conditioning every predicted value on one shared noise vector, together with convolutional weight sharing, is enough to reproduce the true joint spatial and cross-variable distribution of the atmosphere; the training loss itself only enforces correct marginal distributions at each grid point and variable.
What would settle it
Take a fixed initialization, draw 100 latent vectors, and compute a joint diagnostic that cannot factor over grid points—for example, the rank histogram of the minimum of standardized anomalies at two distant stations, or of precipitation accumulated over a catchment. If the ensemble members are pointwise well calibrated but these joint ranks are strongly non-uniform, the shared-noise mechanism has not captured the joint distribution the paper relies on; conversely, uniform joint ranks support the claim.
If this is right
- Probabilistic regional forecasts can be sampled at near-deterministic cost: about 0.5 seconds per ensemble member on one A100 GPU, with members parallelizable through batched inference.
- CRPS-LAM matches the RMSE and CRPS of the diffusion-based baseline on MEPS while showing less underdispersion at longer lead times, so calibration does not have to be sacrificed for speed.
- The shared-noise CRPS recipe transfers from global forecasting models to the limited-area setting, with no need for per-level or per-variable loss weighting.
- Autoregressive rollout to 57 h produces forecasts with fine-scale structure, retaining more high-frequency detail than a graph-based latent-variable model, though not as much as diffusion.
Where Pith is reading between the lines
- Editorial extension: the 32-dimensional latent is the only source of ensemble diversity; varying the latent dimension or adding spatially structured noise is the clearest knob to test and likely to change performance if the joint-distribution claim holds.
- Editorial extension: the paper's evidence for joint fidelity is qualitative—visual fields and energy spectra; a quantitative test such as variogram-based rank histograms or joint exceedance probabilities over connected regions would settle whether marginal calibration is enough.
- Editorial extension: training uses only two autoregressive rollout steps; extending to longer rollouts is a direct next experiment, since operational forecasts beyond 57 h may expose accumulated artifacts from the shared noise vector.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CRPS-LAM, a probabilistic limited-area weather forecasting model. The architecture is a hybrid CNN/GNN backbone largely inherited from Diffusion-LAM, but training replaces diffusion with a fair CRPS objective evaluated per grid point and variable (Eq. 3). A single 32-dimensional Gaussian noise vector z is injected via conditional normalization layers; ensemble members are generated by sampling z and running a single forward pass. Experiments on the MEPS dataset at 10 km resolution compare CRPS-LAM to Diffusion-LAM and Graph-EFM using RMSE, CRPS, spread-skill ratio, energy spectra, and qualitative examples. The paper claims competitive deterministic skill and calibration with roughly 39x faster sampling than diffusion, while retaining fine-scale detail and physically consistent structures.
Significance. If the claims hold, the paper offers a practical speed-quality trade-off for probabilistic limited-area forecasting: it suggests that a marginal CRPS objective with a shared latent can match diffusion-model accuracy at a fraction of sampling cost. The use of the unbiased fair CRPS estimator is methodologically sound, and the autoregressive rollout evaluation procedure is standard in this literature. The reproducibility intent is positive (code and data links are provided, though code is not yet public). However, the strongest claims — that the shared latent induces a correct joint spatial/cross-variable distribution and that forecasts are 'well-calibrated' — are not supported by the evidence presented. The paper's own appendix discloses training collapse to near-deterministic forecasts, and no quantitative joint-distribution metric is reported. These gaps are load-bearing rather than cosmetic, because the value of a probabilistic LAM lies precisely in providing mutually consistent ensemble scenarios.
major comments (4)
- [Section 2, Eq. (3)] The loss in Eq. (3) is a sum of per-grid-point, per-variable univariate CRPS terms; it constrains only the marginal distributions at each location and variable. The following sentence that 'all outputs depend on the same noise vector z to correctly capture the joint forecast distribution' is an assertion, not a demonstrated property. All reported diagnostics (RMSE, CRPS, SSR, per-variable energy spectra) are marginal or single-field; there is no multivariate rank histogram, no multivariate CRPS, no cross-variable correlation check, and no quantitative spatial-coherence measure for ensemble members. Since the central value of a probabilistic LAM is to provide physically consistent scenarios, this missing validation is the weakest load-bearing point. Please add a quantitative joint-distribution evaluation, or explicitly scope the claims to marginal calibration.
- [Appendix B (Experiment Details)] Appendix B discloses that during training the model 'sometimes collapses into producing near-deterministic forecasts, effectively minimizing the mean absolute error of each ensemble member while neglecting the latent variable.' This directly concerns the mechanism by which the joint distribution is supposed to be generated. The manuscript does not state whether the final model exhibits residual collapse, how the issue was detected, or how often it occurred across runs. Since the method depends on z being actively used to create ensemble spread, the final model's sensitivity to z (e.g., output variance as a function of z, or effective ensemble spread) should be reported. Without this, the reader cannot tell whether the reported ensemble members are mostly deterministic outputs with small perturbations.
- [Section 3, Fig. 3 and Fig. 10] The headline claims of 'comparable' errors and 'well-calibrated' forecasts are based on mean curves without error bars, multiple seeds, or significance testing. Per-variable SSR results in Fig. 10 often deviate substantially from 1 (e.g., radiation variables nlwrs_0 and nswrs_0), yet no acceptable calibration tolerance is stated. Please provide confidence intervals (e.g., bootstrap across test cases or across model seeds) and report the number of independent initializations used. Currently, the reader cannot judge whether differences between CRPS-LAM and Graph-EFM are meaningful.
- [Section 3 (speed comparison)] The 'approximately 39x faster' claim is presented without specifying the number of diffusion solver steps used for Diffusion-LAM, whether inference was batched, or whether timings include data transfer and model compilation. The parenthetical 'depending on the number of solver steps' makes the number difficult to interpret. Since sampling speed is a central contribution, please provide the precise timing protocol: hardware, batch size, solver steps, and whether the comparison is per ensemble member or for a full ensemble.
minor comments (5)
- [Eq. (1)-(2)] The phrase 'independent identical random variable' should be 'independent and identically distributed.' Also clarify that the double sum in Eq. (2) includes diagonal terms that vanish, and state that N >= 2 is required for the fair CRPS estimator.
- [Introduction and Method] The abstract and introduction describe a 'hybrid CNN/GNN architecture,' but the method section only describes a U-Net with conditional normalization and MLP blocks. The GNN component (inherited from Larsson et al.) is not described; please specify what parts are convolutional and what parts are graph-based, or adjust the terminology.
- [Figure 3 caption] The caption refers to 'normalized RMSE' and 'normalized CRPS' but does not define the normalization. State whether the normalization is per-variable standard deviation, range, or something else, and include this detail in the text or appendix.
- [Section 3 / Fig. 4] For the energy spectra, clarify whether they are computed on ensemble mean fields, individual members, or both. Also note whether the 10 km subsampling cutoff is consistently applied to all models when interpreting wavenumbers above 10^2.
- [Appendix B, Table 2] The training schedule lists 200 epochs at LR 1e-5 with autoregressive steps 2. Please state whether early stopping was used and how the final checkpoint was selected, especially given the reported training instability.
Circularity Check
No significant circularity: the CRPS objective is an external proper scoring rule, training and evaluation use separate MEPS data, and the joint-distribution claim is an explicitly stated assumption rather than a derived result.
full rationale
The paper's derivation chain is not circular. Eq. (3) defines the fair CRPS loss per grid point and variable, and the model is trained on MEPS training data; RMSE, CRPS, SSR and energy spectra are computed on held-out test forecasts, so the evaluation does not reduce to the training objective or to fitted values. The claimed 39x speedup is an empirical runtime comparison, not a consequence of the loss. The central probabilistic claim is explicitly hedged: 'While Eq. (3) can be minimized by only matching marginal distributions for each location and variable, similarly to [Alet et al., 2025] we rely on the fact that all outputs depend on the same noise vector z to correctly capture the joint forecast distribution.' This is an unquantified assumption about joint fidelity, not a circular reduction—it is a correctness/validation limitation, flagged here because the paper itself notes the objective only constrains marginals and Appendix B discloses near-deterministic collapse during training. Self-citations (Larsson et al. 2025 for the backbone and MEPS metrics; Oskarsson et al. 2024 for the training schedule) are used for architecture, dataset, and baselines, not to establish the paper's conclusions; the MEPS dataset is openly available and the baselines are external comparison methods. No uniqueness theorem or prior result by the same authors is invoked to force the model choice. Hence no step reduces by construction to its inputs; the only minor self-citation is the Diffusion-LAM backbone, which is not load-bearing for the paper's central contribution.
Axiom & Free-Parameter Ledger
free parameters (4)
- Latent noise dimensionality =
32
- Training ensemble size N for fair CRPS =
not reported
- Autoregressive training steps =
2
- Spatial subsampling factor =
every 4th grid point (10 km)
axioms (4)
- domain assumption MEPS analysis fields are adequate ground truth for forecast skill evaluation.
- ad hoc to paper A single 32-dimensional Gaussian noise vector is sufficient to represent forecast uncertainty across all grid points and variables.
- domain assumption Autoregressive rollout of 3 h steps to 57 h remains stable and approximately Markovian.
- standard math The fair CRPS estimator is an unbiased estimator of CRPS.
invented entities (1)
-
Single 32-dimensional latent noise vector z
no independent evidence
read the original abstract
Limited-Area Models (LAMs) enable weather forecasting over regional domains at higher resolutions than what is computationally feasible for global models. At such high resolutions, machine learning approaches for weather prediction increasingly rely on ensemble methods to produce probabilistic forecasts. However, existing machine learning LAMs are not scalable due to relying on computationally costly diffusion models or inefficient graph neural networks. We tackle this by introducing a new hybrid CNN/GNN architecture, tailored to the LAM weather forecasting problem. Using this architecture, we construct the DET-LAM deterministic model, producing LAM forecasts both more efficiently and accurately than its graph-based competitor. We then tackle the ensemble forecasting problem, by using this architecture as a backbone for the generative model CRPS-LAM. CRPS-LAM is trained using a Continuous Ranked Probability Score (CRPS) objective, enabling efficient training and sampling in a single forward pass. This yields a speedup of $\approx \times 39$ compared to diffusion-based baselines. We evaluate our approach on regional domains in northern Europe, demonstrating that CRPS-LAM produces skillful and well-calibrated forecasts across a range of atmospheric variables.
Figures
Reference graph
Works this paper leans on
-
[1]
HRRRCast : a data-driven emulator for regional weather forecasting at convection allowing scales
Daniel Abdi, Isidora Jankov, Paul Madden, Vanderlei Vargas, Timothy A Smith, Sergey Frolov, Montgomery Flora, and Corey Potvin. HRRRCast : a data-driven emulator for regional weather forecasting at convection allowing scales. arXiv preprint arXiv:2507.05658, 2025
Pith/arXiv arXiv 2025
-
[2]
Andersson, Jacklynn Stott, Remi Lam, Matthew Willson, Alvaro Sanchez-Gonzalez, and Peter Battaglia
Ferran Alet, Ilan Price, Andrew El-Kadi, Dominic Masters, Stratis Markou, Tom R. Andersson, Jacklynn Stott, Remi Lam, Matthew Willson, Alvaro Sanchez-Gonzalez, and Peter Battaglia. Skillful joint probabilistic weather forecasting from marginals, 2025. URL https://arxiv.org/abs/2506.10772
Pith/arXiv arXiv 2025
-
[3]
Accurate medium-range global weather forecasting with 3d neural networks
Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium-range global weather forecasting with 3d neural networks. Nature, 619 0 (7970): 0 533--538, 2023
2023
-
[4]
Boris Bonev, Thorsten Kurth, Ankur Mahesh, Mauro Bisson, Jean Kossaifi, Karthik Kashinath, Anima Anandkumar, William D. Collins, Michael S. Pritchard, and Alexander Keller. FourCastNet 3: A geometric approach to probabilistic machine-learning weather forecasting at scale, 2025. URL http://arxiv.org/abs/2507.12144
Pith/arXiv arXiv 2025
-
[5]
Adaspeech: Adaptive text to speech for custom voice, 2021
Mingjian Chen, Xu Tan, Bohan Li, Yanqing Liu, Tao Qin, Sheng Zhao, and Tie-Yan Liu. Adaspeech: Adaptive text to speech for custom voice, 2021. URL https://arxiv.org/abs/2103.00993
Pith/arXiv arXiv 2021
-
[6]
C. A. T. Ferro. Fair scores for ensemble forecasts. Quarterly Journal of the Royal Meteorological Society, 140 0 (683): 0 1917--1923, 2014. doi:https://doi.org/10.1002/qj.2270. URL https://rmets.onlinelibrary.wiley.com/doi/abs/10.1002/qj.2270
doi:10.1002/qj.2270 1917
-
[7]
Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, pages 359--378, 2007
2007
-
[8]
Elucidating the design space of diffusion-based generative models
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Proc. NeurIPS, 2022
2022
-
[9]
Learning skillful medium-range global weather forecasting
Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, et al. Learning skillful medium-range global weather forecasting. Science, 382 0 (6677): 0 1416--1421, 2023
2023
-
[10]
Simon Lang, Mihai Alexe, Matthew Chantry, Jesper Dramsch, Florian Pinault, Baudouin Raoult, Mariana C. A. Clare, Christian Lessig, Michael Maier-Gerber, Linus Magnusson, Zied Ben Bouallègue, Ana Prieto Nemesio, Peter D. Dueben, Andrew Brown, Florian Pappenberger, and Florence Rabier. AIFS -- ECMWF 's data-driven forecasting system, 2024 a . URL https://ar...
Pith/arXiv arXiv 2024
-
[11]
Simon Lang, Mihai Alexe, Mariana C. A. Clare, Christopher Roberts, Rilwan Adewoyin, Zied Ben Bouallègue, Matthew Chantry, Jesper Dramsch, Peter D. Dueben, Sara Hahner, Pedro Maciel, Ana Prieto-Nemesio, Cathal O'Brien, Florian Pinault, Jan Polster, Baudouin Raoult, Steffen Tietsche, and Martin Leutbecher. AIFS-CRPS : Ensemble forecasting using a model trai...
Pith/arXiv arXiv 2024
-
[12]
Simon Lang, Martin Leutbecher, and Pedro Maciel. A multi-scale loss formulation for learning a probabilistic model with proper score optimisation, 2025. URL http://arxiv.org/abs/2506.10868
Pith/arXiv arXiv 2025
-
[13]
Diffusion-lam: Probabilistic limited area weather forecasting with diffusion
Erik Larsson, Joel Oskarsson, Tomas Landelius, and Fredrik Lindsten. Diffusion-lam: Probabilistic limited area weather forecasting with diffusion. In ICLR 2025 Workshop on Tackling Climate Change with Machine Learning, 2025. URL https://www.climatechange.ai/papers/iclr2025/36
2025
-
[14]
Graph-based neural weather prediction for limited area modeling
Joel Oskarsson, Tomas Landelius, and Fredrik Lindsten. Graph-based neural weather prediction for limited area modeling. In NeurIPS 2023 Workshop on Tackling Climate Change with Machine Learning, 2023
2023
-
[15]
Probabilistic weather forecasting with hierarchical graph neural networks
Joel Oskarsson, Tomas Landelius, Marc Peter Deisenroth, and Fredrik Lindsten. Probabilistic weather forecasting with hierarchical graph neural networks. In Advances in Neural Information Processing Systems, volume 37, 2024
2024
-
[16]
Adewoyin, Peter Dueben, and Ritabrata Dutta
Lorenzo Pacchiardi, Rilwan A. Adewoyin, Peter Dueben, and Ritabrata Dutta. Probabilistic forecasting with generative networks via scoring rule minimization. 25 0 (45): 0 1--64, 2024. ISSN 1533-7928. URL http://jmlr.org/papers/v25/23-0038.html
2024
-
[17]
Kilometer-scale convection allowing model emulation using generative diffusion modeling, 2024
Jaideep Pathak, Yair Cohen, Piyush Garg, Peter Harrington, Noah Brenowitz, Dale Durran, Morteza Mardani, Arash Vahdat, Shaoming Xu, Karthik Kashinath, and Michael Pritchard. Kilometer-scale convection allowing model emulation using generative diffusion modeling, 2024. URL https://arxiv.org/abs/2408.10958
Pith/arXiv arXiv 2024
-
[18]
Probabilistic weather forecasting with machine learning
Ilan Price, Alvaro Sanchez-Gonzalez, Ferran Alet, Tom R Andersson, Andrew El-Kadi, Dominic Masters, Timo Ewalds, Jacklynn Stott, Shakir Mohamed, Peter Battaglia, et al. Probabilistic weather forecasting with machine learning. Nature, 637 0 (8044): 0 84--90, 2025
2025
-
[19]
Score-based generative modeling through stochastic differential equations
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021
2021
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.