Pith. sign in

REVIEW 3 major objections 5 minor 22 references

A 20% month-stratified subset of weather data trains a diffusion forecaster as well as the full dataset, and improves ensemble calibration.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Training an autoregressive diffusion weather forecaster on 20% of ERA5 data selected uniformly by calendar month matches full-data CRPS/RMSE and improves the spread-skill ratio on the 2018 test year.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Useful first benchmark of data selection for diffusion weather forecasting, but the SSR headline is not yet identifiable because the training budget is unreported. the 3 major comments →

arxiv 2509.11047 v1 pith:6SBKYYQF submitted 2025-09-14 cs.LG cs.CV

Data-Efficient Ensemble Weather Forecasting with Diffusion Models

Kevin Valencia , Ziyang Liu , Justin Cui This is my paper
classification cs.LG cs.CV
keywords diffusion modelsensemble weather forecastingdata selectionstratified samplingdata efficiencyERA5spread-skill ratioautoregressive forecasting
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that curated data selection can make diffusion-based ensemble weather forecasting much cheaper without losing quality. Using only 20% of the training data, a simple strategy that samples evenly from each calendar month matches the full-data model's accuracy and often produces better-calibrated ensembles. This matters because training autoregressive diffusion models on decades of global weather data is computationally heavy, and high-quality climate data is often scarce or costly. If true, the result opens the door to practical data-efficient training for scientific forecasting models.

Core claim

Stratified time sampling—selecting a fixed number of training examples uniformly from each calendar month—achieves performance comparable to full-data training across the five main ERA5 variables (z500, t850, t2m, u10, v10) while using only 20% of the training data. On the spread/skill ratio (SSR), a calibration metric where values closer to 1 are better, the stratified 20% model matches or slightly beats the full-data model at both 5-day and 10-day lead times, while staying within roughly 3–6% on CRPS and RMSE. The paper argues that temporal representativeness provides a strong domain prior for weather, and that simple static sampling heuristics can serve as a baseline for future adaptive d

What carries the argument

The central mechanism is stratified time sampling: partitioning the training period by calendar month and drawing an equal number of samples from each month, enforcing seasonal coverage. This is applied to an autoregressive 24-hour diffusion model that is rolled out iteratively to produce 10-day ensemble forecasts. The paper's argument depends on this sampling strategy, alongside the fixed 20% data budget and the evaluation metrics CRPS (probabilistic accuracy and sharpness), RMSE (ensemble-mean error), and SSR (spread-skill ratio, calibration).

Load-bearing premise

The comparison assumes that fixing model hyperparameters isolates the effect of data selection, but the paper never states the number of training steps or epochs; if the step budget is the same for all methods, the 20% model sees each sample roughly five times more often, so part of the advantage could come from repetition rather than stratified selection.

What would settle it

Train the full-data and stratified-20% models with matched total optimization steps and matched number of epochs per sample, then compare SSR and CRPS on an independent test year other than 2018 (for example, 2019 or 2020). If the stratified model no longer matches or beats the full-data model on SSR, the paper's central claim about data selection would be weakened.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Training an autoregressive diffusion weather forecaster on 20% of the data can cut storage, preprocessing, and training cost by roughly 80% while preserving forecast accuracy and improving ensemble calibration.
  • Stratified temporal sampling is a strong, simple baseline that future adaptive or model-aware sampling methods should be compared against.
  • The result suggests that for strongly seasonal signals like weather, data coverage across regime space matters more than the total volume of data.
  • Because the 20% models match full-data CRPS and RMSE, data-efficient training is feasible even when data access is restricted or expensive.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported SSR advantage could partly reflect an implicit extra-epochs effect: if the same training-step budget is used for all methods, the 20% model revisits each sample about five times more often, which could improve calibration independent of the sampling strategy—this is not addressed in the paper.
  • Since only the 2018 test year is used, the seasonal-coverage benefit might be exaggerated for a single year; testing on additional years (e.g., 2019–2020) would clarify whether the advantage is robust to interannual variability.
  • The failure of hybrid strategies that combine stratified time with spatial clustering or entropy suggests that heuristics designed on top of temporal stratification can hurt, implying that gains come mostly from even seasonal coverage rather than additional feature-space diversity—an inference that could be tested by ablations.
  • A practical extension would be to combine stratified temporal sampling with online selection based on forecast error or uncertainty, potentially reaching the same performance with even less than 20% data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies data selection for training an autoregressive diffusion model for global ensemble weather forecasting. Using ERA5/WeatherBench data (5.625° resolution), it compares several 20% training-subset strategies—random, k-means, greedy diverse, spatial, herding, and stratified time sampling—against full-data training over 1979–2015, with evaluation on the 2018 test year. Metrics are CRPS, RMSE, and spread/skill ratio (SSR) at 5-day and 10-day lead times. The central claim is that simple month-stratified time sampling matches full-data CRPS/RMSE and improves SSR, despite using only 20% of the training data. The paper explicitly acknowledges the single-test-year limitation and that the evaluated samplers are static heuristics rather than learned or adaptive methods.

Significance. If the central claim is correct, the paper provides a useful and inexpensive baseline for data-efficient training of diffusion-based ensemble weather forecasters. A simple temporal coverage prior that matches or improves calibration at 20% data would be of practical interest and would motivate model-aware sampling. The paper is also honest about several limitations, and the evaluation is not circular: metrics are computed on an independent 2018 test period against ERA5, and no fitted constant is defined in terms of the reported outcomes. However, the empirical support is currently too weak to establish the headline claim: there is no stated optimization budget, no repeated-seed variability information, and only one test year.

major comments (3)
  1. [Section 4.1] The paper states that "the only variable is the subset of training data used," but it never reports the number of training iterations/epochs, batch size, learning-rate schedule, or convergence criterion. If all models are trained for the same number of optimizer steps (the common default), then a 20% subset sees each sample roughly five times more often than the full-data model. The reported SSR improvements (e.g., Table 1: 0.89 vs 0.84 for z500 at 5 days; Table 2: 0.93 vs 0.89 for t850 at 5 days) could then be an increased-epochs-per-sample effect, or an overfitting/regularization artifact, rather than a property of temporal stratification. Because the confound affects all 20% baselines, it also weakens comparisons among sampling methods. Please specify the optimization budget, or better, run controlled comparisons at both equal total updates and equal epochs per sample.
  2. [Tables 1–5] Every table reports a single value per method with no standard deviations, no confidence intervals, and no significance tests, while diffusion training and inference are stochastic and each method appears to be run once. Differences such as Table 1's z500 5-day SSR 0.89 vs 0.84, or Table 4's u10 5-day SSR 0.94 vs 0.92, may be within run-to-run noise. The claim that stratified time "outperforms" full-data training on SSR is not statistically supported. Please provide results over multiple seeds (at least three, preferably more) and report intervals or error bars, and use an appropriate test for paired metric differences.
  3. [Section 5] The authors acknowledge that experiments are limited to a single test year (2018). Since weather exhibits substantial interannual variability, the central claim that stratified 20% training matches or beats full-data training across metrics cannot be established from one year alone. This is not merely a presentation issue: the conclusion is about general forecast skill. Please evaluate on additional independent years (e.g., the 2016–2017 validation years or a multi-year test period) and report per-year and aggregate results.
minor comments (5)
  1. [Section 3.2 / Tables 1–5] The baselines labeled "Spatial" and "Herding" appear in all tables but are described only in one sentence under "Other Baselines." No algorithms, distances, or implementation details are given. Please move them into the main methods or remove them from the tables.
  2. [Appendix A.1] Section 4.5 says full results for ws10 are in Appendix A.1, but the appendix contains tables only for t2m, u10, and v10; no ws10 table appears. Either add the ws10 results or correct the reference.
  3. [Appendix A.2] The appendix states that several hybrid stratified variants were tested but "performance results however were lacking and thus were not included." Omitting the numerical results of experiments that were actually run is a reporting gap; it prevents readers from verifying the claim and risks selective reporting. At minimum, include the results in a supplementary table or specify the quantitative criterion used to exclude them.
  4. [Section 4.1] The diffusion model configuration is described only by reference to prior work. Please state the architecture size, number of diffusion steps, noise schedule, ensemble size at inference, and sampling procedure, since these affect reproducibility and runtime.
  5. [Tables 1–5] The tables are inconsistent in formatting: some entries have two decimals and others one (e.g., Table 1 CRPS 242.66 vs 335.2); the full-data row is labeled "Full Data" in Tables 1–3 and "AR-24h (full data)" in Tables 4–5. Please standardize.

Circularity Check

0 steps flagged

No significant circularity: the paper is an empirical benchmark whose central comparison is evaluated on an independent 2018 test period against ERA5, with static sampling heuristics and no fitted quantity defined in terms of the reported outcomes.

full rationale

This paper contains no derivation chain whose conclusion reduces to its inputs. The central claim is that month-stratified 20% data selection matches or improves on full-data training for an autoregressive diffusion weather forecaster, as measured by CRPS, RMSE, and SSR. These metrics are computed on the 2018 ERA5 test year, which is disjoint from the 1979-2015 training split and the 2016-2017 validation split. The sampling heuristics are static and defined before evaluation: random sampling, k-means on PCA features, greedy diverse selection, and stratified time sampling that 'selects a fixed number of training samples uniformly from each calendar month across the full training period.' No parameter of the sampling rule is fitted to the test-year outcome, and no equation defines the reported metric in terms of the selection criterion. The paper explicitly discloses its main limitation, a single test year (Section 5: 'our experiments are limited to a single test year (2018)'), and Appendix A.2 discloses that hybrid variants were omitted because 'performance results however were lacking.' These are legitimate empirical caveats, not circular reasoning. The self-citations to Andrae et al. [1] and GenCast [14] are used only to adopt a standard AR-24h training setup and benchmark context; they are not invoked as proof of the paper's data-selection findings. The skeptic concern about unequal optimization budgets (e.g., possibly more epochs per sample in the 20% runs) is a question of experimental control and identifiability, not a circular reduction: the reported results could be confounded by training steps, but they are not logically forced by the definition of stratified sampling or by any fitted parameter. Under the stated criteria, the honest finding is no significant circularity with score 0.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim rests on experimental design choices rather than a derivation. The most important listed assumptions are that a single 2018 test year is representative, that one training run per method is enough to rank methods, and that holding training hyperparameters fixed isolates data selection. These are not checked quantitatively in the paper.

free parameters (5)
  • Data budget fraction = 0.20
    All sampling experiments use a manually chosen fixed 20% training budget; the claim of data efficiency is defined by this number, and no sensitivity analysis around it is provided.
  • PCA retained dimensions M = not reported
    K-means baseline projects flattened examples to M dimensions (M much less than D) before clustering, but the exact M is never stated; this choice changes which samples are nearest to cluster centroids.
  • K-means cluster count k = 0.2N
    The number of clusters is set to 20% of the number of training examples so one centroid-nearest sample per cluster yields the budget. The paper does not test alternative cluster counts.
  • Distance and spatial averaging choices = Euclidean/cosine, unspecified averaging
    Greedy diverse and spatial sampling rely on distances between spatially averaged fields and cosine distances, but the exact field averaging and metric are left as examples, so the selected subsets are underdetermined.
  • Diffusion model and training configuration = not reported
    Architecture, diffusion noise schedule, sampling steps, ensemble size, optimizer, learning rate, batch size, and training iterations are fixed but not enumerated, and are attributed to prior work [1,14]. The observed performance ordering depends on all of them.
axioms (5)
  • domain assumption ERA5/WeatherBench at 5.625 degree resolution is a sufficient proxy for global weather forecasting skill.
    Section 3.1 selects this benchmark and five variables; the conclusions about data-efficient training are scoped to this dataset.
  • domain assumption The 2018 test year is representative of forecasting performance.
    All metrics are evaluated only on 2018 (Sections 3.1 and 4.1); the authors acknowledge in Section 5 that a single year may miss longer-term variability.
  • ad hoc to paper One randomly initialized training run per sampling method is enough to rank methods.
    No seeds or multiple runs are reported in Section 4.3, yet the paper concludes stratified time outperforms full data on SSR; this assumes away training noise.
  • domain assumption RMSE, CRPS, and SSR are sufficient and correctly computed evaluation metrics.
    Section 4.2 defines the metrics via citations [6,7]; the SSR target of 1 and ensemble size are not described, and no uncertainty estimates on the scores are given.
  • ad hoc to paper Fixed hyperparameters across data subsets isolate the effect of data selection.
    Section 4.1 states only the data subset changes. If the number of training steps is fixed, the 20% model sees each sample roughly five times more often than full data, confounding selection with optimization budget; the paper does not report training duration.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Data-Efficient Ensemble Weather Forecasting with Diffusion Models." pith.science (2026). https://pith.science/paper/6SBKYYQF

@misc{pith2026250911047,
  author       = {Pith},
  title        = {Pith review of: Data-Efficient Ensemble Weather Forecasting with Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6SBKYYQF}},
  note         = {Machine review of arXiv:2509.11047}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Although numerical weather forecasting methods have dominated the field, recent advances in deep learning methods, such as diffusion models, have shown promise in ensemble weather forecasting. However, such models are typically autoregressive and are thus computationally expensive. This is a challenge in climate science, where data can be limited, costly, or difficult to work with. In this work, we explore the impact of curated data selection on these autoregressive diffusion models. We evaluate several data sampling strategies and show that a simple time stratified sampling approach achieves performance similar to or better than full-data training. Notably, it outperforms the full-data model on certain metrics and performs only slightly worse on others while using only 20% of the training data. Our results demonstrate the feasibility of data-efficient diffusion training, especially for weather forecasting, and motivates future work on adaptive or model-aware sampling methods that go beyond random or purely temporal sampling.

Figures

Figures reproduced from arXiv: 2509.11047 by Justin Cui, Kevin Valencia, Ziyang Liu.

Figure 1
Figure 1. Figure 1: SSR across sampling methods for z500. Stratified time performs best. 4.4. Sampling Discussion Stratified time sampling performs best among the 20% data baselines, likely due to its coverage across all months in the training period. By ensuring that each season is rep￾resented across the 12 months, the model is exposed to a more balanced variety of atmospheric phenomena (e.g., winter storms, summer heatwave… view at source ↗
Figure 2
Figure 2. Figure 2: SSR across different sampling methods for [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 7 linked inside Pith

  1. [1]

    Continuous ensemble weather forecast- ing with diffusion models

    Martin Andrae, Tomas Landelius, Joel Oskarsson, and Fredrik Lindsten. Continuous ensemble weather forecast- ing with diffusion models. InThe Thirteenth International Conference on Learning Representations, 2025. 2, 3

    2025

  2. [2]

    The ECMWF scalability programme: Progress and plans

    Peter Bauer, Tiago Quintino, Nils Wedi, Antonio Bonanni, Marcin Chrust, Willem Deconinck, Michail Diamantakis, Peter D ¨uben, Stephen English, Johannes Flemming, et al. The ECMWF scalability programme: Progress and plans. European Centre for Medium Range Weather Forecasts,

  3. [3]

    Accurate medium-range global weather forecasting with 3d neural networks.Nature, 619:533 – 538,

    Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiao- tao Gu, and Qi Tian. Accurate medium-range global weather forecasting with 3d neural networks.Nature, 619:533 – 538,

  4. [4]

    Flexible dataset distillation: Learn labels instead of images

    Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images. arXiv preprint arXiv:2006.08572, 2020. 2

    Pith/arXiv arXiv 2006

  5. [5]

    Dataset distillation by matching training trajectories

    George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4750–4759, 2022. 2

    2022

  6. [6]

    Why should ensemble spread match the rmse of the ensemble mean?Journal of Hydrometeorology, 15(4):1708–1713, 2014

    Vincent Fortin, Mabrouk Abaza, Francois Anctil, and Raphael Turcotte. Why should ensemble spread match the rmse of the ensemble mean?Journal of Hydrometeorology, 15(4):1708–1713, 2014. 3

    2014

  7. [7]

    Decomposition of the continuous ranked probability score for ensemble prediction systems.Weather and Forecasting, 15(5):559–570, 2000

    Hans Hersbach. Decomposition of the continuous ranked probability score for ensemble prediction systems.Weather and Forecasting, 15(5):559–570, 2000. 3

    2000

  8. [8]

    Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020. 1

    2020

  9. [9]

    Video dif- fusion models.Advances in Neural Information Processing Systems, 35:8633–8646, 2022

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video dif- fusion models.Advances in Neural Information Processing Systems, 35:8633–8646, 2022. 1

    2022

  10. [10]

    Elucidating the design space of diffusion-based generative models.Advances in neural information processing systems, 35:26565–26577, 2022

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models.Advances in neural information processing systems, 35:26565–26577, 2022. 1

    2022

  11. [11]

    Concept-skill transferability-based data selection for large vision-language models.arXiv preprint arXiv:2406.10995, 2024

    Jaewoo Lee, Boyang Li, and Sung Ju Hwang. Concept-skill transferability-based data selection for large vision-language models.arXiv preprint arXiv:2406.10995, 2024. 2

    Pith/arXiv arXiv 2024

  12. [12]

    A core- set selection of coreset selection literature: Introduction and recent advances.arXiv preprint arXiv:2505.17799, 2025

    Brian B Moser, Arundhati S Shanbhag, Stanislav Frolov, Federico Raue, Joachim Folz, and Andreas Dengel. A core- set selection of coreset selection literature: Introduction and recent advances.arXiv preprint arXiv:2505.17799, 2025. 2

    arXiv 2025

  13. [13]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021. 1

    Pith/arXiv arXiv 2021

  14. [14]

    Probabilistic weather forecasting with machine learn- ing.Nature, 637(8044):84–90, 2025

    Ilan Price, Alvaro Sanchez-Gonzalez, Ferran Alet, Tom R Andersson, Andrew El-Kadi, Dominic Masters, Timo Ewalds, Jacklynn Stott, Shakir Mohamed, Peter Battaglia, et al. Probabilistic weather forecasting with machine learn- ing.Nature, 637(8044):84–90, 2025. 2, 3

    2025

  15. [15]

    Weather- bench: a benchmark data set for data-driven weather fore- casting.Journal of Advances in Modeling Earth Systems, 12 (11):e2020MS002203, 2020

    Stephan Rasp, Peter D Dueben, Sebastian Scher, Jonathan A Weyn, Soukayna Mouatadid, and Nils Thuerey. Weather- bench: a benchmark data set for data-driven weather fore- casting.Journal of Advances in Modeling Earth Systems, 12 (11):e2020MS002203, 2020. 2

    2020

  16. [16]

    Score-based generative modeling through stochastic differential equa- tions.arXiv preprint arXiv:2011.13456, 2020

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions.arXiv preprint arXiv:2011.13456, 2020. 1

    Pith/arXiv arXiv 2011

  17. [17]

    Dataset distillation.arXiv preprint arXiv:1811.10959, 2018

    Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation.arXiv preprint arXiv:1811.10959, 2018. 2

    Pith/arXiv arXiv 2018

  18. [18]

    Sustainable ai: Environmental implications, challenges and opportunities

    Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. Sustainable ai: Environmental implications, challenges and opportunities. Proceedings of Machine Learning and Systems, 4:795–813,

  19. [19]

    Icons: Influence consensus for vision-language data selection.arXiv preprint arXiv:2501.00654, 2024

    Xindi Wu, Mengzhou Xia, Rulin Shao, Zhiwei Deng, Pang Wei Koh, and Olga Russakovsky. Icons: Influence consensus for vision-language data selection.arXiv preprint arXiv:2501.00654, 2024. 2

    arXiv 2024

  20. [20]

    Dataset condensation with differ- entiable siamese augmentation

    Bo Zhao and Hakan Bilen. Dataset condensation with differ- entiable siamese augmentation. InInternational Conference on Machine Learning, pages 12674–12685. PMLR, 2021. 2

    2021

  21. [21]

    Dataset condensation with gradient matching.arXiv preprint arXiv:2006.05929, 2020

    Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching.arXiv preprint arXiv:2006.05929, 2020. 2

    Pith/arXiv arXiv 2006

  22. [22]

    Ladcast: A latent dif- fusion model for medium-range ensemble weather forecast- ing.arXiv preprint arXiv:2506.09193, 2025

    Yilin Zhuang and Karthik Duraisamy. Ladcast: A latent dif- fusion model for medium-range ensemble weather forecast- ing.arXiv preprint arXiv:2506.09193, 2025. 2 5 Data-Efficient Ensemble Weather Forecasting with Diffusion Models Supplementary Material A. Appendix A.1. Full Results Here we show the tables and results for our sampling meth- ods ont2m,u 10,v...

    Pith/arXiv arXiv 2025

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.