Pith. sign in

REVIEW 3 major objections 4 minor 28 references

Do AI Forecast Ensembles Sample the Correct Conditional Distribution?

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AI forecast ensembles can pass every marginal test while getting the joint spatial structure wrong—worse than climatology—and more training data will not fix it.

desk verdict A genuinely useful empirical finding about marginal-joint decoupling in AI forecast ensembles, but the 'structural inadequacy' conclusion overreaches because the Lorenz-96 control conditions on a deterministic target. read the letter →

arxiv 2608.08954 v1 pith:V5ONQG5M submitted 2026-08-09 physics.ao-ph cs.AIcs.LGstat.APstat.ML

Lucas J. Howard , Elizabeth A. Barnes This is my paper · ORCID
classification physics.ao-phcs.AIcs.LGstat.APstat.ML
keywords ensemble-forecastingdiffusion-modelsconditional-distributionvariogram-scoreCRPSjoint-calibrationcoastal-sea-levelLorenz-96
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Ensemble forecasting is only honest if its members sample the full conditional distribution of outcomes, including how locations vary together. This paper tests that joint requirement for an AI forecast model and finds it fails in a specific way: a diffusion model trained on reanalysis to predict subseasonal coastal sea level is skillful at each of eight tide gauge stations individually, but its spatial correlations are worse than drawing randomly from the historical record. The paper shows this marginal-vs-joint decoupling is invisible to the energy score but exposed by the variogram score, and that it persists in Lorenz-96 experiments from 0.7 to 170 equivalent years of training data. Because a linear baseline reproduces the gap and a dynamical ensemble does not, the authors conclude the joint miscalibration is a structural property of learned emulators rather than a data shortage or a quirk of diffusion models. This matters because standard verification and training metrics would certify such forecasts as good while their spatial risk information is actively misleading.

What carries the argument

The load-bearing object is the denoising diffusion probabilistic model (DDPM)—a generative model that learns to reverse a noise process in order to draw samples from a learned conditional distribution—used here to produce 500-member ensembles of sea surface height anomalies at eight stations simultaneously, conditioned on a 131-dimensional vector of ocean and atmosphere predictors. The evaluation machinery is the trio of proper scoring rules: CRPS for marginal skill, energy score for multivariate skill (shown to be insensitive to joint structure), and variogram score with power $p=0.5$, which is specifically sensitive to the dependence structure and is the detector of the failure. The shuffle-based permutation decomposition separates each score's joint contribution by permuting ensemble members across stations, preserving marginals while destroying spatial correlation. The Lorenz-96 system, its linear multivariate normal baseline, and the dynamical-versus-deterministic emulator comparisons are the controls that convert a single-model observation into the claim of structural inadequacy.

What would settle it

Retrain the Lorenz-96 DDPM with an additional variogram-score-based penalty or with a covariance-matching postprocessor; if the CRPS-variogram gap closes without reducing CRPS, the paper's claim that the joint failure is a structural property of learned emulators, rather than a property of the training objective, would be refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that a trained probabilistic AI forecast ensemble can have positive marginal skill and negative joint skill at the same time. For the eight-station coastal sea level diffusion model, CRPS skill is positive at all leads of 2–12 weeks while variogram score skill is negative at all leads, meaning the ensemble's cross-station spatial structure is worse than a climatological draw. A shuffle-based permutation decomposition shows why standard metrics miss this: the energy score is dominated by marginal contributions, so the DDPM's joint structure barely differs from climatology under ES, whereas the variogram score detects a large, significantly non-climatological joint contribution that is still miscalibrated. Lorenz-96 experiments demonstrate that the CRPS-variogram gap does not close with training data out to 170 equivalent years and is shared by a linear multivariate normal baseline. Comparing ensemble types at matched spread, a dynamical ensemble shows no gap while a deterministic AI emulator does, which the authors interpret as evidence that the failure is intrinsic to data-driven emulation rather than to ensemble forecasting in chaotic systems.

Load-bearing premise

The 'structural, won't improve with data' conclusion assumes the idealized Lorenz-96 system reproduces the same joint-calibration failure as the real coastal sea level forecast, and that the climatological reference pool from the training period correctly defines what 'worse than climatology' means in the test period.

Editorial extensions

If this is right

  • Standard marginal verification is insufficient: a forecast can post positive CRPS skill at every station and positive short-lead energy score while its joint spatial structure is worse than climatology.
  • Adding training data is not a guaranteed remedy for joint miscalibration in learned emulators; the CRPS-variogram gap persists through 170 equivalent years in the idealized experiments.
  • Deterministic AI emulators used with perturbed initial conditions inherit the joint calibration deficit, so downstream users should not assume ensemble spread encodes correct spatial uncertainty.
  • Variogram-score-style dependence checks should be part of routine evaluation and training of AI forecast ensembles, since CRPS and energy score can certify the wrong distribution.
  • Dynamical ensembles retain a role as a complement or reference for AI forecasts, because they do not show the same joint failure mode.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the shuffle-decomposition diagnostic suggests a concrete fix the paper does not test—training with a variogram-score auxiliary loss or post-processing the ensemble with a copula or correlation reordering. If that closes the gap without eroding CRPS, the 'structural' conclusion would need to be softened to 'structural under current training objectives.'
  • Inference: the same decoupling likely affects AI weather emulators used operationally, where users extract spatial products such as storm surge, wind energy, or hydrology; the paper notes consistency with data-assimilation covariance errors, but the verification implication extends beyond sea level.
  • Inference: because the linear MVN baseline shows the same insensitivity to data volume, the bottleneck may be the conditional-mean or regression flavor of the learned mapping rather than neural-network capacity—a testable hypothesis distinguishing diffusion, flow-based, and GAN ensembles of comparable marginal skill.
  • Inference: the per-pair variogram attribution implies a targeted postprocessing route: instead of global calibration, correct only the over-correlated pairs (for example Atlantic City–Battery); the heterogeneity the paper documents suggests a low-rank correlation repair could recover most joint skill.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper trains a denoising diffusion probabilistic model (DDPM) to predict weekly coastal sea-level anomalies at eight US East Coast tide gauges, conditioned on ocean-atmosphere predictors, and evaluates 500-member ensembles on a locked 2022-2025 test set against a climatological reference. Using CRPS, energy score, and variogram score, the authors report positive marginal CRPS skill at all lead times but negative variogram-score skill, and a shuffle-based decomposition indicates that the energy score is largely insensitive to the joint miscalibration. To test whether the gap closes with more data, they run Lorenz-96 experiments with training volumes equivalent to 0.7-170 years and find the CRPS-versus-variogram gap persists for both the DDPM and a multivariate-normal baseline. A dynamical ensemble shows a VS/CRPS ratio near unity, while a deterministic emulator reproduces the gap. The paper concludes that the joint-calibration failure is a structural inadequacy of learned emulators that more data cannot remedy.

Significance. The empirical finding that a probabilistic AI ensemble can be marginally skillful yet jointly miscalibrated is important and timely, and the evaluation design using proper scoring rules, block-bootstrap confidence intervals, and a locked test set is a strength. The pair-level attribution of the variogram-score deficit is also informative. If the structural conclusion survives revision with a nondegenerate idealized target, this would be a valuable caution for the growing use of generative AI forecast ensembles. The main weakness is that the Lorenz-96 experiments, which carry the load of the 'structural' claim, condition on exact initial states of a deterministic system and therefore do not by themselves support the generalization to realistic, nondegenerate conditional forecast problems.

major comments (3)
  1. [2.3, 3.2.1] The Lorenz-96 experiments condition on the exact initial state of a deterministic ODE (Section 2.3: the system is integrated with a fourth-order Runge-Kutta scheme with no stochastic forcing, and the DDPM is 'conditioned on the initial state'). The true conditional forecast distribution is therefore a point mass. A probabilistic generative model must add variance that the target does not have, and the MVN baseline's residual covariance is constant and cannot shrink to zero with more data. The persistence and plateau of the CRPS-versus-VS gap with training volume (Figures 3a-3d) is thus an expected property of any stochastic model class on a degenerate target, not evidence that learned emulators cannot represent realistic joint conditional distributions with nonzero, state-dependent spread. This undermines the abstract and Conclusions 2 and 4, which claim a structural inadequacy that is insensitive to data volume. I recommend adding stochastic forcing to the L96 system, conditioning on noisy or partial observations so the true conditional distribution is nondegenerate, or explicitly restricting the claim to degenerate targets.
  2. [3.1, Figure 1] The abstract states 'positive skill at every station and lead time marginally,' but the only displayed CRPS result (Figure 1a) is the station-mean CRPS skill. No per-station CRPS values are shown in the main text. If per-station results are in the supplement, cite them explicitly; otherwise the 'every station' claim is unsupported and the text should be qualified to the station-mean result.
  3. [3.1, Figure 1c] The text states that variogram-score skill is 'negative at all lead times' and uses this to conclude that the joint spatial structure is 'worse than a climatological draw,' but it does not state whether the negative skill is statistically significant at each lead. Because block-bootstrap confidence intervals are shown, the authors should state explicitly which leads have 95% CIs excluding zero. The Figure 1 caption also says the DDPM achieves positive ES skill at all leads, which contradicts the text reporting negative (though not significant) ES skill at 8 and 12 weeks; the caption should be corrected.
minor comments (4)
  1. [Throughout] There are numerous typographical and encoding issues, including 'submitted toGeophysical Research Letters' (missing space), 'Ni˜ no 3.4' (broken tilde), 'variogramscore' in Section 3.2.1 (missing space), and several formatting artifacts in the reference list; these should be cleaned up before resubmission.
  2. [Figure 3] Figure 3 uses two x-axes with duplicated labels ('1 yr 3 yr 7 yr10 yr...' appears twice); simplifying the axis labeling would improve readability.
  3. [2.3] Section 2.3 reports 'approximately 1.7 Lyapunov times' at 1 TU, but the Lyapunov time in model time units is not defined; please specify the Lyapunov time so the reader can interpret the predictability regimes.
  4. [2.2, 3.1] The climatological reference is drawn from the training period (1993-2018) and evaluated on 2022-2025; any trend or mean-state change between periods will affect the absolute skill scores. The relative decoupling of CRPS and VS is not affected by this choice, but the phrase 'worse than climatological draws' should be qualified as 'worse than a training-period climatological draw.'

Circularity Check

0 steps flagged · score 2.0 of 10

No circularity: the evaluation is a locked-test proper-score comparison; the only self-citation is background L96 usage and is not load-bearing.

full rationale

The paper's central derivation is an empirical evaluation, not a derivation from fitted parameters. The DDPM is trained on 1993-2018 and evaluated on a locked 2022-2025 test set; skill scores use a climatological draw as reference, and all confidence intervals come from block bootstrap. The CRPS/VS gap is a measured outcome, not an identity: there is no equation in the paper that makes VS skill negative by construction, and the shuffle decomposition (Eq. 1) is a diagnostic, not a fitted prediction. The spread-matched sigma_ic is a fairness adjustment for comparing dynamical and AI ensembles, not a parameter whose value is then 'predicted.' The Lorenz-96 experiments use a standard chaotic testbed; the MVN baseline is fit to the same data and exhibits the same gap, but this is an experimental control, not a circular reuse of the target result. The claimed 'structural inadequacy' is an inference from the persistence of the gap with training volume. A critic could argue that conditioning on exact initial states of a deterministic ODE makes the true conditional distribution a point mass, so the plateau is an artifact of the experimental design rather than evidence about real nondegenerate targets; that is a threat to external validity, not a circular reduction. The only self-citation is L. J. Howard et al. (2024) in a list of prior L96 usage (Section 2.3); it is background and not load-bearing. No fitted quantity is renamed as a prediction, and no uniqueness theorem from the authors' prior work is invoked. Therefore no circular step can be exhibited, and the circularity burden is low.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on proper scoring rules, a climatological reference, and the transferability of Lorenz-96 experiments. No new physical entities or parameters are introduced; the fitted values are methodological choices that affect the magnitude of the reported gap but not the existence of the gap. The most consequential assumption is the L96-to-Earth transferability.

free parameters (3)
  • sigma_ic for spread matching = 0.158 (climatological SD units)
    Calibrated by grid search in Section 3.2.2 so dynamical and deterministic ensembles match the DDPM's average ensemble spread at lead=20. It is a fairness adjustment, not a hidden predictor, but it is a fitted number used in the comparison.
  • EOF mode truncations = 15 SSH, 15 SST, 10 wind
    Chosen in Section 2.2 to retain 75%, 75%, and 71% of variance respectively; these are hand-chosen compression settings that could influence model quality.
  • Variogram score power p = 0.5
    Selected as the recommended default from Scheuerer and Hamill (2015); the magnitude of the VS skill gap depends on this metric choice.
assumptions (5)
  • standard math CRPS, energy score, and variogram score are proper scoring rules, with VS more sensitive to joint dependence structure.
    The evaluation framework follows Gneiting and Raftery (2007), Gneiting and Katzfuss (2014), and Scheuerer and Hamill (2015); the paper does not prove these properties.
  • domain assumption The Lorenz-96 system with N=40, F=8 is chaotic and its conditional forecast problem is representative of the coastal sea level forecasting problem.
    Section 2.3 uses L96 to draw conclusions about structural inadequacy of learned emulators; this transferability is assumed, not established.
  • domain assumption Climatological draws from the training-period anomaly pool are an appropriate reference forecast for computing skill scores.
    Section 2.2 defines the reference as 500 samples stratified by calendar week; if the test period's climatology shifted, negative VS skill could partly reflect nonstationarity.
  • domain assumption EOF compression of predictors preserves the joint dependence structure relevant to sea level forecasts.
    Section 2.2 retains 15/15/10 modes; if leading EOFs discard correlated small-scale variance, the model's joint output may be handicapped from the start.
  • domain assumption Block bootstrap with block length 3 weeks accurately accounts for temporal autocorrelation in the 184 test initializations.
    Section 2.2 states this choice; the results of significance tests depend on it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do AI Forecast Ensembles Sample the Correct Conditional Distribution?." pith.science (2026). https://pith.science/paper/V5ONQG5M

@misc{pith2026260808954,
  author       = {Pith},
  title        = {Pith review of: Do AI Forecast Ensembles Sample the Correct Conditional Distribution?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V5ONQG5M}},
  note         = {Machine review of arXiv:2608.08954}
}
read the original abstract

Ensemble forecasting aims to sample the conditional distribution of outcomes; whether AI forecast ensembles do this correctly in a joint sense remains largely untested. We train a diffusion model for probabilistic subseasonal coastal sea level forecasts at eight US East Coast tide gauge stations, with sea level derived from reanalysis, and find that marginal and joint forecast quality decouple: positive skill at every station and lead time marginally, while joint spatial structure is worse than climatological draws. A shuffle-based permutation decomposition reveals this failure is invisible to the energy score but detected by the variogram score. Lorenz-96 experiments across 0.7-170 equivalent years show the gap persists regardless of training volume and is reproduced by a linear baseline, indicating structural inadequacy of the learned distribution. A dynamical ensemble does not replicate the failure while a deterministic emulator does, suggesting it is specific to learned emulators rather than ensemble forecasting generally.

Figures

Figures reproduced from arXiv: 2608.08954 by the authors.

Figure 1
Figure 1. DDPM SSH forecast skill evaluated using three probabilistic metrics with 95% confidence intervals indicated with shading and error bars. (a) CRPS skill score; positive values indicate improvement over a climatological forecast. (b) Energy Score (ES) skill score of the DDPM ensemble (filled circles), the spatially shuffled DDPM ensemble (open squares), and a shuffled GLORYS climatological draw (triangles) (c) As in (… view at source ↗
Figure 2
Figure 2. Pairwise SSH correlation structure and per-pair VS skill deficit across the eight tide gauge stations. (a) Arc diagram of pairwise Pearson r between stations ordered north (left) to south (right); arc color and thickness indicate correlation strength (GLORYS 1993-2018 training period). Inset shows station locations along the US East Coast. (b) The multivariate skill deficit vs. the observed correlation between stati… view at source ↗
Figure 3
Figure 3. CRPS and Variogram Score (VS) skill as a function of training data volume for a DDPM trained on the Lorenz-96 system at three forecast lead times. (a, c, e) CRPS skill (blue) and VS skill (orange) scores with 95% confidence intervals as a function of training period length (x-axis, in equivalent years) for lead times of 5, 10, and 15 days, respectively. The ERA5 equiv￾alent record length is indicated by a solid gree… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: CRPS (blue) and VS (orange) skill scores for each of the ensemble modeling ap￾proaches (a). The DDPM is natively probabilistic, while the dynamical and deterministic emu￾lators use two sets of perturbed initial conditions to generate ensemble forecasts. The first uses …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 6 canonical work pages

  1. [1]

    Predicting

    Bachèlery, Marie-Lou and Brajard, Julien and Patacchiola, Massimiliano and Illig, Serena and Keenlyside, Noel , month = apr, year =. Predicting. Science Advances , publisher =. doi:10.1126/sciadv.ads5185 , abstract =

    work page doi:10.1126/sciadv.ads5185

  2. [2]

    Strictly

    Gneiting, Tilmann and Raftery, Adrian E , month = mar, year =. Strictly. Journal of the American Statistical Association , publisher =. doi:10.1198/016214506000001437 , abstract =

    doi:10.1198/016214506000001437

  3. [3]

    Journal of Advances in Modeling Earth Systems , author =

    A. Journal of Advances in Modeling Earth Systems , author =. 2024 , note =. doi:10.1029/2023MS003774 , abstract =

    doi:10.1029/2023ms003774 2024

  4. [4]

    , month = apr, year =

    Scheuerer, Michael and Hamill, Thomas M. , month = apr, year =. Variogram-. Monthly Weather Review , publisher =. doi:10.1175/MWR-D-14-00269.1 , abstract =

    doi:10.1175/mwr-d-14-00269.1

  5. [5]

    Quarterly Journal of the Royal Meteorological Society , author =

    Assessing the calibration of multivariate probabilistic forecasts , volume =. Quarterly Journal of the Royal Meteorological Society , author =. 2024 , note =. doi:10.1002/qj.4647 , abstract =

    work page doi:10.1002/qj.4647 2024

  6. [6]

    Journal of Computational Science , author =

    Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations:. Journal of Computational Science , author =. 2020 , keywords =. doi:10.1016/j.jocs.2020.101171 , abstract =

    arXiv 2020

  7. [7]

    Arnold, H. M. and Moroz, I. M. and Palmer, T. N. , month = may, year =. Stochastic parametrizations and model uncertainty in the. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , publisher =. doi:10.1098/rsta.2011.0479 , abstract =

    arXiv

  8. [8]

    Journal of Advances in Modeling Earth Systems , author =

    Machine. Journal of Advances in Modeling Earth Systems , author =. 2020 , note =. doi:10.1029/2019MS001896 , abstract =

    doi:10.1029/2019ms001896 2020

Show all 28 references
  1. [9]

    Geophysical Research Letters , author =

    Assimilating. Geophysical Research Letters , author =. 2025 , note =. doi:10.1029/2024GL114396 , abstract =

    2025 doi

  2. [10]

    Lellouche, Jean-Michel and Greiner, Eric and Bourdallé-Badie, Romain and Garric, Gilles and Melet, Angélique and Drévillon, Marie and Bricaud, Clément and Hamon, Mathieu and Le Galloudec, Olivier and Regnier, Charly and Candela, Tony and Testut, Charles-Emmanuel and Gasparin, ...

  3. [11]

    Quarterly Journal of the Royal Meteorological Society , author =

    The. Quarterly Journal of the Royal Meteorological Society , author =. 2020 , note =. doi:10.1002/qj.3803 , abstract =

    2020 doi

  4. [12]

    Denoising

    Ho, Jonathan and Jain, Ajay and Abbeel, Pieter , month = dec, year =. Denoising. doi:10.48550/arXiv.2006.11239 , abstract =

    2006 doi arXiv

  5. [13]

    , month = may, year =

    Lorenz, Edward N. , month = may, year =. Designing. Journal of the Atmospheric Sciences , publisher =. doi:10.1175/JAS3430.1 , abstract =

    doi

  6. [14]

    Geophysical Research Letters , author =

    Evaluating. Geophysical Research Letters , author =. 2026 , note =. doi:10.1029/2025GL119402 , abstract =

    2026 doi

  7. [15]

    arXiv preprint arXiv:2211.02556 , author =

    Pangu‑. arXiv preprint arXiv:2211.02556 , author =

    arXiv

  8. [16]

    arXiv preprint arXiv:2212.12794 , author =

    arXiv

  9. [17]

    arXiv preprint arXiv:2312.15796 , author =

    arXiv

  10. [18]

    Andre and Kwa, Anna and McGibbon, Jeremy and Arcomano, Troy and Clark, Spencer K

    Perkins, W. Andre and Kwa, Anna and McGibbon, Jeremy and Arcomano, Troy and Clark, Spencer K. and Watt-Meyer, Oliver and Bretherton, Christopher S. and Harris, Lucas M. , month = feb, year =. doi:10.48550/arXiv.2512.18224 , abstract =

    doi

  11. [19]

    The quiet revolution of numerical weather prediction , volume =

    Bauer, Peter and Thorpe, Alan and Brunet, Gilbert , month = sep, year =. The quiet revolution of numerical weather prediction , volume =. Nature , publisher =. doi:10.1038/nature14956 , abstract =

    doi

  12. [20]

    Quarterly Journal of the Royal Meteorological Society , author =

    The. Quarterly Journal of the Royal Meteorological Society , author =. 2019 , note =. doi:10.1002/qj.3383 , abstract =

    2019 doi

  13. [21]

    and Becker, Emily and Kirtman, Ben P

    Arcodia, Marybeth C. and Becker, Emily and Kirtman, Ben P. , month = feb, year =. Subseasonal. Weather and Forecasting , publisher =. doi:10.1175/WAF-D-23-0002.1 , abstract =

    doi

  14. [22]

    and Han, Weiqing and Rajagopalan, Balaji and Yanto and Jasinski, Mike , month = jul, year =

    Kenigson, Jessica S. and Han, Weiqing and Rajagopalan, Balaji and Yanto and Jasinski, Mike , month = jul, year =. Decadal. Journal of Climate , publisher =. doi:10.1175/JCLI-D-17-0403.1 , abstract =

    doi

  15. [23]

    Thompson, David W. J. and Wallace, John M. , month = mar, year =. Annular. Journal of Climate , publisher =. doi:10.1175/1520-0442(2000)013<1000:AMITEC>2.0.CO;2 , abstract =

    2000 doi

  16. [24]

    Journal of Computational Physics , author =

    Ensemble forecasting , volume =. Journal of Computational Physics , author =. 2008 , keywords =. doi:10.1016/j.jcp.2007.02.014 , abstract =

    2008 doi

  17. [25]

    Annual Review of Statistics and Its Application , author =

    Probabilistic. Annual Review of Statistics and Its Application , author =. 2014 , note =. doi:10.1146/annurev-statistics-062713-085831 , abstract =

    2014 doi

  18. [27]

    Journal of Advances in Modeling Earth Systems , author =

    Sub-. Journal of Advances in Modeling Earth Systems , author =. 2021 , keywords =. doi:10.1029/2021MS002502 , abstract =

    2021 doi

  19. [28]

    arXiv preprint arXiv:2406.01465 , author =

    arXiv

  20. [29]

    Howard, Lucas , year =. Do. doi:10.5281/zenodo.21864536 , url =

    doi

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.