Pith. sign in

REVIEW 4 major objections 4 minor 15 references

EPT-2 Technical Report

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EPT-2 claims a ten-member AI ensemble beats the gold-standard 50-member ECMWF ensemble.

desk verdict EPT-2 makes a strong operational claim on thin evidence: the deterministic curves look plausible, but the probabilistic story is undercut by a wrong CRPS formula and a missing ENS curve. read the letter →

arxiv 2507.09703 v1 pith:BWHVIKJ2 submitted 2025-07-13 cs.LG cs.AI

Roberto Molinaro , Niall Siegenheim , Niels Poulsen , Jordan Dane Daubinet , Henry Martin , Mark Frey , Kevin Thiart , Alexander Jakob Dautel
show 7 more authors
Andreas Schlueter Alex Grigoryev Bogdan Danciu Nikoo Ekhtiari Bas Steunebrink Leonie Wagner Marvin Vincent Gabler
This is my paper
classification cs.LGcs.AI
keywords AIweatherforecastingensembleenergymeteorologytransformermodelprobabilisticwindspeedsolarradiationstationbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that EPT-2, a transformer-based AI weather model, outperforms both ECMWF's operational high-resolution forecast (IFS HRES) and the leading AI model Aurora on root-mean-square error for 10 m and 100 m wind speed, 2 m temperature, and surface solar radiation over the 0 to 240 hour horizon. It also claims that EPT-2e, a ten-member ensemble made by perturbing the initial conditions, achieves lower RMSE and a lower continuous ranked probability score (CRPS) than the 50-member ECMWF ENS mean for 2 m temperature and 10 m wind speed against global station observations. The motivation is energy forecasting: these surface variables drive power markets, and the paper argues AI provides them at hourly resolution and a fraction of the compute of operational numerical ensembles. If the claims hold, a small AI ensemble could replace the traditional 50-member ensemble as the operational reference for medium-range probabilistic forecasts.

What carries the argument

The engine is the learned mapping $f_\theta$ that approximates $X_{t+\Delta t} \approx f_\theta(X_t, X_{t-1}, \ldots)$, trained by minimizing a latitude-weighted mean absolute error over all variables and grid cells. The latitude weights $w_i = \cos(\text{lat}_i) / \frac{1}{H}\sum_j \cos(\text{lat}_j)$ compensate for the denser grid spacing near the poles. Dynamic lead-time conditioning allows the same weights to produce forecasts at any $\Delta t$, yielding native hourly outputs, and EPT-2e builds its ensemble by perturbing the initial conditions so that CRPS measures whether the resulting spread matches the observed skill.

What would settle it

Re-run the station benchmark with the authors' stationbench code on a year not used in training (e.g., 2024), initializing EPT-2 and Aurora from the same IFS HRES initial conditions; if EPT-2's RMSE and CRPS advantages over Aurora and the ECMWF ENS mean disappear, the state-of-the-art claim fails. A second check is to audit the EPT-2 training set for overlap with 2023 or with the station observations used as ground truth.

Watch

Extended reading notes

Core claim

The central discovery claim is that a single dynamically conditioned transformer, trained with a latitude-weighted loss on gridded atmospheric data, can beat the most accurate operational deterministic numerical model (IFS HRES) and the leading AI baseline (Aurora) on energy-relevant surface variables across the entire 0–240 h range. The companion claim is that an ensemble of ten perturbed initial states propagated through the same model yields better RMSE and CRPS than the 50-member ECMWF ENS mean when evaluated against in-situ station observations. The report supports these claims with 2023 benchmarks computed from forecasts initialized at 00 and 12 UTC, with the WeatherReal-ISD station dataset treated as neutral ground truth.

Load-bearing premise

The evaluation assumes that the 2023 station observations are a neutral ground truth and that every model was initialized from identical input data, with no model postprocessed or fine-tuned on station data; if any of those conditions slips, the reported gaps over Aurora and the ECMWF ENS mean could shrink or reverse.

Editorial extensions

If this is right

  • If the station-based scores are right, a 10-member AI ensemble matches or beats the operational 50-member ENS, implying ensemble size is not the main constraint on probabilistic skill.
  • Energy-market users would gain hourly 0–240 h forecasts of wind, temperature, and solar radiation at roughly 9 km resolution, improving forecasts of wind ramps and temperature peaks that six-hourly models miss.
  • The reported training cost (8 H100 GPUs for 10 days) and roughly 25% faster inference than Aurora make high-frequency operational deployment economically practical.
  • Because EPT-2 uses dynamic lead-time conditioning, a single model covers every forecast horizon, removing the need for separate six-hourly output heads.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the neutrality premise would be to run the same station benchmark on a year outside the reported 2023 evaluation and check that EPT-2's margins over Aurora and the ENS mean persist.
  • If the margins hold, a likely driver is the choice of high-resolution IFS HRES initial conditions for training and initialization; a controlled study initializing Aurora from the same inputs would isolate that effect.
  • The paper mentions large-scale training experiments with up to 10 billion parameters but does not report their skill; publishing that scaling curve would show whether the small 8-GPU model sits near a performance plateau.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript introduces EPT-2, a deterministic weather forecasting model, and EPT-2e, a ten-member perturbation ensemble. It claims that EPT-2 outperforms Aurora and IFS HRES on deterministic RMSE for 2m temperature, 10m/100m wind speed, and solar radiation over the 0–240 h forecast range, and that EPT-2e outperforms the ECMWF ENS mean on station-based RMSE and CRPS for 2m temperature and 10m wind speed, all while being computationally cheaper. Evaluations are reported against both gridded initial conditions and WeatherReal station observations over the single year 2023.

Significance. If the deterministic claims hold, they would add a useful data point to the growing evidence that learned weather models can rival operational NWP on surface variables. The ensemble claim would be more significant, since a ten-member AI ensemble surpassing the 50-member ECMWF ENS would be a notable practical result. The paper also ships a public GitHub benchmark for station-based point evaluation, which is a genuine reproducibility asset. However, the current evidence is not sufficient to establish the headline probabilistic claim: the reported CRPS formula is not a proper scoring rule as written, the CRPS figures omit the ECMWF ENS comparison that the abstract promises, and all quantitative conclusions rest on one year of curves without error bars or significance tests. The deterministic results are plausible but equally lack the numerical reporting needed to assess effect sizes.

major comments (4)
  1. [Section III-C, Eq. (2)] The CRPS definition in Eq. (2) uses squared differences in both the skill and spread terms, whereas the standard energy-form CRPS for an ensemble uses absolute differences. As written, Eq. (2) is not a proper scoring rule: for fixed ensemble mean it approximately reduces to the mean squared error of the members minus the ensemble variance, which rewards overconfident ensembles and discards calibration information. If Figures 10 and 11 were computed with this formula, the claimed probabilistic superiority of EPT-2e is not supported, and the abstract's CRPS claim is unsubstantiated. Please correct the formula to the absolute-difference energy form, or state explicitly what metric was actually implemented and show that it is a proper scoring rule.
  2. [Section IV-B, Figures 10 and 11] The abstract and introduction state that EPT-2e achieves lower CRPS than the ECMWF ENS mean across all key variables and lead times, but Figures 10 and 11 show only Aurora and EPT-2e; no ECMWF ENS CRPS curves are presented. The RMSE figures (8 and 9) do include ENS, but the probabilistic claim in the abstract and the 'probabilistic standpoint' statement in Section IV-B are about CRPS. Without ENS CRPS curves, the gold-standard comparison on probabilistic skill is unsupported. Please add the missing ENS CRPS curves or revise the claim.
  3. [Section III-C and Section IV] All results are evaluated over a single year (2023) and are presented only as curves without numeric values, error bars, or significance tests. The repeated use of 'significantly surpasses' and 'consistently outperforms' is not backed by any statistical analysis. Please provide tabulated scores with sample sizes, confidence intervals, and, where appropriate, paired significance tests (e.g., over the 2023 forecast dates), so that readers can assess whether the reported gaps are meaningful rather than noise.
  4. [Section III-A and Section III-C] Section III-A states that 'none of the models discussed in this work have been postprocessed or finetuned using weather station data,' but this does not rule out training on station observations or on 2023 data. Since the evaluation target is station observations in 2023, a statement about the training data cutoff and whether station data or any 2023 data were used in pretraining is required to preclude train/test leakage. Similarly, Section III-C says EPT-2 is initialized with 'the same input data as the other models'; please specify whether this means identical IC ERA5/HRES fields at identical resolution and interpolation, since differing initializations could explain part of the skill gap.
minor comments (4)
  1. [Section III-B] The text refers to 'HRES's 50' members in Section IV-B, but HRES is a deterministic model; the ECMWF ensemble (ENS) has 50 members. This should be corrected.
  2. [Section IV-B, Figure 15] The claim that Aurora 'misses' the wind-speed event around 36 h may be an artifact of Aurora's 6-hourly output cadence rather than a genuine forecast failure; this should be phrased more carefully.
  3. [Abstract and Section I] The phrases 'a AI-based model' and 'a AI model' are grammatically incorrect and should be 'an AI-based model' and 'an AI model.'
  4. [Figures 6–11] The vertical axis ranges in the station-based RMSE figures (Figures 6–9) are compressed, making differences between curves difficult to discern. Numeric tables of RMSE and CRPS values at selected lead times (e.g., 24, 72, 120, 240 h) would greatly improve readability and would allow readers to reproduce the plots.

Circularity Check

1 steps flagged · score 6.0 of 10

Probabilistic CRPS claim reduces by construction to ensemble-mean RMSE; the deterministic benchmark remains external.

  1. renaming known result [Section III-C, Eq. (2); interpreted in Section IV-B, Figures 10-11]
    "For ensemble forecasts, the CRPS can be decomposed into two terms: CRPS = 1/N Σ_{i=1}^N w_i (X̃_i − X_i)^2 − 1/(2N^2) Σ_{i=1}^N Σ_{j=1}^N w_i (X̃_i − X̃_j)^2, (2) ... The first term measures accuracy (closeness to the observation), while the second captures ensemble spread (internal variability)."

    For the station benchmarks used in Figures 10-11, w_i = 1. Algebra then gives (1/N)Σ(X̃_i−X)^2 − (1/(2N^2))ΣΣ(X̃_i−X̃_j)^2 = (m−X)^2, where m is the ensemble mean: the spread term exactly cancels the within-ensemble variance in the first term. Thus the quantity labeled CRPS is, by the paper's own equation, just the squared error of the ensemble mean. The 'probabilistic standpoint' in Figures 10-11 therefore contains no information beyond the ensemble-mean RMSE already reported in Figures 8-9; the CRPS result is a renaming of the deterministic ensemble-mean skill, not an independent probabilistic evaluation. The headline claim of 'lower RMSE and CRPS' consequently presents the same ensemble-mean error twice under two metric names.

full rationale

The deterministic SOTA claim against Aurora, IFS HRES, and ECMWF ENS is an external benchmark based on in-situ station observations and initial-condition comparisons; it does not reduce to EPT-2's training objective or to a self-citation, so that part is not circular. The self-citations to EPT-1.5 and Vilhelm are contextual, not load-bearing for the central performance claim. The one genuine reduction is in the probabilistic evaluation: Eq. (2) defines CRPS with squared deviations, and for the station-based setting this expression simplifies exactly to the squared error of the ensemble mean. Consequently, the reported CRPS superiority is not a separate probabilistic result; it is the ensemble-mean RMSE result re-labeled. This is a partial circularity: the paper's 'probabilistic' headline claim is forced by its own definition to coincide with its deterministic headline claim, while the deterministic external benchmark retains independent content.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on benchmark fairness rather than on a derivation. The report discloses no model weights or exact evaluation numbers, and the ensemble perturbation and training split are unstated. The main hand-chosen input is the perturbation recipe; the main domain assumptions concern ground truth and out-of-sample status.

free parameters (1)
  • Ensemble perturbation recipe
    EPT-2e and Aurora ensembles are produced by perturbing initial conditions, but the amplitude, spatial correlation, and perturbation construction are not specified; CRPS and spread skill depend on this choice.
assumptions (5)
  • domain assumption ERA5 and IFS HRES initial conditions are valid ground truth for grid benchmark evaluation.
    Section III-A treats ICs as widely used reference; this is standard but not a direct observation.
  • domain assumption WeatherReal station observations are a neutral, more faithful ground truth, and no model was postprocessed or finetuned on them.
    Section III-A asserts this for all models without showing an audit trail.
  • domain assumption All compared models can be fairly initialized from the same input data at 00/12 UTC during 2023.
    Section III-C gives no detail on input variables, levels, or preprocessing; differences would bias the comparison.
  • standard math Equation (2) is the correct CRPS decomposition for ensemble forecasts.
    The report uses squared differences in Eq. (2), unlike the standard energy score form with absolute differences, so the probabilistic evaluation relies on a non-standard metric.
  • domain assumption The 2023 evaluation period is out of sample for EPT-2 and all baselines.
    Not stated anywhere; if any model was trained on 2023 data, the comparison is contaminated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EPT-2 Technical Report." pith.science (2026). https://pith.science/paper/BWHVIKJ2

@misc{pith2026250709703,
  author       = {Pith},
  title        = {Pith review of: EPT-2 Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BWHVIKJ2}},
  note         = {Machine review of arXiv:2507.09703}
}
read the original abstract

We present EPT-2, the latest iteration in our Earth Physics Transformer (EPT) family of foundation AI models for Earth system forecasting. EPT-2 delivers substantial improvements over its predecessor, EPT-1.5, and sets a new state of the art in predicting energy-relevant variables-including 10m and 100m wind speed, 2m temperature, and surface solar radiation-across the full 0-240h forecast horizon. It consistently outperforms leading AI weather models such as Microsoft Aurora, as well as the operational numerical forecast system IFS HRES from the European Centre for Medium-Range Weather Forecasts (ECMWF). In parallel, we introduce a perturbation-based ensemble model of EPT-2 for probabilistic forecasting, called EPT-2e. Remarkably, EPT-2e significantly surpasses the ECMWF ENS mean-long considered the gold standard for medium- to longrange forecasting-while operating at a fraction of the computational cost. EPT models, as well as third-party forecasts, are accessible via the app.jua.ai platform.

Figures

Figures reproduced from arXiv: 2507.09703 by the authors.

Figure 1
Figure 1. Spatial distribution of the weather stations for benchmarking [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. RMSE of 2m air temperature forecasts against ICs over the 0 – [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. RMSE of 10m wind speed forecasts against ICs over the 0 – 240 [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: RMSE of 100m wind speed forecasts against ICs over the 0 – 240 [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: RMSE of solar radiation forecasts against ICs over the 0 – 240 h [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: RMSE of 2m air temperature forecasts against in-situ weather [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 8
Figure 8. Figure 8: RMSE of 2m air temperature (ensemble mean) forecasts against [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 10
Figure 10. Figure 10: CRPS with weather station as reference for 2 m air temperature [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 12
Figure 12. Figure 12: 2m air temperature predictions from EPT-2, EPT-1.5, and Aurora on the 2023-10-01. The top row shows the initial condition (IC), while the [PITH_FULL_IMAGE:figures/full_fig_p008_12.png]
Figure 13
Figure 13. Figure 13: Time series of 2m temperature forecasts at a Zurich location on the 2023-10-01. EPT models provide hourly resolution, while Aurora outputs [PITH_FULL_IMAGE:figures/full_fig_p008_13.png]
Figure 14
Figure 14. Figure 14: 10m wind speed predictions from EPT-2, EPT-1.5, and Aurora on the 2023-10-01. The top row shows the initial condition (IC), while the second, [PITH_FULL_IMAGE:figures/full_fig_p009_14.png]
Figure 15
Figure 15. Figure 15: Time series of 10m wind speed forecasts at a Zurich location on the 2023-10-01. EPT models provide hourly resolution, while Aurora outputs [PITH_FULL_IMAGE:figures/full_fig_p009_15.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 10 canonical work pages

  1. [1]

    A foundation model for the earth system,

    C. Bodnar, W. P. Bruinsma, A. Lucic, M. Stanley, A. Vaughan, J. Brandstetter, P. Garvan, M. Riechert, J. A. Weyn, H. Dong, J. K. Gupta, K. Thambiratnam, A. T. Archibald, C.-C. Wu, E. Heider, M. Welling, R. E. Turner, and P. Perdikaris, “A foundation model for the earth system,” 2024. [Online]. Available: https://arxiv.org/abs/2405.13063

    arXiv 2024

  2. [2]

    Learning skillful medium-range global weather forecasting,

    R. Lam, A. Sanchez-Gonzalez, M. Willson, P. Wirnsberger, M. For- tunato, F. Alet, S. Ravuri, T. Ewalds, Z. Eaton-Rosen, W. Hu et al. , “Learning skillful medium-range global weather forecasting,” Science, vol. 382, no. 6677, pp. 1416–1421, 2023

    work page 2023

  3. [3]

    Aifs – ecmwf’s data-driven forecasting system,

    S. Lang, M. Alexe, M. Chantry, J. Dramsch, F. Pinault, B. Raoult, M. C. A. Clare, C. Lessig, M. Maier-Gerber, L. Magnusson, Z. B. Bouallègue, A. P. Nemesio, P. D. Dueben, A. Brown, F. Pappenberger, and F. Rabier, “Aifs – ecmwf’s data-driven forecasting system,” 2024. [Online]. Available: https://arxiv.org/abs/2406.01465

    arXiv 2024

  4. [4]

    Accurate medium-range global weather forecasting with 3d neural networks,

    K. Bi, L. Xie, H. Zhang, X. Chen, X. Gu, and Q. Tian, “Accurate medium-range global weather forecasting with 3d neural networks,” Nature, vol. 619, no. 7970, pp. 533–538, 2023. 6 CRPS (K) Lead Time (hours) 2.1 2.2 2.0 1.9 1.8 1.6 1.7 1.5 1.4 6h 54h 102h 150h 198h 240h Aurora EPT-2e Air Temperature 2m - CRPS Figure 10. CRPS with weather station as referenc...

    work page 2023

  5. [5]

    WeatherMesh-3: Fast and accurate operational global weather forecasting

    H. Du, L. Kim, J. Creus-Costa, J. Michaels, A. Shetty, T. Hutchinson, C. Riedel, and J. Dean, “Weathermesh-3: Fast and accurate operational global weather forecasting,” 2025. [Online]. Available: https://arxiv.org/abs/2503.22235

    work page Pith review arXiv 2025

  6. [6]

    Aardvark weather: end-to-end data-driven weather forecasting,

    A. Vaughan, S. Markou, W. Tebbutt, J. Requeima, W. P. Bruinsma, T. R. Andersson, M. Herzog, N. D. Lane, M. Chantry, J. S. Hosking, and R. E. Turner, “Aardvark weather: end-to-end data-driven weather forecasting,” 2024. [Online]. Available: https://arxiv.org/abs/2404.00411

    arXiv 2024

  7. [7]

    Vilhelm: A Novel AI-based Global Weather Forecasting System for the Prediction of Atmospheric Hazards,

    M. V . Gabler, J. Wuilloud, H. Taheri Shahraiyni, D. Neupert, A. Grig- oryev, R. Almeida, A. Galimzhanov, G. M. Hernandez, J. D. Daubinet, N. Ekhtiari, R. J. Song, P. Dudbridge, and E. Tarakci, “Vilhelm: A Novel AI-based Global Weather Forecasting System for the Prediction of Atmospheric Hazards,” in EMS Annual Meeting 2023, Bratislava, Slovakia, 2023, eM...

    work page 2023

  8. [8]

    A Novel High- Resolution AI-based Global Precipitation Forecasting System,

    M. V . Gabler, J. Wuilloud, H. Taheri Shahraiyni, D. Neupert, A. Grig- oryev, R. Almeida, A. Galimzhanov, G. M. Hernandez, J. D. Daubinet, N. Ekhtiari, R. J. Song, P. Dudbridge, and E. Tarakci, “A Novel High- Resolution AI-based Global Precipitation Forecasting System,” in EMS Annual Meeting 2023, Bratislava, Slovakia , 2023, eMS2023-355

    work page 2023

Show all 15 references
  1. [9]

    Ept-1.5 technical report,

    R. Molinaro, J. D. Daubinet, A. J. Dautel, A. Schlueter, A. Grigoryev, N. Ekhtiari, B. Steunebrink, K. Thiart, R. J. Song, H. Martin, L. Wagner, A. Giussani, and M. V . Gabler, “Ept-1.5 technical report,”

  2. [10]

    Fuxi weather: A data-to-forecast machine learning system for global weather,

    X. Sun, X. Zhong, X. Xu, Y . Huang, H. Li, J. D. Neelin, D. Chen, J. Feng, W. Han, L. Wu, and Y . Qi, “Fuxi weather: A data-to-forecast machine learning system for global weather,” 2024. [Online]. Available: https://arxiv.org/abs/2408.05472

    2024

  3. [11]

    Weather- bench 2: A benchmark for the next generation of data-driven global weather models,

    S. Rasp, S. Hoyer, A. Merose, I. Langmore, P. Battaglia, T. Russell, A. Sanchez-Gonzalez, V . Yang, R. Carver, S. Agrawal et al., “Weather- bench 2: A benchmark for the next generation of data-driven global weather models,” Journal of Advances in Modeling Earth Systems , vol. ...

    2024

  4. [12]

    The era5 global reanalysis,

    H. Hersbach, B. Bell, P. Berrisford, S. Hirahara, A. Horányi, J. Muñoz- Sabater, J. Nicolas, C. Peubey, R. Radu, D. Schepers et al. , “The era5 global reanalysis,” Quarterly Journal of the Royal Meteorological Society, vol. 146, no. 730, pp. 1999–2049, 2020

    1999

  5. [13]

    Weatherreal: A benchmark based on in-situ observations for evaluating weather models,

    W. Jin, J. Weyn, P. Zhao, S. Xiang, J. Bian, Z. Fang, H. Dong, H. Sun, K. Thambiratnam, and Q. Zhang, “Weatherreal: A benchmark based on in-situ observations for evaluating weather models,” 2024. [Online]. Available: https://arxiv.org/abs/2409.09371

    2024 arXiv

  6. [14]

    Atmospheric model high resolution 10-day forecast (set i - hres),

    European Centre for Medium-Range Weather Forecasts (ECMWF), “Atmospheric model high resolution 10-day forecast (set i - hres),” 2024. [Online]. Available: https://www.ecmwf.int/en/forecasts/datasets/set-i 7 VII. S UPPLEMENTARY MATERIAL Figure 12. 2m air temperature predictions...

    2024

  7. [2024]

    Available: https://arxiv.org/abs/2410.15076

    [Online]. Available: https://arxiv.org/abs/2410.15076

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.