Pith. sign in

REVIEW 3 major objections 5 minor 16 references

Self-attentive Transformer for Fast and Accurate Postprocessing of Temperature and Wind Speed Forecasts

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read A self-attentive Transformer corrects temperature and wind-speed ensemble forecasts for twenty lead times simultaneously, cutting the continuous ranked probability score by up to 16.5% while training roughly six times faster than a…

desk verdict Useful gridded wind benchmark entry with joint-lead-time Transformer; solid relative comparison, but absolute wind skill rests on ERA5 as ground truth. read the letter →

arxiv 2412.13957 v2 pith:33HMHM2P submitted 2024-12-18 cs.LG physics.ao-ph

classification cs.LGphysics.ao-ph
keywords ensembleweatherforecastpostprocessingTransformermulti-headself-attentioncontinuousrankedprobabilityscoretwo-metertemperaturewindspeedEUPPBenchmember-by-member
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to show that one self-attentive Transformer can postprocess gridded ensemble weather forecasts for three variables — two-meter temperature, ten-meter wind speed, and one-hundred-meter wind speed — over twenty lead times at once, rather than fitting a separate model per lead time. Measured by the continuous ranked probability score (CRPS), the authors report that the Transformer improves on the raw ECMWF ensemble forecasts by 16.5% for temperature and by 10% and 9% for the two wind speeds, and that it beats a classical member-by-member correction over most of the domain while training roughly five to six times faster. If these results hold, operational forecasters could replace a stack of per-lead-time statistical corrections with a single model that also exchanges information across variables, grid points, and lead times, with direct benefits for applications such as renewable-energy forecasting. The paper also provides the first gridded wind-speed baselines in the EUPPBench dataset.

What carries the argument

The central object is the self-attentive ensemble Transformer, an adaptation of Finn's ensemble Transformer. The input is a tensor $Z \in \mathbb{R}^{b\times k\times t\times h\times w\times c}$ of $k$ ensemble members, $t$ lead times, a spatial grid of size $h\times w$, and $c$ predictors; a linear layer projects the $c$ predictors to $\tilde{c}$ features, then $n=4$ transformer blocks apply multi-headed self-attention with 8 heads, computing the softmax across the ensemble-member axis while the query, key, and value weights are shared across space, time, and features. This sharing is what lets information propagate implicitly across grid points and lead times, analogous to an ensemble Kalman filter, so one model can postprocess all 20 lead times at once. The output is projected back to a single variable, and training minimizes the CRPS, analytically for temperature and via a regularized kernel CRPS for wind speed.

What would settle it

Compare the Transformer's wind-speed corrections against independent offshore and coastal observations such as buoys, masts, or lidar instead of ERA5; if the CRPS advantage over raw forecasts shrinks or reverses at coastal grid points where ERA5 is known to underestimate strong winds, part of the claimed wind-speed gain is an artifact of the reference field.

Watch

Extended reading notes

Core claim

The central claim is that a self-attentive ensemble Transformer, using multi-headed self-attention across ensemble members with weights shared over spatial, temporal, and predictor dimensions, can correct all twenty lead times of gridded ECMWF ensemble forecasts simultaneously and more accurately than a classical member-by-member method. The authors report CRPS reductions over raw forecasts of 16.5% for two-meter temperature, 10% for ten-meter wind speed, and 9% for one-hundred-meter wind speed, with the Transformer ahead of classical MBM at every lead time and over nearly the whole EUPPBench domain. They further report that training takes about one-sixth of the time for temperature and one-fifth for wind speed, and that the Transformer attains spread-error ratios close to one, indicating markedly better calibrated ensembles than the raw forecasts. The paper presents this as the first postprocessing of gridded ten- and one-hundred-meter wind speed in EUPPBench.

Load-bearing premise

The evaluation treats the ERA5 reanalysis, a computer-generated weather reconstruction, as the true observation for all three variables; because ERA5 is known to underestimate strong winds near coasts and offshore, part of the measured wind-speed improvement could be correction toward a biased reference rather than genuine forecast skill.

Editorial extensions

If this is right

  • A single Transformer can replace per-lead-time postprocessing models for the three target variables, cutting training time by roughly a factor of five to six while improving CRPS at every lead time.
  • The method supplies the first gridded wind-speed baselines in EUPPBench, giving future postprocessing studies a direct comparison point for 10 m and 100 m wind speed.
  • Adding predictors is cheap: increasing from three to ten predictors raises training time by about 7%, so operational systems can include richer input sets without retraining per lead time.
  • The Transformer's spread-error ratios near one and more uniform rank histograms imply a substantially better calibrated ensemble than raw forecasts, which matters for probabilistic decision-making.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested next step is whether the same shared-attention design transfers to non-smooth variables such as precipitation or gusts; the paper's evidence covers only temperature and wind speed.
  • The attention maps could be used diagnostically to locate where operational corrections are most needed, a use the paper sketches but does not develop into a method.
  • Because ERA5 is also the training target, the reported wind-speed gains should be re-evaluated against in-situ coastal observations before being used to justify offshore renewable-energy applications.
  • The speed comparison is hardware-dependent (GPU versus CPU), so an end-to-end operational test at national-forecasting scale is needed to confirm the wall-clock advantage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a self-attentive Transformer for gridded postprocessing of ensemble weather forecasts, applied to two-meter temperature and ten- and one-hundred-meter wind speed over the EUPPBench domain. Unlike many previous postprocessing models, the Transformer processes 20 lead times simultaneously and exchanges information across ensemble members, variables, spatial locations, and lead times via multi-headed self-attention. The model is trained on EUPPBench forecasts with ERA5 as the reference and is compared against a classical member-by-member (MBM) baseline. The authors report CRPS improvements over the raw ensemble of 16.5% for temperature, 10% for ten-meter wind speed, and 9% for one-hundred-meter wind speed, with the Transformer outperforming classical MBM on average CRPS and spread-error ratio (SER) while requiring substantially less training time.

Significance. If the reported results hold, the paper makes a useful practical contribution: a single postprocessing model covering multiple variables and all lead times, with better average CRPS than raw forecasts and a classical MBM baseline, and with substantially lower training cost. The use of a held-out test year, the gridded multi-lead-time setting, and the first application to one-hundred-meter wind speed in EUPPBench are concrete advances. The evaluation is generally coherent, and the authors are transparent about several limitations, including ERA5's known weaknesses for coastal and offshore winds. The main risks are the lack of uncertainty quantification for the headline scores, the ambiguous selection procedure for the MBM baseline variant, and the dependence of the wind-speed claims on ERA5 as ground truth in exactly the coastal regimes where ERA5 is least reliable.

major comments (3)
  1. [Section 3, Table 3] All headline results are point estimates over a single test year, with no confidence intervals or significance tests. The Transformer-versus-MBM differences for wind speed are small (0.467 vs. 0.477 m/s for w10 and 0.732 vs. 0.747 m/s for w100), and the verification scores are strongly spatially and temporally correlated, so it is unclear whether these differences are sampling noise. Please add block-bootstrap confidence intervals on CRPS/SER differences or paired significance tests, and restrict the word 'significantly' to statistically supported claims.
  2. [Section 2.4] The sentence 'For each target variable, we report results for the most competitive MBM variant' does not specify how the variant was selected. If the choice between the single-predictor and multi-predictor versions of classical MBM was made using the 2017 test data, the benchmark is optimistically biased and the comparison is not a fair baseline. Please state explicitly that variant selection used the 2016 validation split, or report both variants on the test set.
  3. [Section 4 and Figures A.2-A.3] The wind-speed evaluation is anchored entirely to ERA5 as ground truth, and the paper itself cites evidence that ERA5 underestimates strong winds offshore and struggles with coastal variability (Gandoin and Garza, 2024; Alkhalidi et al., 2025). The Transformer's negative bias in the northwest Netherlands and its increasingly negative bias with lead time appear in the regime where ERA5 is least reliable. Part of the measured CRPS gain for wind speed could therefore be convergence toward a biased reference rather than genuine forecast skill. Please add a robustness check against independent observations (e.g., coastal stations for w10 and tall mast or lidar data for w100, where available), or explicitly restrict the wind-speed claims to skill relative to ERA5 and soften the offshore renewable-energy interpretation.
minor comments (5)
  1. [Section 2.4 and Appendix A] The description of the attention-map computation is inconsistent: Section 2.4 mentions averaging the element-wise product of the key and value matrix, while Appendix A and the surrounding text define the map via Q ⊙ K. Please harmonize the wording.
  2. [Eq. (2)] The scaling factor sc is defined just before the attention equation but does not appear in Eq. (2); either include it in the softmax argument or remove the definition.
  3. [Eq. (4)] In the definition of ϵm = V^m − \bar{V}, the symbol \bar{V} is not defined; presumably it is the ensemble mean of the target variable, while \bar{V}_i denotes the ensemble mean of predictor i. Please clarify the notation.
  4. [Table 1] The units for geopotential height at 500 hPa are given as 'm 2s−2'; this should be m² s⁻² or geopotential metres, and the entry should be made consistent.
  5. [Section 2.4 and Abstract] The 'up to six times faster' claim is based on indicative wall-clock times on different hardware (GPU for the Transformer, CPU for MBM), as acknowledged in Section 2.4; please carry this caveat into the abstract or phrase the claim as 'up to six times faster in our setup.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the claimed CRPS improvements are empirical results computed on a held-out test year; the self-citations concern architecture and interpretability, not the evidence for the result.

full rationale

Walking the derivation chain, the central claims are the CRPS improvements in Table 3 (16.5% for t2m, 10% for w10, 9% for w100) over raw forecasts and classical MBM. These are obtained by training on 1997-2015, validating on 2016, and scoring on 2017 (Section 2.3), so the reported numbers are out-of-sample evaluations rather than quantities defined by construction. The scoring rules (Eq. 6; Eqs. A.2-A.5) are standard and do not contain the model's fitted parameters; the fair ensemble CRPS used for wind verification is independent of the kernel-CRPS training loss, which weakens any concern that training and evaluation share a forced objective. The reported kernel-CRPS constants (lambda=0.0275/0.05, k=2.7/2.0) are hyperparameters of the loss, not fitted outputs renamed as predictions. The architecture is adapted from Finn (2021), with code also from Finn (2021) and Ashkboos et al. (2022), and the attention-interpretation claim cites Finn (2023); these are implementation and interpretability aids and are not what establishes the empirical skill numbers. The ERA5 ground-truth limitation in Section 4 - 'some studies report that it underestimates strong wind events offshore and struggles to capture variability at coastal regions' - is a genuine external-validity caveat for the absolute wind-speed gains, but it is acknowledged, and classical MBM is trained and scored on the same ERA5 target, so the head-to-head comparison is not a self-referential reduction. No equation in the paper defines a prediction in terms of the claimed improvement, and no load-bearing argument rests on a self-citation chain. Therefore no circular step is present.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The empirical claims rest on reference data quality, distributional assumptions, and architecture hyperparameters, rather than on theoretical derivations. No new physical or conceptual entities are introduced.

free parameters (5)
  • Kernel CRPS spread penalty lambda = 0.0275 (w10), 0.05 (w100)
    Tunable constant added to training loss to penalize excessive ensemble spread; chosen by hand and central to wind calibration.
  • Kernel CRPS spread penalty k = 2.7 (w10), 2.0 (w100)
    Tunable constant in the spread penalty term; chosen by hand with no reported sensitivity analysis.
  • Learning rate = 0.001
    Adam optimizer setting inherited from prior work; affects convergence and final skill.
  • Batch size = 2
    Small batch used in training; affects gradient noise and wall-clock time.
  • Transformer depth and width = n=4 blocks, h_n=8 heads, m_n=4 MLP factor
    Architecture hyperparameters inherited from the ensemble Transformer line of work; not optimized per variable.
assumptions (5)
  • domain assumption ERA5 reanalysis is an adequate ground truth for 2m temperature and 10m/100m wind speed in the EUPPBench domain.
    Used as the observation reference for all training and verification; the paper later cites known ERA5 weaknesses for coastal and offshore wind.
  • domain assumption A normal predictive distribution is adequate for 2m temperature.
    Used to derive the analytic CRPS in Appendix B and to train and evaluate temperature; standard but still an assumption.
  • domain assumption The chronological split prevents leakage and the 2017 test year is representative.
    Only a chronological split is described; no checks for spatial auto-correlation or regime shift are reported.
  • domain assumption Shared self-attention across the ensemble dimension transfers useful spatiotemporal information.
    Central architectural premise, argued via low-rank factorization and prior sensitivity tests in Appendix A, not proven for this dataset.
  • ad hoc to paper The regularized kernel-based CRPS is a suitable training target for wind speed.
    A regularized energy score is used instead of a standard proper scoring rule; the propriety of the score after adding the penalty is not discussed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Self-attentive Transformer for Fast and Accurate Postprocessing of Temperature and Wind Speed Forecasts." pith.science (2026). https://pith.science/paper/33HMHM2P

@misc{pith2026241213957,
  author       = {Pith},
  title        = {Pith review of: Self-attentive Transformer for Fast and Accurate Postprocessing of Temperature and Wind Speed Forecasts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/33HMHM2P}},
  note         = {Machine review of arXiv:2412.13957}
}
read the original abstract

Current postprocessing techniques often require separate models for each lead time and disregard possible inter-ensemble relationships by either correcting each member separately or by employing distributional approaches. In this work, we tackle these shortcomings with an innovative, fast and accurate Transformer which postprocesses each ensemble member individually while allowing information exchange across variables, spatial dimensions and lead times by means of multi-headed self-attention. Weather forecasts are postprocessed over 20 lead times simultaneously while including up to fifteen meteorological predictors. We use the EUPPBench dataset for training which contains ensemble predictions from the European Center for Medium-range Weather Forecasts' integrated forecasting system alongside corresponding observations. The work presented here is the first to postprocess the ten and one hundred-meter wind speed forecasts within this benchmark dataset, while also correcting two-meter temperature. Our approach significantly improves the original forecasts, as measured by the CRPS, with 16.5\% for two-meter temperature, 10\% for ten-meter wind speed and 9\% for one hundred-meter wind speed, outperforming a classical member-by-member approach employed as a competitive benchmark. Furthermore, being up to six times faster, it fulfills the demand for rapid operational weather forecasts in various downstream applications, including renewable energy forecasting.

Figures

Figures reproduced from arXiv: 2412.13957 by the authors.

Figure 1
Figure 1. The data is batched as a tensor Z ∈ R b×k×t×h×w×c , where b stands for the batch [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The area, including Belgium, the Netherlands, Luxembourg, Switzerland and [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Top row: Improvement in CRPS when comparing the postprocessed forecasts of temperature of the Transformer with (a) the original forecasts and (b) classical MBM. The bluer the region, the better the performance of the Transformer. A map, corresponding to head 6, showing attentive regions of high influence in the last attention module for temperature is shown in Figure (c). A darker color means that particular region … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Top row: Improvement in CRPS when comparing the postprocessed forecasts of ten-meter wind speed of the Transformer with (a) the original forecasts and (b) classical MBM. The bluer the region, the better the performance of the Transformer. A map, corre￾sponding to head …
Figure 5
Figure 5. Figure 5: Top row: Improvement in CRPS when comparing the postprocessed forecasts of one hundred-meter wind speed of the Transformer with (a) the original forecasts and (b) classical MBM. The bluer the region, the better the performance of the Transformer. A map, corresponding t…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 10 canonical work pages

  1. [1]

    Al-Dabbous, S

    Alkhalidi, M., A. Al-Dabbous, S. Al-Dabbous, and D. Alzaid, 2025: Evaluating the accuracy of the era5 model in predicting wind speeds across coastal and offshore regions. Journal of Marine Science and Engineering , 13 (1), URL https://www.mdpi.com/2077-1312/ 13/1/149. Arroyo, ´A., ´A. Herrero, V. Tricio, and E. Corchado, 2017: Analysis of meteorological c...

  2. [2]

    URL https://confluence.ecmwf.int/display/ CKB/ERA5%3A+data+documentation, known issues with analysed near surface winds and their diurnal cycle, Accessed: 2024-12-05

    ECMWF, 2024: ERA5: Data documentation. URL https://confluence.ecmwf.int/display/ CKB/ERA5%3A+data+documentation, known issues with analysed near surface winds and their diurnal cycle, Accessed: 2024-12-05. Evensen, G., 1994: Sequential data assimilation with a nonlinear quasi-geostrophic model using monte carlo methods to forecast error statistics. Journa...

  3. [4]

    Dueben, P

    Earth System Science Data , 15 (6), 2635–2653. Dueben, P. D., M. G. Schultz, M. Chantry, D. J. Gagne, D. M. Hall, and A. McGovern, 2022: Challenges and benchmark datasets for machine learning in the atmospheric sci- ences: Definition, status, and outlook. Artificial Intelligence for the Earth Systems, 1 (3), e210

  4. [11]

    Messori, 2021: Ensemble methods for neural network-based weather forecasts

    Scher, S., and G. Messori, 2021: Ensemble methods for neural network-based weather forecasts. Journal of Advances in Modeling Earth Systems , 13 (2). Schultz, M. G., C. Betancourt, B. Gong, F. Kleinert, M. Langguth, L. H. Leufen, A. Mozaf- fari, and S. Stadtler, 2021: Can deep learning beat numerical weather prediction? Philo- sophical Transactions of the...

  5. [27]

    B., and Coauthors, 2023: The rise of data-driven weather forecasting

    Bouallegue, Z. B., and Coauthors, 2023: The rise of data-driven weather forecasting. URL https://arxiv.org/abs/2307.10128, 2307.10128. Bremnes, J. B., 2020: Ensemble postprocessing using quantile function regression based on neural networks and bernstein polynomials. Monthly Weather Review, 148 (1), 403–414. Burlando, M., M. Pizzo, M. Repetto, G. Solari, ...

  6. [30]

    Veldkamp, S., K. Whan, S. Dirksen, and M. Schmeits, 2021: Statistical postprocessing of wind speed forecasts using convolutional neural networks. Monthly Weather Review , 149 (4), 1141–1152. 22 Wessel, J. B., C. A. Ferro, and F. Kwasniok, 2024: Lead-time-continuous statistical post- processing of ensemble weather forecasts. Quarterly Journal of the Royal ...

  7. [70]

    Journal of Pharmaceutical Analysis , 101081

    Jiang, J., and Coauthors, 2024: A review of transformers in drug discovery and beyond. Journal of Pharmaceutical Analysis , 101081. Jung, C., and D. Schindler, 2019: Wind speed distribution selection – a review of recent development and progress. Renewable and Sustainable Energy Reviews , 114, 109 290, https://doi.org/https://doi.org/10.1016/j.rser.2019.1...

  8. [92]

    Gimpel, 2016: Bridging nonlinearities and stochastic regularizers with gaussian error linear units

    Hendrycks, D., and K. Gimpel, 2016: Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415, URL http://arxiv.org/abs/ 1606.08415, 1606.08415. Hersbach, H., and Coauthors, 2020: The era5 global reanalysis. Quarterly Journal of the Royal Meteorological Society, 146 (730), 1999–2049. 20 H¨ ohlein, K., B. Sc...

Show all 16 references
  1. [97]

    Lerch, 2022: Machine learning methods for postprocessing ensemble forecasts of wind gusts: A systematic comparison

    Schulz, B., and S. Lerch, 2022: Machine learning methods for postprocessing ensemble forecasts of wind gusts: A systematic comparison. Monthly Weather Review , 150 (1), 235–257. Sørensen, J., G. Larsen, and A. Cazin-Bourguignon, 2021: Production and cost assessment of offshore...

  2. [107]

    Naseer, M

    Khan, S., M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, 2022: Transformers in vision: A survey. ACM computing surveys (CSUR) , 54 (10s), 1–41. Lakatos, M., S. Lerch, S. Hemri, and S. Baran, 2023: Comparison of multivariate post- processing methods using global ecm...

  3. [162]

    S., 2021: Self-attentive ensemble transformer: Representing ensemble interactions in neural networks for earth system models

    Finn, T. S., 2021: Self-attentive ensemble transformer: Representing ensemble interactions in neural networks for earth system models. arXiv preprint arXiv:2106.13924 . Finn, T. S., 2023: Self-attentive ensemble transformer: Representing ensemble interactions in neural network...

  4. [301]

    super sapphire– phailin

    Mlakar, P., J. Merˇ se, and J. Faganeli Pucer, 2024: Ensemble weather forecast post- processing with a flexible probabilistic neural network approach. Quarterly Journal of the Royal Meteorological Society, 150 (764), 4156–4177. Mohanty, U., and Coauthors, 2015: A great escape ...

  5. [412]

    Pinson, P., and J. W. Messner, 2018: Application of postprocessing for renewable energy. Statistical postprocessing of ensemble forecasts, Elsevier, 241–266. Rabier, F., 2024: Longer ranges. URL hhttps://www.ecmwf.int/en/newsletter/179/ editorial/longer-ranges, accessed on May 7,

  6. [735]

    Zhang, and M

    Zhang, P., J. Zhang, and M. Chen, 2017: Economic impacts of climate change on agriculture: The importance of additional climatic variables other than tempera- ture and precipitation. Journal of Environmental Economics and Management , 83, 8–31, https://doi.org/https://doi.org/...

  7. [987]

    Bi, K., L. Xie, H. Zhang, X. Chen, X. Gu, and Q. Tian, 2022: Pangu-weather: A 3d high-resolution model for fast and accurate global weather forecast. arXiv preprint arXiv:2211.02556. Bouall` egue, Z. B., J. A. Weyn, M. C. Clare, J. Dramsch, P. Dueben, and M. Chantry, 2024: Imp...

  8. [2024]

    Lerch, 2018: Neural networks for postprocessing ensemble weather fore- casts

    Rasp, S., and S. Lerch, 2018: Neural networks for postprocessing ensemble weather fore- casts. Monthly Weather Review , 146 (11), 3885–3900. Rumes, B., R. Brabant, and L. Vigin, 2022: Offshore renewable energy in the belgian part of the north sea. MEMOIRS,

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.