REVIEW 3 major objections 4 minor 23 references
Exploring Design Choices for Autoregressive Deep Learning Climate Models
T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The paper establishes that stable 10-year climate rollouts are achievable with all three tested deep-learning weather architectures when training uses multiple autoregressive steps and moderate model capacity, and that SFNO is the most…
desk verdict A useful empirical sweep of autoregressive training choices for stable climate rollouts, though the statistical-fidelity claim outruns the metrics used to support it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the multi-step autoregressive training loss, defined as the summed area-weighted mean squared error over M consecutive 6-hour forward steps, with gradients computed only after summing all M steps. This objective forces the model to remain stable under its own errors rather than only one step ahead. Around it, the evaluation uses a single 10-year rollout initialized on 1 January 2009 and compares the rollout's temporal mean and temporal standard deviation to ERA5 via area-weighted normalized RMSE on five variables (tas, uas, vas, ta850, zg500). The three architectures embody different inductive biases: SFNO applies spherical harmonics transforms on the sphere, FourCastNet uses adaptive Fourier neural operators on the latitude–longitude grid, and ClimaX uses a vision transformer with variable tokenization; the paper's grid search over steps, layers, hidden dimension, and variable count is what connects these biases to observed stability.
What would settle it
Take a configuration the paper reports as stable, such as SFNO with 4 training steps, 33 prognostic variables, hidden dimension 512, and seed 597, initialize the same trained model on several different dates (for instance 1 January 1979, 1 January 1990, and 1 July 2009), and recompute the area-weighted normalized RMSE of the temporal mean and temporal standard deviation for the five evaluated variables; if any initialization gives an RMSE clearly above the climatology baseline or produces diverging values for any variable, the claim that this configuration preserves reference climate statistics fails.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that long-run stability is a property of the training objective and capacity, not of the architecture's spherical geometry. Ten-year rollouts with area-weighted normalized RMSE close to the climatology baseline are achieved by FourCastNet, SFNO, and ClimaX alike, as long as training unrolls over multiple autoregressive steps (M=2 or 4) and uses a moderate hidden dimension for the non-geometry models (D=128 or 256). SFNO tolerates larger capacity and more variable sets; for SFNO the trend is that bigger hidden dimensions improve the rollout, while FourCastNet and ClimaX overfit near the poles when capacity grows. The paper verifies that stable rollouts also match the temporal standard deviation of the reference period, and it treats climatology as the lower bound because the models are not driven by greenhouse-gas or ocean forcings. At the same time, every architecture contains configurations where some random seeds cause the 10-year rollout to blow up, and stability for one set of prognostic variables does not imply stability for the other.
Load-bearing premise
The conclusion assumes that a single 10-year rollout that matches ERA5's temporal mean and standard deviation for five variables is enough to certify that a model has stable, physically consistent climate statistics.
Editorial extensions
If this is right
- Multi-step autoregressive training (two or four 6-hour steps per loss term) should become the default for climate-length rollouts: one-step training rarely reaches RMSE close to climatology.
- Non-spherical architectures are viable climate emulators if capacity is kept moderate (hidden dimension 128 or 256), which expands the set of usable backbones beyond those with spherical geometry.
- Stability results should be reported per seed: a configuration can look stable on average while individual seeds blow up, so seed counts and failure rates are necessary parts of any long-term stability claim.
- A configuration that is stable for one set of prognostic variables can be unstable for another, so the variable set is a design choice that must be tuned together with the architecture.
- At this coarse resolution, the identified stable configurations provide a reproducible baseline for studying data-driven climate emulation at decade scales, since they reproduce the reference mean and variability.
Reading between the lines
- If the same stability plateau holds at higher resolution, the paper's recipe—multi-step training, moderate hidden dimension, and a small prognostic set for dense models—would give a concrete starting point for kilometer-scale learned climate models; the paper explicitly leaves this untested.
- The strong seed dependence suggests that averaging several seed-trained models, or adding spectral or noise regularization, might reduce divergence rates; that is a testable extension of the paper's own finding.
- Because only one initialization date was used, an immediate stress test is to roll out the same stable configurations from different starting dates and seasons and compare the spread of the 10-year means; preliminary tests by the authors suggest no effect, but the evidence is not shown.
- The evaluation metric is a necessary condition rather than a full certificate: checking spatial patterns, spectral energy, and teleconnections would reveal whether low RMSE masks physically wrong variability.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper reports a systematic empirical study of three autoregressive deep-learning weather/climate models (FourCastNet, SFNO, ClimaX) trained on WeatherBench1 ERA5 data at 5.625 degrees resolution. The authors grid-search the number of autoregressive training steps (1, 2, 4), layer count (4, 6, 8), hidden dimension (128, 256, 512), and two prognostic-variable sets (8 and 33 variables), with 10 random seeds per configuration, and evaluate 10-year rollouts initialized on 1 Jan 2009. The primary metric is the area-weighted normalized RMSE between the rollout's temporal mean and the ERA5 temporal mean over 2009-2018, averaged over five variables, with a climatology baseline; Appendix Fig. 4 reports the same comparison for temporal standard deviation. The main empirical findings are that all three architectures can produce rollouts with RMSE near climatology, that multi-step training strongly improves stability, that FCN and ClimaX require moderate hidden dimensions while SFNO benefits from larger capacity, and that SFNO is the most robust across hyperparameters, although every architecture exhibits severe seed-dependent divergence for some configurations.
Significance. If the results hold, the paper provides a useful and unusually broad empirical map of design choices for stable long-horizon autoregressive atmospheric models, complementing prior architecture-specific demonstrations. Strengths include the systematic grid over architecture, training depth, capacity, variable set, and seeds; the use of held-out ERA5 data with a clear climatology baseline; and public code. The paper also honestly documents seed-induced instability across all architectures, which is a valuable caution for practitioners. The main limitation is that 'preserving statistical properties' is operationalized only through first and second marginal temporal moments of five variables from a single initial condition; spatial, spectral, and multivariate statistics are not checked, so the headline claim is stronger than the evidence.
major comments (3)
- [Section 2, Evaluation; Section 3] The claim that stable rollouts 'preserv[e] the statistical properties of the reference dataset' (Abstract) is under-supported by the evaluation, which compares only the temporal mean (and, in Appendix Fig. 4, the temporal standard deviation) of five variables via area-weighted RMSE. Models matching these marginal moments can still have incorrect spatial correlations, spectral energy distributions, storm-track variability, or cross-variable joint distributions; I ask the authors to add diagnostics such as spatial pattern correlations, wavenumber spectra, or joint statistics, or to explicitly reframe the claim as stability of the first two marginal moments.
- [Section 2, Evaluation] The evaluation relies on a single 10-year rollout from 1 Jan 2009, and the statement that multiple initial conditions 'does not effect the long-term behavior' is not supported by any documented experiment. Because the paper's conclusion is about the learned attractor rather than one trajectory, the preliminary experiment should be reported, or rollouts from several initial conditions should be included.
- [Section 3, Figures 1-3] The robustness ranking (for example, 'substantially higher' number of SFNO runs with RMSE near climatology) is based on visual thresholds in Figures 1-3, and the notion of an RMSE score 'close to climatology' is never given a quantitative threshold or uncertainty estimate. Please specify the threshold and, ideally, report confidence intervals or a simple statistical comparison across seeds, since the central ranking claims rest on these counts.
minor comments (4)
- [Section 2] The text calls the metric 'mean squared error (RMSE)'; this is inconsistent, and the paper should state whether the plotted values are root mean square errors or mean squared errors.
- [Appendix Figures 1-5] The figures say 'Traininig Variables' where 'prognostic variables' is meant; this typo appears in several captions.
- [Section 2, Evaluation] The paper should state whether the temporal standard deviation is computed from 6-hourly values or from daily means, and over which period.
- [Section 2, Evaluation] The phrase 'does not effect the long-term behavior' should be 'does not affect the long-term behavior.'
Circularity Check
No circularity: the paper makes empirical comparisons against held-out ERA5 data and a climatology baseline, with no fitted parameter or derivation that reduces to its conclusions.
full rationale
This paper is an empirical benchmark study, not a derivation. It trains three existing architectures on ERA5-derived WeatherBench data and evaluates 10-year rollouts by comparing temporal means and standard deviations against the held-out test period, with climatology from the training period used only as a pre-registered baseline. There is no fitted parameter that is later renamed a prediction, no quantity defined in terms of the target metric, and no uniqueness theorem or load-bearing self-citation invoked to force the conclusions. The hyperparameters are inherited from prior work (e.g., Karlbauer et al. 2024b), but those citations are ordinary engineering choices and do not establish the paper's central claims. The authors also explicitly acknowledge the scope limitation that stability is only a necessary criterion and that full climate-model evaluation is more complex; this is an honest scope statement, not a circular step. All load-bearing evidence is generated from held-out data and external baselines, so the central claims are not equivalent to their inputs by construction.
Assumptions & free parameters
assumptions (5)
- domain assumption ERA5 reanalysis interpolated to 5.625 degrees (WeatherBench1) is a sufficient reference for studying long-term climate stability.
- domain assumption Stability and preservation of statistical properties can be measured by area-weighted RMSE of the temporal mean and temporal standard deviation of five common variables.
- domain assumption A single 10-year rollout from 1 January 2009 is representative of long-term model behavior.
- domain assumption Climatology computed over the training period is an appropriate upper-bound baseline because external forcings such as greenhouse gases and ocean state are absent.
- standard math Area weights in the loss and evaluation correctly account for grid distortion on the regular latitude-longitude grid.
Cite this review
Pith. "Pith review of Exploring Design Choices for Autoregressive Deep Learning Climate Models." pith.science (2026). https://pith.science/paper/YDKG43L5
@misc{pith2026250502506,
author = {Pith},
title = {Pith review of: Exploring Design Choices for Autoregressive Deep Learning Climate Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/YDKG43L5}},
note = {Machine review of arXiv:2505.02506}
}
read the original abstract
Deep Learning models have achieved state-of-the-art performance in medium-range weather prediction but often fail to maintain physically consistent rollouts beyond 14 days. In contrast, a few atmospheric models demonstrate stability over decades, though the key design choices enabling this remain unclear. This study quantitatively compares the long-term stability of three prominent DL-MWP architectures - FourCastNet, SFNO, and ClimaX - trained on ERA5 reanalysis data at 5.625{\deg} resolution. We systematically assess the impact of autoregressive training steps, model capacity, and choice of prognostic variables, identifying configurations that enable stable 10-year rollouts while preserving the statistical properties of the reference dataset. Notably, rollouts with SFNO exhibit the greatest robustness to hyperparameter choices, yet all models can experience instability depending on the random seed and the set of prognostic variables
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Table 1: Model Specific Hyperparameters ClimaX FourCastNet SFNO Patch size: (2,2) Patch size: (1,1) Scale Factor: 1 # Heads: 8 # Blocks: 4 Big Skip: False Decoder Depth: 2 Use Pos. Embed.: False Use Pos. Embed.: False MLP Ratio: 4 MLP Ratio: 4 Use MLP: True Drop Rate: 0.0 Drop Rate: 0.0 Grid: ”Equiangular” Drop Path: 0.0 Drop Path: 0.0 factorization: Null...
work page 2025
-
[3]
Nathaniel Cresswell-Clay, Bowen Liu, Dale Durran, Andy Liu, Zachary I
doi: 10.48550/arXiv.2304.07029. Nathaniel Cresswell-Clay, Bowen Liu, Dale Durran, Andy Liu, Zachary I. Espinosa, Raul Moreno, and Matthias Karlbauer. A Deep Learning Earth System Model for Stable and Efficient Simulation of the Current Climate. (arXiv:2409.16247), September
-
[4]
doi: 10.48550/arXiv.2409.16247. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learnin...
-
[7]
John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro
doi: 10.48550/arXiv.2405.16297. John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro. Efficient Token Mixing for Transformers via Adaptive Fourier Neural Operators. InInternational Conference on Learning Representations, October
-
[8]
The mean (dots) and standard deviation (error bars) were computed across 10 seeds (crosses) with finite RMSE. For better readability, the y-axis is cut-off at 0.5 and the number of displayed runs out of 10 is shown on the x-axis. 9 Published as a workshop paper at ”Tackling Climate Change with Machine Learning”, ICLR 2025 0 0/10 1 10/10 2 10/10 3 2/10 4 1...
work page 2025
-
[9]
Matthias Karlbauer, Nathaniel Cresswell-Clay, Dale R
doi: 10.1002/qj.3803. Matthias Karlbauer, Nathaniel Cresswell-Clay, Dale R. Durran, Raul A. Moreno, Thorsten Kurth, Boris Bonev, Noah Brenowitz, and Martin V . Butz. Advancing Parsimonious Deep Learning Weather Prediction Using the HEALPix Mesh. Journal of Advances in Modeling Earth Systems, 16(8):e2023MS004021, July 2024a. doi: 10.1029/2023MS004021. Matt...
-
[10]
doi: 10.1038/s41586-024-07744-y. Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Fer- ran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, Alexander Merose, Stephan Hoyer, George Holland, Oriol Vinyals, Jacklynn Stott, Alexander Pritzel, Shakir Mo- hamed, and Peter Battaglia. Learning skillful medium-ra...
-
[11]
doi: 10.1126/science.adi2336. Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhat- tacharya, Andrew Stuart, and Anima Anandkumar. Fourier Neural Operator for Parametric Partial Differential Equations. In International Conference on Learning Representations, October
Show all 23 references
-
[13]
FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, Pedram Has- sanzadeh, Karthik Kashinath, and Animashree Anandkumar. FourCastNet: A Global Data-driven Hig...
-
[16]
6 Published as a workshop paper at ”Tackling Climate Change with Machine Learning”, ICLR 2025 Sebastian Scher and Gabriele Messori
doi: 10.1029/ 2020MS002203. 6 Published as a workshop paper at ”Tackling Climate Change with Machine Learning”, ICLR 2025 Sebastian Scher and Gabriele Messori. Weather and climate forecasting with neural networks: Using general circulation models (GCMs) with different complexi...
2025
-
[18]
Oliver Watt-Meyer, Gideon Dresdner, Jeremy McGibbon, Spencer K
doi: 10.48550/arXiv.2406.08632. Oliver Watt-Meyer, Gideon Dresdner, Jeremy McGibbon, Spencer K. Clark, Brian Henn, James Duncan, Noah D. Brenowitz, Karthik Kashinath, Michael S. Pritchard, Boris Bonev, Matthew E. Peters, and Christopher S. Bretherton. ACE: A fast, skillful lea...
- [19]
-
[20]
doi: 10.1029/2020MS002109. 7 Published as a workshop paper at ”Tackling Climate Change with Machine Learning”, ICLR 2025 A P ARAMETER SETTINGS We train each model configuration for 20 epochs at float32 precision and employ early stopping when the validation loss has not improv...
2024 doi
-
[23]
For better readability, the y-axis is cut-off at 0.5 and the number of displayed runs out of 10 is shown on the x-axis
The mean (dots) and standard deviation (error bars) were computed across 10 seeds (crosses) with finite RMSE. For better readability, the y-axis is cut-off at 0.5 and the number of displayed runs out of 10 is shown on the x-axis. 0 0/10 1 10/10 2 10/10 3 2/10 4 10/10 5 10/10 6...
2025
-
[2016]
Haiwen Guan, Troy Arcomano, Ashesh Chattopadhyay, and Romit Maulik
doi: 10.5194/gmd-9-1937-2016. Haiwen Guan, Troy Arcomano, Ashesh Chattopadhyay, and Romit Maulik. LUCIE: A Lightweight Uncoupled ClImate Emulator with long-term stability and physical consistency for O(1000)- member ensembles. (arXiv:2405.16297), May
1937 arXiv
-
[2017]
Michael McCabe, Peter Harrington, Shashank Subramanian, and Jed Brown
doi: 10.5194/gmd-10-2247-2017. Michael McCabe, Peter Harrington, Shashank Subramanian, and Jed Brown. Towards Stability of Autoregressive Neural Operators. Transactions on Machine Learning Research, June
2017 doi
-
[2019]
Chenggong Wang, Michael S
doi: 10.5194/gmd-12-2797-2019. Chenggong Wang, Michael S. Pritchard, Noah Brenowitz, Yair Cohen, Boris Bonev, Thorsten Kurth, Dale Durran, and Jaideep Pathak. Coupled Ocean-Atmosphere Dynamics in a Machine Learning Earth System Model. (arXiv:2406.08632), June
2019 arXiv
-
[2020]
Meehl, Catherine A
Veronika Eyring, Sandrine Bony, Gerald A. Meehl, Catherine A. Senior, Bjorn Stevens, Ronald J. Stouffer, and Karl E. Taylor. Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization. Geoscientific Model Development , 9(5):1937–...
1937
-
[2021]
Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, Andr´as Hor´anyi, Joaqu´ın Mu˜noz-Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, Adrian Simmons, Cornel Soci, Saleh Abdalla, Xavier Abellan, Gianpaolo Balsamo, Peter Bechtold, Gionata Biavati, Je...
2025
-
[2022]
Ilan Price, Alvaro Sanchez-Gonzalez, Ferran Alet, Tom R
doi: 10.48550/arXiv.2202.11214. Ilan Price, Alvaro Sanchez-Gonzalez, Ferran Alet, Tom R. Andersson, Andrew El-Kadi, Do- minic Masters, Timo Ewalds, Jacklynn Stott, Shakir Mohamed, Peter Battaglia, Remi Lam, and Matthew Willson. Probabilistic weather forecasting with machine le...
-
[2023]
Boris Bonev, Thorsten Kurth, Christian Hundt, Jaideep Pathak, Maximilian Baust, Karthik Kashinath, and Anima Anandkumar
doi: 10.1038/s41586-023-06185-3. Boris Bonev, Thorsten Kurth, Christian Hundt, Jaideep Pathak, Maximilian Baust, Karthik Kashinath, and Anima Anandkumar. Spherical Fourier Neural Operators: Learning Stable Dy- namics on the Sphere. In Proceedings of the 40th International Conf...
-
[2024]
Long-term instabilities of deep learning-based digital twins of the climate system: The cause and a solution
Ashesh Chattopadhyay and Pedram Hassanzadeh. Long-term instabilities of deep learning-based digital twins of the climate system: The cause and a solution. (arXiv:2304.07029), April
-
[2025]
Stephan Rasp, Peter D
doi: 10.1038/s41586-024-08252-9. Stephan Rasp, Peter D. Dueben, Sebastian Scher, Jonathan A. Weyn, Soukayna Mouatadid, and Nils Thuerey. WeatherBench: A Benchmark Data Set for Data-Driven Weather Forecasting. Journal of Advances in Modeling Earth Systems , 12(11):e2020MS002203,
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.