Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Across five models, wider beats deeper and more data beats more parameters in weather forecasting scaling.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 20:32 UTC pith:CZWOASSN

load-bearing objection A genuinely useful first cross-model scaling study for weather ML, with a real width-over-depth finding; the compute-optimal 'favor D over N' conclusion is the weak load-bearing piece and needs refitting or major tempering. the 4 major comments →

arxiv 2602.22962 v2 pith:CZWOASSN submitted 2026-02-26 cs.LG

Scaling Laws of Global Weather Models

Yuejiang Yu , Langwen Huang , Alexandru Calotoiu , Torsten Hoefler This is my paper
classification cs.LG
keywords scaling lawsweather forecastingcompute-optimal trainingmodel width vs depthdata scalingparameter scalingglobal weather modelsvalidation loss
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that global data-driven weather models obey quantitative scaling laws: validation loss falls as a power law in both model size (parameters N) and training data (D), and these fitted curves can guide architecture and compute allocation. Its distinctive finding is that weather models scale differently from language models: at matched parameter counts, wider configurations consistently beat deeper ones, and GraphCast and SFNO perform well even at depth one, suggesting six-hour dynamics are nearly linear. It also finds that under a fixed compute budget, allocating FLOPs to more training data beats allocating them to more parameters across the tested models. The authors report model-specific exponents — Aurora has the steepest data-scaling exponent (β≈0.51) while GraphCast is the most parameter-efficient — and caution that variables scale heterogeneously, so aggregate loss is only a rough performance indicator. If correct, the practical prescription is to build future weather models wider and train them longer on more data.

Core claim

The paper's central claim is that the 6-hour one-step forecast error, measured as an area- and variance-normalized weighted MSE on ERA5, obeys L(D)=αD^{-β} and L(N)=γN^{-δ} across five diverse weather models trained under unified conditions. Fitting these curves yields data exponents β from 0.30 to 0.51 and parameter exponents δ from 0.14 to 0.32, with Aurora showing the strongest data scaling (a 10x data increase cuts loss up to 3.2x) and GraphCast the strongest parameter efficiency. The distinctive result is shape: for roughly equal parameter counts, wider variants always reach lower validation loss than deeper variants, and GraphCast and SFNO perform well even at depth 1, which the paper

What carries the argument

The load-bearing machinery is the empirical power-law relation L(X)=αX^{-β} for X∈{D,N}, fitted by log-log linear regression, combined with a compute-accounting model that expresses training FLOPs per step for each architecture (e.g., GraphCast's parameter count N=(24+8d)w+(18+7d)w², and three-times-forward FLOP estimates). The power-law fits convert raw training runs into cross-model comparisons — data-scaling exponent β, parameter-scaling exponent δ — and the compute-optimal analysis fits loss against data volume at fixed compute budgets (parabolas) whose minima give the optimal N/D split. This machinery is what allows the paper to attribute differences to architecture and shape rather tha

Load-bearing premise

The analysis's load-bearing premise is that a single-step, 6-hour, variance-normalized mean squared error is a faithful proxy for the forecast skill operational users care about; the paper itself shows variables scale heterogeneously and does not connect the fitted scaling to multi-step or longer-lead verification.

What would settle it

Train the wide and narrow variants of one model (e.g., GraphCast width 256 depth 1 versus width 128 depth 12) to convergence on the same data, then evaluate RMSE and CRPS at 24h, 72h, and 168h lead times. If the deeper variant wins at longer leads, or if the compute-optimal N/D split reverses under multi-step evaluation, the paper's central claims are artifacts of the 6-hour proxy.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Future global weather models should be designed wider and shallower: matched-parameter comparisons across all five models favor width, and GraphCast and SFNO perform well even at depth 1.
  • Under a fixed compute budget, allocating FLOPs to additional training data yields more forecast-error reduction than allocating to additional parameters; operational systems should prioritize smaller models trained for longer durations.
  • Aurora's data-scaling exponent (β≈0.51) means a 10x data increase cuts its validation loss by up to 3.2x, so data collection and curation is a high-leverage investment for that architecture.
  • Because variable-specific scaling is heterogeneous, aggregate validation loss is only a rough indicator; per-variable evaluation is needed to rank models for specific forecast products.
  • Hardware utilization matters: GraphCast's superior parameter efficiency does not translate to wall-clock efficiency, so scaling-law comparisons should account for achieved FLOPs per second.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the 6-hour one-step scaling transfers to multi-step and longer-range forecasts, the width-over-depth result implies operational models could be made much shallower (even depth 1), cutting latency and memory; a direct test would be training depth-1 variants and evaluating 10-day skill.
  • The near-linear dynamics at 6 hours suggest that much of the nonlinear capacity in existing deep weather models is idle at short lead times; an interesting extension is to test whether adding a shallow but wide correction module on top of a linear baseline reproduces the scaling exponents.
  • The paper's compute-optimal analysis treats compute as FLOPs, but GraphCast's 0.017% GPU utilization versus Aurora's 37.2% shows that wall-clock-aware scaling laws could change the optimal N/D split; a time-budgeted version of the parabola analysis is a natural next step.
  • Variable heterogeneity implies a multi-task scaling-law objective; fitting per-variable exponents might allow training curricula that allocate data and parameters to the hardest variables rather than a single weighted loss.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports a cross-model empirical study of scaling laws for five global weather forecasting models (Aurora, AIFS, Pangu, GraphCast, SFNO) trained on ERA5 under a unified evaluation protocol. It claims that validation loss follows power laws in dataset size D and model size N, that weather models consistently favor wider over deeper architectures at fixed parameter counts, and that compute-optimal allocation favors increasing training data over model size. It also reports variable-specific scaling heterogeneity and hardware-utilization differences across models. The authors provide a reproducibility-oriented appendix with model configurations, FLOP formulas, and a unified loss definition.

Significance. If the claims hold, this is a useful first cross-architecture scaling-law study for data-driven weather forecasting, with practical guidance for model design and data allocation. The paper's strengths include a broad model coverage (five distinct architectures), a standardized validation loss, a large experimental effort (430,000+ GPU hours), and a public code release. The width-over-depth finding, in particular, is a clear and reproducible empirical contribution that contrasts with language-model scaling behavior. However, the compute-optimal recommendation, one of the two headline prescriptive claims, is currently under-supported by the reported evidence, and several statistical quantities that underpin the model rankings are reported without uncertainty.

major comments (4)
  1. [Sec. 3.3, Fig. 5] The central compute-optimal claim — that 'as compute budgets increase, the optimal allocation strategy still favors increasing D over N' — is not derivable from the reported IsoFLOP evidence. The text states that GraphCast and AIFS show 'mainly the left half of the parabolas,' that SFNO's parabolas are 'mostly in the left half,' and that for Pangu 'we do not observe a clear relationship between L and C.' One-sided parabola segments do not bracket a minimum, and no fitted values of Nopt, Dopt, or the exponents a,b in Nopt∝C^a, Dopt∝C^b are reported anywhere. The claim therefore appears to be read off from incomplete curves rather than from a fitted compute-optimal frontier. The authors should either provide complete parabolas, fitted minima with confidence intervals, or substantially weaken the conclusion to a suggestive trend.
  2. [Sec. 1 vs. Sec. 3.1, Figs. 2–3] The paper defines D as cumulative training samples ('training progress measured in samples seen') in Section 1, but the abstract, figure axes, and Section 3.1 treat D as dataset size in TB (e.g., 'train using different values of D,' 'increasing the training dataset by 10x'). These are different quantities: if D is cumulative samples, increasing D can mean more epochs over the same fixed data, not a larger dataset. The conflation affects the interpretation of the data-scaling exponents and the compute-optimal recommendation. The authors must clarify whether D was varied by changing the number of unique training samples or by changing the number of training steps, and adjust the language consistently.
  3. [Sec. 3.1, Table 1, Figs. 2–3] The fitted scaling exponents β and δ are load-bearing for the cross-model rankings ('Aurora demonstrates the strongest data-scaling behavior,' 'GraphCast demonstrates the highest parameter efficiency'). Yet the paper reports only point estimates (e.g., β≈0.51 for Aurora vs. β≈0.46 for AIFS in Table 1) without confidence intervals, the number of training runs per fit, or goodness-of-fit statistics such as R² or residual error. With few data points and steep fits, these rankings may not be statistically distinguishable. The authors should report uncertainties, the number of configurations per fit, and the fit quality for every exponent.
  4. [Sec. 2.3, Sec. 3.4, Sec. 4] All scaling-law analyses and prescriptive recommendations are based on a single-step, 6-hour, normalized validation MSE. The paper itself acknowledges in Section 3.4 that 'the weighted loss across all variables is only a rough indicator of overall model performance' and shows that variable-specific scaling is heterogeneous. Without connecting the 6-hour one-step loss to multi-step forecast skill (e.g., 24-hour to 10-day verification) or to at least a consistent variable-wise scaling analysis, the operational recommendations about architecture and data allocation rest on an unvalidated proxy. The authors should either provide such verification or explicitly scope the conclusions to 6-hour one-step loss.
minor comments (5)
  1. [Abstract / Sec. 1] The abstract says 'allocating resources to more total training data' while Section 1 says 'allocating resources to longer training durations'; these are inconsistent, especially given the ambiguous definition of D. Please align the wording.
  2. [Appendix B] The symbol C is used both for compute budget and for the effective number of input channels in Appendix B (e.g., 'C denotes the effective number of input channels'). This is confusing and should be resolved with distinct notation.
  3. [Appendix E.1] The text says 'Chapter 3' but should refer to 'Section 3.' Also, Figure 7 panels are labeled 'a b c' without a clear legend tying them to the three variables; please clarify.
  4. [References / Sec. 2.1] The reference 'V on Oswald' appears in the text and references; this should be 'Von Oswald.' Additionally, the citation 'Bouallegue et al., 2024' appears both for the BAMS paper and for ArchesWeather; the two entries have overlapping author names and could be confused.
  5. [Fig. 5] The figure caption and panel labels ('a) c)') are inconsistent; the panels should be clearly enumerable and referenced in the text.

Circularity Check

0 steps flagged

No significant circularity: exponents and compute-optimal curves are empirical fits reported as fits; the weakly supported 'favor D over N' claim is an evidence limitation, not a circular derivation.

full rationale

The paper's central claims are empirical scaling-law fits, not derivations that reduce to their inputs. The data-scaling statement 'increasing the training dataset by 10x reduces validation loss by up to 3.2x' is a direct restatement of the fitted exponent β≈0.51 in L(D)=αD^−β, but the paper presents it as a finding ('we find', Sec. 3.1), not as an out-of-sample prediction; reporting the implication of a fitted power law is the normal content of a scaling law, not circularity. The compute-optimal conclusion is read from parabolic fits to the same training runs, which is standard scaling-law methodology rather than a self-definitional reduction. The paper itself states limitations that weaken the compute-optimal claim: 'For GraphCast and AIFS ... the parabolas appear primarily in the left half,' 'the right half of the parabolas need even smaller width,' and 'for Pangu, we do not observe a clear relationship between L and C' (Sec. 3.3). These admissions show the 'favor D over N' conclusion is under-supported by the reported evidence, but under-support is a correctness/robustness concern, not circularity. There is no load-bearing self-citation chain: the only reference involving a present author (Bauer et al., 2021, with Hoefler) supports a general historical sentence about data-driven forecasting and is not used to justify any scaling-law result. No uniqueness theorem is imported, no ansatz is smuggled in via self-citation, and no known result is merely renamed. The width-over-depth finding rests on the paper's own matched-parameter experiments (Fig. 4), and the variable-specific scaling analysis is an additional empirical observation. Therefore, no specific reduction from a claimed derivation to its own inputs can be exhibited.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The paper's central quantitative content is almost entirely fitted: the α, β, γ, δ exponents and the compute-optimal parabola coefficients are numbers obtained from the authors' own training runs. The FLOPs formulas introduce several hand-chosen constants (AIFS sparsity factor, SFNO transform constant, 3× training multiplier). The main non-fitted, directly observed result is the width-over-depth comparison, which does not depend on the fitted power laws.

free parameters (7)
  • Data-scaling exponents β per model = Aurora 0.51, AIFS 0.46, Pangu 0.43, GraphCast 0.36, SFNO 0.34
    Fitted by log-log regression of L(D) in Figure 2; used to rank models by data efficiency. No confidence intervals or number of fitted points are reported.
  • Data-scaling intercepts α per model = Values in Figure 2, e.g., 11.54, 6.95, 8.29, 6.48, 1.02, etc.
    Fitted intercepts of L(D)=αD^-β; model ranking by α depends on the D range and is not emphasized.
  • Parameter-scaling exponents δ per model = Not stated numerically; shown as curves in Figure 3
    Fitted from L(N) at D=15 and 30 TB; used to assess parameter efficiency, but values and error bars are omitted.
  • Parameter-scaling intercepts γ per model = Not stated numerically
    Fitted in Figure 3; used to compare initial performance at small N.
  • Compute-optimal parabola coefficients = Not reported
    The parabolas in Figure 5 are fitted to L vs D at fixed C; the claimed exponents N_opt ∝ C^a, D_opt ∝ C^b are never reported, making the compute-optimal recommendation hard to reproduce.
  • AIFS attention sparsity factor = N_h^2/16
    Ad hoc constant in the Appendix B FLOPs formula for AIFS processor attention; directly affects the compute accounting used for compute-optimal claims.
  • SFNO spherical-transform complexity constant α = Unspecified
    Hand-chosen constant in the Appendix B FLOPs formula for spherical harmonic transforms; affects SFNO compute estimates.
axioms (5)
  • domain assumption Power-law functional forms L(D)=αD^−β and L(N)=γN^−δ hold for weather models.
    Sec. 3.1 assumes these forms from the NLP scaling-law literature and fits them; no alternative functional forms or goodness-of-fit tests are reported.
  • domain assumption D, defined as cumulative training samples seen, is treated as 'dataset size' in the interpretation.
    Sec. 1 says D is cumulative samples ingested, but the abstract, Figure 2, and recommendations interpret it as training dataset size. This conflation underpins the data-scaling claims.
  • ad hoc to paper Compute scales as C ~ N·B for graph models and C ≈ 6ND/p² for transformers, with a+b=1.
    Sec. 3.3 and Appendix B. The FLOPs formulas are hand-derived and not validated against measured runtimes; the a+b=1 constraint is imposed from prior work.
  • domain assumption Training FLOPs equal 3× forward FLOPs for all models.
    Appendix B states this uniform 3-to-1 training-to-forward ratio for every model without empirical measurement.
  • domain assumption 6-hour single-step normalized validation MSE is a sufficient proxy for forecast skill.
    All main scaling curves use 6-hour validation loss; no multi-step forecast verification is connected to the fitted scaling behavior, and Sec. 3.4 shows variable-specific scaling is heterogeneous.

pith-pipeline@v1.3.0-alltime-deepseek · 17210 in / 13250 out tokens · 128460 ms · 2026-08-02T20:32:02.665633+00:00 · methodology

0 comments
read the original abstract

Data-driven models are revolutionizing weather forecasting. To optimize training efficiency and model performance, this paper analyzes empirical scaling laws within this domain. We investigate the relationship between model performance (validation loss) and three key factors: model size ($N$), dataset size ($D$), and compute budget ($C$). Across a range of models, we find that Aurora exhibits the strongest data-scaling behavior: increasing the training dataset by 10x reduces validation loss by up to 3.2x. GraphCast demonstrates the highest parameter efficiency, yet suffers from limited hardware utilization. Our compute-optimal analysis indicates that, under fixed compute budgets, allocating resources to more total training data yields greater performance gains than increasing model size. Furthermore, we analyze model shape and uncover scaling behaviors that differ fundamentally from those observed in language models: weather forecasting models consistently favor increased width over depth. These findings suggest that future weather models should prioritize wider architectures and larger effective training datasets to maximize predictive performance.

Figures

Figures reproduced from arXiv: 2602.22962 by Alexandru Calotoiu, Langwen Huang, Torsten Hoefler, Yuejiang Yu.

Figure 1
Figure 1. Figure 1: Scaling behavior of global weather models. We report the validation loss during training to evaluate model performance. We also report the training data required to reach 25 Pflop compute budget. availability of high-resolution observational data and ad￾vances in large-scale deep-learning infrastructure have been pushing these models even further. In the near future, it is expected that weather models will… view at source ↗
Figure 2
Figure 2. Figure 2: Data-scaling laws across weather forecasting models: L(D) = αD−β . Aurora (red) achieves the best L at D = 100 TB and also has the best β value, representing most efficient scaling with more data. Zhai et al., 2022), model shape plays a significant role in weather models. Then we explain why compute-optimal scaling of weather models is different from language models. We also compare scaling performance of … view at source ↗
Figure 4
Figure 4. Figure 4: Wider models perform better. For each model, we use two configurations: one is wider and one is narrower. They have roughly the same N but different shapes. In all models, wider configurations consistently achieve lower validation loss. This implies weather forecasting benefits more from representational capacity (width) than from additional nonlinear transformations (depth). In addition to studying the ov… view at source ↗
Figure 5
Figure 5. Figure 5: Compute-Optimal Training. The panels illustrate L as a function of D, with each curve representing a fixed C. The resulting parabolas identify the compute-optimal frontier—the specific ratio of N and D that minimizes loss for a given C. For each model, we have C ∼ ND, which implies N ∼ C/D along each curve. For Pangu, the loss is predominantly determined by D. Aurora and SFNO exhibit a clear minimum in the… view at source ↗
Figure 6
Figure 6. Figure 6: Variable-specific loss scaling for 10m u-component of wind (10U) and 2m temperature (2T). We compare Aurora, GraphCast, Pangu, SFNO, and AIFS. Validation RMSE is plotted against the volume of training data in TB. Each line represents a different model configuration, with line color indicating N (lighter colors for narrower models, darker for wider). For 10U, Aurora achieves the lowest RMSE at all dataset s… view at source ↗
Figure 7
Figure 7. Figure 7: CRPS comparison for 2m temperature (2T), temperature at 500 hPa (t500), and 10 m u-velocity of wind (u10) using 10 ensemble members. Results with 10 ensembles are within 2–5% of those obtained with 50 members, indicating minimal loss of accuracy and supporting the efficiency of reduced ensemble sizes. Model specifications are provided in [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Earth System Foundation Model (ESFM): A unified framework for heterogeneous data integration and forecasting

    physics.ao-ph 2026-04 unverdicted novelty 6.0

    ESFM is a single open foundation model that unifies heterogeneous Earth data sources and forecasts missing regions while preserving inter-variable physical relationships.

Reference graph

Works this paper leans on

36 extracted references · 11 linked inside Pith · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    D., Hoefler, T., Quintino, T., Schulthess, T

    Bauer, P., Dueben, P. D., Hoefler, T., Quintino, T., Schulthess, T. C., and Wedi, N. P. The digital revolution of earth-system science. Nature Computational Science, 1 0 (2): 0 104--113, 2021

    2021

  3. [3]

    C., Magnusson, L., Gascon, E., Maier-Gerber, M., Janou s ek, M., Rodwell, M., Pinault, F., Dramsch, J

    Ben Bouallegue, Z., Clare, M. C., Magnusson, L., Gascon, E., Maier-Gerber, M., Janou s ek, M., Rodwell, M., Pinault, F., Dramsch, J. S., Lang, S. T., et al. The rise of data-driven weather forecasting: A first statistical assessment of machine learning--based weather forecasts in an operational-like context. Bulletin of the American Meteorological Society...

    2024

  4. [4]

    Accurate medium-range global weather forecasting with 3d neural networks

    Bi, K., Xie, L., Zhang, H., Chen, X., Gu, X., and Tian, Q. Accurate medium-range global weather forecasting with 3d neural networks. Nature, 619 0 (7970): 0 533--538, 2023

    2023

  5. [5]

    P., Lucic, A., Stanley, M., Brandstetter, J., Garvan, P., Riechert, M., Weyn, J., Dong, H., Vaughan, A., et al

    Bodnar, C., Bruinsma, W. P., Lucic, A., Stanley, M., Brandstetter, J., Garvan, P., Riechert, M., Weyn, J., Dong, H., Vaughan, A., et al. Aurora: A foundation model of the atmosphere. arXiv preprint arXiv:2405.13063, 1 0 (8), 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Spherical fourier neural operators: Learning stable dynamics on the sphere

    Bonev, B., Kurth, T., Hundt, C., Pathak, J., Baust, M., Kashinath, K., and Anandkumar, A. Spherical fourier neural operators: Learning stable dynamics on the sphere. In International conference on machine learning, pp.\ 2806--2823. PMLR, 2023

    2023

  7. [7]

    B., Clare, M

    Bouallegue, Z. B., Clare, M. C., Magnusson, L., Gascon, E., Maier-Gerber, M., Janou s ek, M., Rodwell, M., Pinault, F., Dramsch, J. S., Lang, S. T., et al. Archesweather & archesweathergen: a deterministic and generative model for efficient ml weather forecasting. arXiv preprint arXiv:2412.12971, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  9. [9]

    Fengwu: Pushing the skillful global medium-range weather forecast beyond 10 days lead

    Chen, K., Han, T., Gong, J., Bai, L., Ling, F., Luo, J.-J., Chen, X., Ma, L., Zhang, T., Su, R., et al. Fengwu: Pushing the skillful global medium-range weather forecast beyond 10 days lead. arXiv preprint arXiv:2304.02948, 2023 a

    arXiv 2023

  10. [10]

    Fuxi: a cascade machine learning forecasting system for 15-day global weather forecast

    Chen, L., Zhong, X., Zhang, F., Cheng, Y., Xu, Y., Qi, Y., and Li, H. Fuxi: a cascade machine learning forecasting system for 15-day global weather forecast. npj climate and atmospheric science, 6 0 (1): 0 190, 2023 b

    2023

  11. [11]

    and Rao, J

    Deng, W. and Rao, J. Mega: More efficient graph attention for gnns. In 2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS), pp.\ 71--81. IEEE, 2024

    2024

  12. [12]

    ai-models: Run ai-based weather forecasting models with ecmwf data, 2024

    ECMWF Lab . ai-models: Run ai-based weather forecasting models with ecmwf data, 2024. URL https://github.com/ecmwf-lab/ai-models. GitHub repository

    2024

  13. [13]

    and Bengio, Y

    Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp.\ 249--256. JMLR Workshop and Conference Proceedings, 2010

    2010

  14. [14]

    Fengwu-ghr: Learning the kilometer-scale medium-range global weather forecasting

    Han, T., Guo, S., Ling, F., Chen, K., Gong, J., Luo, J., Gu, J., Dai, K., Ouyang, W., and Bai, L. Fengwu-ghr: Learning the kilometer-scale medium-range global weather forecasting. arXiv preprint arXiv:2402.00059, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp.\ 1026--1034, 2015

    2015

  16. [16]

    Scaling laws for transfer

    Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. Scaling laws for transfer. arXiv preprint arXiv:2102.01293, 2021

    Pith/arXiv arXiv 2021

  17. [17]

    The era5 global reanalysis

    Hersbach, H., Bell, B., Berrisford, P., Hirahara, S., Hor \'a nyi, A., Mu \ n oz-Sabater, J., Nicolas, J., Peubey, C., Radu, R., Schepers, D., et al. The era5 global reanalysis. Quarterly journal of the royal meteorological society, 146 0 (730): 0 1999--2049, 2020

    1999

  18. [18]

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    Pith/arXiv arXiv 2022

  19. [19]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  20. [20]

    Learning skillful medium-range global weather forecasting

    Lam, R., Sanchez-Gonzalez, A., Willson, M., Wirnsberger, P., Fortunato, M., Alet, F., Ravuri, S., Ewalds, T., Eaton-Rosen, Z., Hu, W., et al. Learning skillful medium-range global weather forecasting. Science, 382 0 (6677): 0 1416--1421, 2023

    2023

  21. [21]

    C., Lessig, C., Maier-Gerber, M., Magnusson, L., et al

    Lang, S., Alexe, M., Chantry, M., Dramsch, J., Pinault, F., Raoult, B., Clare, M. C., Lessig, C., Maier-Gerber, M., Magnusson, L., et al. Aifs--ecmwf's data-driven forecasting system. arXiv preprint arXiv:2406.01465, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    (mis)fitting scaling laws: A survey of scaling law fitting techniques in deep learning

    Li, M., Kudugunta, S., and Zettlemoyer, L. (mis)fitting scaling laws: A survey of scaling law fitting techniques in deep learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=xI71dsS3o4

    2025

  23. [23]

    Deeper insights into graph convolutional networks for semi-supervised learning

    Li, Q., Han, Z., and Wu, X.-M. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  24. [24]

    Lorenz, E. N. The predictability of a flow which possesses many scales of motion. Tellus, 21 0 (3): 0 289--307, 1969

    1969

  25. [25]

    and Gros, C

    Neumann, O. and Gros, C. Scaling laws for a multi-agent reinforcement learning model. arXiv preprint arXiv:2210.00849, 2022

    Pith/arXiv arXiv 2022

  26. [26]

    K., and Grover, A

    Nguyen, T., Brandstetter, J., Kapoor, A., Gupta, J. K., and Grover, A. Scaling transformer neural networks for skillful and reliable medium-range weather forecasting. Science Advances, 10 0 (8): 0 eadi2336, 2024

    2024

  27. [27]

    NVIDIA H100 Tensor Core GPU datasheet, 2022

    NVIDIA Corporation . NVIDIA H100 Tensor Core GPU datasheet, 2022. URL https://www.nvidia.com/en-us/data-center/h100/. Accessed: 2024-05-20

    2022

  28. [28]

    Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators

    Pathak, J., Subramanian, S., Harrington, P., Raja, S., Chattopadhyay, A., Mardani, M., Kurth, T., Hall, D., Li, Z., Azizzadenesheli, K., et al. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202.11214, 2022

    Pith/arXiv arXiv 2022

  29. [29]

    R., El-Kadi, A., Masters, D., Ewalds, T., Stott, J., Mohamed, S., Battaglia, P., et al

    Price, I., Sanchez-Gonzalez, A., Alet, F., Andersson, T. R., El-Kadi, A., Masters, D., Ewalds, T., Stott, J., Mohamed, S., Battaglia, P., et al. Gencast: Diffusion-based ensemble forecasting for medium-range weather. arXiv preprint arXiv:2312.15796, 2023

    Pith/arXiv arXiv 2023

  30. [30]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  31. [31]

    Weatherbench 2: A benchmark for the next generation of data-driven global weather models

    Rasp, S., Hoyer, S., Merose, A., Langmore, I., Battaglia, P., Russell, T., Sanchez-Gonzalez, A., Yang, V., Carver, R., Agrawal, S., et al. Weatherbench 2: A benchmark for the next generation of data-driven global weather models. Journal of Advances in Modeling Earth Systems, 16 0 (6): 0 e2023MS004019, 2024

    2024

  32. [32]

    Scaling laws literature review, 2023

    Villalobos, P. Scaling laws literature review, 2023. URL https://epoch.ai/blog/scaling-laws-literature-review. Epoch AI Report

    2023

  33. [33]

    Transformers learn in-context by gradient descent

    Von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pp.\ 35151--35174. PMLR, 2023

    2023

  34. [34]

    Simplifying graph convolutional networks

    Wu, F., Souza, A., Zhang, T., Fifty, C., Yu, T., and Weinberger, K. Simplifying graph convolutional networks. In International conference on machine learning, pp.\ 6861--6871. Pmlr, 2019

    2019

  35. [35]

    Scaling vision transformers

    Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers. arXiv preprint arXiv:2202.05924, 2022

    Pith/arXiv arXiv 2022

  36. [36]

    Q., Magnusson, L., Buizza, R., Lin, S.-J., Chen, J.-H., and Emanuel, K

    Zhang, F., Sun, Y. Q., Magnusson, L., Buizza, R., Lin, S.-J., Chen, J.-H., and Emanuel, K. What is the predictability limit of midlatitude weather? Journal of the Atmospheric Sciences, 76 0 (4): 0 1077--1091, 2019

    2019