REVIEW 3 major objections 5 minor 34 references
A full-attention video transformer with 64-bin classification and log-frequency weighting claims first place in the 2025 rainfall nowcasting challenge.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 22:15 UTC pith:ZC547V4M
load-bearing objection A solid competition-winning system whose headline CRPS is a test-selected extremum, so the SOTA claim is provisional but the simple recipe is a useful baseline. the 3 major comments →
A Space-Time Transformer for Precipitation Nowcasting
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that a categorical reformulation of precipitation estimation—binning the continuous target into 64 intervals and training with a cross-entropy loss weighted by the log of each bin's relative frequency—solves the long-tail problem that otherwise makes models collapse to 'no rain' predictions. On top of that, the paper claims that full space-time self-attention, where every patch token at every time step attends to all other tokens, outperforms factorized space-time attention on this task. The ablation reports bin-weighted CRPS falling from 6.91 (unweighted) to 2.64 (weighted), and full attention besting both space-then-time and time-then-space variants. Together these cho
What carries the argument
Two mechanisms carry the argument. First, full 3D self-attention: the input is four 32×32 satellite frames divided into non-overlapping patches, and every patch token attends to every other token across time and space, with a prepended class token whose final representation drives prediction. Second, the class-weighted cross-entropy loss with weights w_i = -log(|D_i|/|D_total|), where D_i is the training set of samples falling into bin i; this log-frequency prior offsets the heavy skew toward low and zero rainfall. The paper also relies on the choice of 64 bins as the resolution at which CRPS is minimized before class sparsity degrades results.
Load-bearing premise
The loss weights w_i = -log(|D_i|/|D_total|) presuppose that every one of the n bins contains at least one training sample; the paper's Figure 2 shows large gaps in target coverage, so for large n some bins are likely empty and the formula as written is undefined.
What would settle it
Count the number of training targets falling into each of the 64 bins used for the main result, along with the 128 and 256 bins from Table 2. If any bin is empty, the stated loss cannot be evaluated, and re-running the ablation with a smoothing rule should produce different CRPS numbers; the difference would show that the published numbers depend on an unspecified convention.
If this is right
- Classification with log-frequency reweighting is an effective replacement for direct regression on long-tailed precipitation targets, and can be applied to other single-value forecasting tasks with skewed distributions.
- Full space-time attention is a viable architectural choice for low-token weather inputs, contradicting the usual preference for factorized attention in video models.
- Bin count is a critical hyperparameter: the paper's ablation shows 64 bins is optimal on this challenge, with coarse or extremely fine binning hurting CRPS.
- Because the method is end-to-end and uses a generic video transformer, it can be adapted to other satellite-based earth-observation tasks without bespoke physics-inspired modules.
Where Pith is reading between the lines
- The 64-bin optimum likely reflects a trade-off between quantisation error and class sparsity; on a larger training set, the optimum could shift to finer bins, so the reported bin-count sweep should be re-run as data grows.
- A smoothed or clipped version of the log-frequency weights would make the method robust in transfer scenarios where new regions introduce rainfall values outside the training distribution, a situation the current formulation does not address.
- Extending the class token to a per-pixel classification head could turn the single-number cumulative forecast into a full rainfall map, reusing the same loss and attention machinery.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SaTformer, a video transformer with full space-time self-attention, for the NeurIPS Weather4Cast 2025 'Cumulative Rainfall' challenge. Precipitation regression is reformulated as classification into n discrete bins, with a log-frequency class-weighted cross-entropy loss to address the long-tailed target distribution. The model is trained on satellite radiance sequences and predicts cumulative rainfall from a class token. The authors report a challenge-set CRPS of 3.135, which they state ranked first in the competition, and ablations on the validation set comparing loss weighting and attention variants (space-then-time, time-then-space, and full space-time). Code and model weights are released.
Significance. If the reported result is valid, the paper provides a clean and reproducible demonstration that a generic video transformer with full attention can achieve strong performance on a low-token precipitation nowcasting task. The categorical reformulation and class-weighted loss are simple, effective ideas that could transfer to other long-tailed Earth-system regression problems. The official first-place ranking is an externally meaningful signal, and the release of code and weights strengthens the contribution. However, the empirical evaluation has a structural selection problem that affects the headline CRPS claim, and the loss definition has a reproducibility gap for empty bins. These issues must be addressed before the scientific claims can be accepted.
major comments (3)
- [§5.4–5.5, Table 2] The headline CRPS of 3.135 is selected on the challenge test set. Section 5.5 states that 'we conclude evaluations to determine optimal bin-size using the W4CC challenge set,' and Table 2 reports CRPS for bin counts 4 through 512 on that same set. The 64-bin result is therefore the minimum of a sweep over the test set, not an independent estimate. No validation-set sweep, multiple-seed statistics, or multiple-testing correction is provided. The official competition rank is not in question, but the scientific claim that SaTformer achieves state-of-the-art skill at CRPS=3.135 is not supported by this selection procedure. Please report a holdout validation-based bin-count selection, or explicitly reframe the test-set result as a selected extremum and provide an unbiased estimate of skill.
- [Eq. (8), Fig. 2] The class-frequency weight w_i = -log(|D_i|/|D_total|) is undefined for any empty bin. Figure 2 shows large gaps in target coverage at the extremes, and Table 2 sweeps up to 512 bins, where empty or near-empty bins are highly plausible. The paper does not specify smoothing, clipping, or a rule for excluding bins. Without such a rule, the loss in Eq. (8), the ablations in Table 1, and the bin-count comparison in Table 2 are not reproducible as written. Please state exactly how empty or low-count bins are handled in the implementation.
- [Table 1] The ablation results are reported from single runs with no error bars or significance tests. The differences among attention variants (S→T, T→S, S+T) are not large (e.g., BW-Top-3 0.250 vs. 0.272), and the paper's conclusion that full space-time attention 'yields superior model performance' rests on these isolated numbers. Please provide standard errors over multiple seeds, or explicitly caveat the comparison as preliminary.
minor comments (5)
- [Table 1] The bottom block labels both 'space followed by time' and 'time then space' as '(S→T)'; the second should presumably be '(T→S)'.
- [§5.4] The metrics BW-Top-3 and BW-CRPS are not explicitly defined. Please specify how bin weighting is applied to these validation metrics so the ablation numbers are interpretable.
- [Eq. (9)] The formula for y_reg uses T', H', W' but the text describes averaging over a 32×32 pixel area and 16 slots; the connection between the notation and the described OPERA dimensions could be made clearer.
- [Throughout] There are minor typos, e.g., 'satallite' in the Introduction and 'diving' in the quoted challenge description. A light proofread is recommended.
- [§6.2] The limitations paragraph is helpful, but it could also acknowledge the test-set selection issue in §5.5 and describe how future work should avoid it.
Circularity Check
Headline CRPS is the minimum of a test-set bin-count sweep, so the central 'best model' result is post-selection rather than an independent prediction.
specific steps
-
fitted input called prediction
[Section 5.5 (Table 2) and Section 5.4 (Results)]
"Satisfied with our choice of loss function and encoder design, we conclude evaluations to determine optimal bin-size using the W4CC challenge set. We note that while choosing to predict more output classes (i.e., using smaller values of δ) generally yields better CRPS scores in theory, in practice models tend to produce degenerate solutions as class representation becomes increasingly sparse. A reasonable middle ground of 64 classes achieves the best empirical results (Table 2)."
The number of bins is a hyperparameter selected by minimizing CRPS on the same W4CC challenge set that is then used for the headline evaluation. Table 2 reports CRPS on that set for 4,8,...,512 bins, and 64 bins is the minimum (3.135). Section 5.4 then presents 'Our best performing model achieves a CRPS score of 3.135 on the challenge set' as the paper's central result. Thus the reported 'best' score is the minimum of a test-set sweep: the evaluation labels were used to choose the configuration, so the claim is a post-selection extremum, not an out-of-sample prediction. The official competition rank may stand, but the scientific claim of best performance is selected using the test data rather than independently estimated.
full rationale
The architecture itself is not self-definitional: full space-time attention and the class-weighted loss are trained on the training split and compared on the validation set (Table 1), so those ablations have independent content. There are no load-bearing self-citations, imported uniqueness theorems, or ansatz-smuggling citations. The one substantial circularity-adjacent step is the bin-count selection on the challenge set: the paper explicitly says the optimal bin-size was determined using the W4CC challenge set, then reports the 64-bin CRPS of 3.135 on that same set as the best result. This fits the fitted-input-called-prediction pattern: the headline score is the minimum of the Table 2 sweep, so it is statistically forced by the selection procedure. Separately, Eq. 8 defines w_i = -log(|D_i|/|D_total|), which is undefined for any empty bin; Figure 2 indicates large gaps in target coverage at high bin counts, and no smoothing/clipping rule is given. That is a reproducibility gap, not a circularity. Overall, the central numerical claim is partially circular via test-set selection, while the methodological claims remain independent.
Axiom & Free-Parameter Ledger
free parameters (4)
- number of output bins n / bin width δ =
n=64 (best of {4,8,16,32,64,128,256,512} on challenge test set)
- patch size P =
4
- transformer hyperparameters (L=12, d=512, heads=8, head_dim=64, LR=1e-5, batch=128, epochs=200)
- class-weight smoothing/clipping for empty bins
axioms (4)
- domain assumption HRIT 11-channel satellite radiances over 1 hour contain sufficient information to predict 4-hour cumulative OPERA rainfall at 32x32 resolution.
- domain assumption CRPS computed on binned cumulative distributions faithfully measures forecast skill, including for extreme precipitation events.
- domain assumption A randomly initialized class token can aggregate spatiotemporal information for the scalar target.
- standard math Standard transformer backpropagation with Adam optimization is an appropriate learning procedure for this data size.
read the original abstract
Until recently, numerical weather prediction (NWP) models have stood rivalless in operational forecasting despite a few limitations. Namely, physically-based models are computationally demanding and struggle at short lead times, reducing their utility for nowcasting. Motivated by these shortcomings, recent work proposes AI-weather prediction (AI-WP) alternatives that emulate analysis data with neural networks. While these data-driven approaches have achieved high skill for medium-range forecasting-applications of AI-WP to precipitation and to nowcasting are less explored. To these ends, this paper discusses \textit{SaTformer}: a video transformer adapted for precipitation nowcasting. To ameliorate some problems related to what is essentially a fat-tailed regression task, we find it prudent to formulate nowcasting as a classification problem and employ a frequency-weighted loss. This straightforward approach scored first on the NeurIPS Weather4Cast 2025 ``Cumulative Rainfall'' challenge. Code and model weights are available: \texttt{\href{github.com/leharris3/w4c-25}{github.com/leharris3/satformer}}.
Figures
Reference graph
Works this paper leans on
-
[1]
Daniel Abdi et al.HRRRCast: a data-driven emulator for regional weather forecasting at convection allowing scales. 2025. arXiv: 2507 . 05658 [physics.ao-ph].URL: https : //arxiv.org/abs/2507.05658
Pith/arXiv arXiv 2025
-
[2]
Anurag Arnab et al.ViViT: A Video Vision Transformer. 2021. arXiv: 2103.15691 [cs.CV]. URL:https://arxiv.org/abs/2103.15691
Pith/arXiv arXiv 2021
-
[3]
Precipitation nowcasting with generative diffusion models
Andrea Asperti et al. “Precipitation nowcasting with generative diffusion models”. In:Applied Intelligence55.3 (Dec. 2024).ISSN: 1573-7497.DOI: 10.1007/s10489- 024- 06048- y . URL:http://dx.doi.org/10.1007/s10489-024-06048-y
doi:10.1007/s10489- 2024
-
[4]
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.Layer Normalization. 2016. arXiv: 1607.06450 [stat.ML].URL:https://arxiv.org/abs/1607.06450
Pith/arXiv arXiv 2016
-
[5]
Yury Belousov, Sergey Polezhaev, and Brian Pulfer.Solving the Weather4cast Challenge via Visual Transformers for 3D Images. 2022. arXiv: 2212.02456 [cs.CV] .URL: https: //arxiv.org/abs/2212.02456
Pith/arXiv arXiv 2022
-
[6]
arXiv: 2102.05095 [cs.CV].URL: https://arxiv.org/ abs/2102.05095
Gedas Bertasius, Heng Wang, and Lorenzo Torresani.Is Space-Time Attention All You Need for Video Understanding?2021. arXiv: 2102.05095 [cs.CV].URL: https://arxiv.org/ abs/2102.05095
Pith/arXiv arXiv 2021
-
[8]
Özgün Çiçek et al.3D U-Net: Learning Dense V olumetric Segmentation from Sparse Annota- tion. 2016. arXiv:1606.06650 [cs.CV].URL:https://arxiv.org/abs/1606.06650
Pith/arXiv arXiv 2016
-
[9]
Hybrid physics-AI outperforms numerical weather prediction for extreme precipitation nowcasting
Puja Das et al. “Hybrid physics-AI outperforms numerical weather prediction for extreme precipitation nowcasting”. In:npj Climate and Atmospheric Science7.1 (Nov. 2024).ISSN: 2397-3722.DOI: 10.1038/s41612-024-00834-8 .URL: http://dx.doi.org/10.1038/ s41612-024-00834-8
-
[10]
Atharva Deshpande et al.A conditional Generative Adversarial network model for the Weather4Cast 2024 Challenge. 2024. arXiv: 2412.00451 [cs.CV].URL: https://arxiv. org/abs/2412.00451
Pith/arXiv arXiv 2024
-
[11]
Alexey Dosovitskiy et al.An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. 2021. arXiv: 2010 . 11929 [cs.CV].URL: https : / / arxiv . org / abs / 2010 . 11929
2021
-
[12]
WoFSCast: A Machine Learning Model for Pre- dicting Thunderstorms at Watch-to-Warning Scales
Montgomery L. Flora and Corey Potvin. “WoFSCast: A Machine Learning Model for Pre- dicting Thunderstorms at Watch-to-Warning Scales”. In:Geophysical Research Letters52.10 (May 2025).ISSN: 1944-8007.DOI: 10.1029/2024gl112383.URL: http://dx.doi.org/ 10.1029/2024GL112383
-
[13]
Zhihan Gao et al.Earthformer: Exploring Space-Time Transformers for Earth System F orecast- ing. 2023. arXiv:2207.05833 [cs.LG].URL:https://arxiv.org/abs/2207.05833
Pith/arXiv arXiv 2023
-
[14]
arXiv: 2207.08815 [cs.LG].URL: https: //arxiv.org/abs/2207.08815
Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux.Why do tree-based models still outperform deep learning on tabular data?2022. arXiv: 2207.08815 [cs.LG].URL: https: //arxiv.org/abs/2207.08815
Pith/arXiv arXiv 2022
-
[15]
Multi-task Challenges for Rain Movie Prediction on the Road to Hi-Res Foundation Models
Aleksandra Gruca et al. “Multi-task Challenges for Rain Movie Prediction on the Road to Hi-Res Foundation Models”. In:OpenReview(2024). NeurIPS 2024 Competition Track.URL: https://openreview.net/forum?id=AZ9WzDxoTf
2024
-
[16]
Weather4cast at NeurIPS 2022: Super-Resolution Rain Movie Pre- diction under Spatio-temporal Shifts
Aleksandra Gruca et al. “Weather4cast at NeurIPS 2022: Super-Resolution Rain Movie Pre- diction under Spatio-temporal Shifts”. In:Proceedings of the NeurIPS 2022 Competitions Track. Ed. by Marco Ciccone, Gustavo Stolovitzky, and Jacob Albrecht. V ol. 220. Proceedings of Machine Learning Research. PMLR, 28 Nov–09 Dec 2022, pp. 292–313.URL: https: //proceed...
2022
-
[17]
Vincent Le Guen and Nicolas Thome.Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction. 2020. arXiv: 2003.01460 [cs.CV].URL: https: //arxiv.org/abs/2003.01460. 9
Pith/arXiv arXiv 2020
-
[18]
A Deep Learning Model for Precipitation Nowcasting Using Multiple Optical Flow Algorithms
Ji-Hoon Ha and Hyesook Lee. “A Deep Learning Model for Precipitation Nowcasting Using Multiple Optical Flow Algorithms”. In:Weather and F orecasting39.1 (Jan. 2024), pp. 41–53. ISSN: 1520-0434.DOI: 10.1175/waf- d- 23- 0104.1 .URL: http://dx.doi.org/10. 1175/WAF-D-23-0104.1
-
[19]
Lu Han et al.Learning Robust Precipitation F orecaster by Temporal Frame Interpolation
-
[20]
Ali Hatamizadeh et al.Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images. 2022. arXiv: 2201.01266 [eess.IV].URL: https://arxiv.org/ abs/2201.01266
Pith/arXiv arXiv 2022
-
[21]
The effect of rainfall changes on economic production
Maximilian Kotz, Anders Levermann, and Leonie Wenz. “The effect of rainfall changes on economic production”. In:Nature601.7892 (Jan. 2022), pp. 223–227.ISSN: 1476-4687.DOI: 10.1038/s41586-021-04283-8 .URL: http://dx.doi.org/10.1038/s41586-021- 04283-8
-
[22]
Ajitabh Kumar.Skilful Precipitation Nowcasting Using NowcastNet. 2023. arXiv: 2311 . 17961 [physics.ao-ph].URL:https://arxiv.org/abs/2311.17961
Pith/arXiv arXiv 2023
-
[23]
Pak Hay Kwok and Qi Qi.A V ariational U-Net for Weather F orecasting. 2021. arXiv:2111. 03476 [cs.LG].URL:https://arxiv.org/abs/2111.03476
Pith/arXiv arXiv 2021
-
[24]
Learning skillful medium-range global weather forecasting
Remi Lam et al. “Learning skillful medium-range global weather forecasting”. In:Science 382.6677 (Dec. 2023), pp. 1416–1421.ISSN: 1095-9203.DOI: 10.1126/science.adi2336. URL:http://dx.doi.org/10.1126/science.adi2336
-
[25]
Xinzhe Li et al.Precipitation Prediction Using an Ensemble of Lightweight Learners. 2023. arXiv:2401.09424 [physics.ao-ph].URL:https://arxiv.org/abs/2401.09424
Pith/arXiv arXiv 2023
-
[26]
Yang Li et al.Super-resolution Probabilistic Rain Prediction from Satellite Data Using 3D U-Nets and EarthF ormers. 2022. arXiv:2212.02998 [cs.CV].URL: https://arxiv.org/ abs/2212.02998
Pith/arXiv arXiv 2022
-
[27]
Gavin D. Madakumbura et al. “Anthropogenic influence on extreme precipitation over global land areas seen in multiple observational datasets”. In:Nature Communications12.1 (July 2021).ISSN: 2041-1723.DOI: 10.1038/s41467- 021- 24262- x.URL: http://dx.doi. org/10.1038/s41467-021-24262-x
doi:10.1038/s41467- 2021
-
[28]
Jinyoung Park et al.RainUNet for Super-Resolution Rain Movie Prediction under Spatio- temporal Shifts. 2022. arXiv: 2212.04005 [cs.CV] .URL: https://arxiv.org/abs/ 2212.04005
Pith/arXiv arXiv 2022
-
[29]
Jaideep Pathak et al.F ourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive F ourier Neural Operators. 2022. arXiv: 2202.11214 [physics.ao-ph] . URL:https://arxiv.org/abs/2202.11214
Pith/arXiv arXiv 2022
-
[30]
Jiˇrí Pihrt et al.WeatherFusionNet: Predicting Precipitation from Satellite Data. 2022. arXiv: 2211.16824 [cs.CV].URL:https://arxiv.org/abs/2211.16824
Pith/arXiv arXiv 2022
-
[31]
Probabilistic weather forecasting with machine learning
Ilan Price et al. “Probabilistic weather forecasting with machine learning”. In:Nature637.8044 (Dec. 2024), pp. 84–90.ISSN: 1476-4687.DOI: 10.1038/s41586- 024- 08252- 9 .URL: http://dx.doi.org/10.1038/s41586-024-08252-9
doi:10.1038/s41586- 2024
-
[32]
GA-SmaAt-GNet: Generative adversarial small attention GNet for extreme precipitation nowcasting
Eloy Reulen, Jie Shi, and Siamak Mehrkanoon. “GA-SmaAt-GNet: Generative adversarial small attention GNet for extreme precipitation nowcasting”. In:Knowledge-Based Systems 305 (Dec. 2024), p. 112612.ISSN: 0950-7051.DOI: 10.1016/j.knosys.2024.112612 . URL:http://dx.doi.org/10.1016/j.knosys.2024.112612
arXiv 2024
-
[33]
Yuzhe Yang et al.Delving into Deep Imbalanced Regression. 2021. arXiv: 2102 . 09554 [cs.LG].URL:https://arxiv.org/abs/2102.09554
Pith/arXiv arXiv 2021
-
[34]
Skilful nowcasting of extreme precipitation with NowcastNet
Yuchen Zhang et al. “Skilful nowcasting of extreme precipitation with NowcastNet”. In:Nature 619.7970 (July 2023), pp. 526–532.ISSN: 1476-4687.DOI: 10.1038/s41586-023-06184-4 . URL:http://dx.doi.org/10.1038/s41586-023-06184-4. 10
-
[2023]
arXiv:2311.18341 [cs.LG].URL:https://arxiv.org/abs/2311.18341
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.