REVIEW 3 major objections 5 minor 13 references
Stop using root-mean-square error as a precipitation target!
T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read This paper argues that the standard training loss for precipitation models, root-mean-square error, should be replaced by the Tweedie deviance, a likelihood-based loss that matches rainfall's zero-inflated, heavy-tailed distribution, and sh
desk verdict A genuinely useful proposal to replace MSE with Tweedie deviance for precipitation ML, let down by nowcasting claims with no error bars on a single test month. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Tweedie deviance: a likelihood-based loss from the exponential–dispersion family with variance function V(µ)=µ^p. For 1<p<2 it is the compound Poisson–Gamma distribution, and its unit deviance has a closed form. The power p is fit once from the variance–mean power law; because p is continuous, the same loss interpolates from Poisson-like (p→1) to Gamma-like (p→2) behavior. The paper uses this deviance as the training loss in place of MSE, with a softplus output to keep predictions non-negative and dispersion set to 1.
What would settle it
Estimate p separately for stratified storm regimes (monsoon versus frontal versus orographic) at one location; if the differences exceed the ordinary least-squares confidence intervals, a single p is misspecified and the deviance would need a mixture or spatially varying p. A second check: retrain the same models with p obtained from profile likelihood instead of block-OLS estimation and compare extreme recall and wet-pixel MAE.
Extended reading notes
Core claim
At the center is the unit Tweedie deviance, a closed-form likelihood-based loss from the exponential–dispersion family whose variance function is V(µ)=µ^p. For 1<p<2 it corresponds to a compound Poisson–Gamma distribution, with a point mass at zero and continuous positive amounts, which matches observed precipitation. The paper derives that RMSE is the p→0 Gaussian limit of this same family, so using RMSE imposes a Gaussian likelihood on a non-Gaussian target. The author estimates p from log Var(Y)=c+p log E[Y] over non-overlapping blocks and finds p in [1.18, ~2] across gauge, reanalysis, and satellite data, increasing with accumulation period toward the Gamma limit. Training identical down
Load-bearing premise
The paper assumes that a single spatially averaged Tweedie power p, estimated by least-squares fitting of the variance–mean relationship, correctly describes the rainfall distribution for every location and case; if the true p varies by storm type, terrain, or intensity, or if that fit is biased, the Tweedie deviance is misspecified as a likelihood and the statistical justification for the observed gains weakens.
Editorial extensions
If this is right
- Downscaling models trained with Tweedie deviance match RMSE-trained models on overall distributional fit, wet-pixel MAE, and dry hit rate, while recalling about 60% versus 50% of 99th-percentile extremes; operational downscaling can recover extreme events without sacrificing average skill.
- Nowcasting models trained with Tweedie deviance improve wet-pixel MAE and dry-pixel hit rate at every lead time, and the improvements grow with autoregressive rollout—about 2% MAE at t+1 growing to about 16% at t+4—so the loss choice compounds over the forecast horizon.
- The estimated Tweedie power p rises from near Poisson at 5-minute accumulation to near Gamma for monthly totals, so the same continuous loss family adapts to any accumulation period once p is estimated from the training data.
- Because p is not known a priori and differs between locations and datasets, users should estimate p on the target data at the target accumulation period rather than fixing it.
Reading between the lines
- A natural extension the author leaves implicit: because the Tweedie deviance is a proper scoring rule for the conditional mean, it could replace MSE in other zero-inflated, heavy-tailed geophysical targets such as wind gusts, cloud water, or air-quality concentrations, with p estimated per variable.
- A testable refinement: replacing the single global p with a spatially or seasonally indexed p, or parameterizing p as part of the network, could improve conditional tails further; the paper's own single-p design leaves this open.
- The unexplained 'weekly dip' in p may reflect real meteorological structure rather than sampling noise; if it persists, p itself could serve as a diagnostic of rainfall regime, not just a tuning parameter.
- The same likelihood framing extends naturally to probabilistic outputs: training a distribution head with Tweedie deviance would give a calibrated predictive distribution and explicit uncertainty, which a point-loss RMSE baseline cannot provide.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that RMSE is a misspecified training objective for precipitation and proposes the Tweedie deviance as a differentiable, likelihood-based replacement. Section 2 derives the unit deviance for the exponential dispersion family and recovers the Gaussian, Poisson, and Gamma limits. Section 4.1 estimates the Tweedie power p from the variance–mean power law for a UK rain gauge, ERA5, and GPM-IMERG data, showing p increases with accumulation length. Section 4.2 compares RMSE and Tweedie training of a conditional diffusion downscaler over Beijing; Section 4.3 compares RMSE and Tweedie training of a ConvLSTM nowcaster over Kolkata. The paper claims improved extreme recall in downscaling and improved wet-pixel MAE and dry hit rates in nowcasting, with gains that compound with lead time.
Significance. If the empirical gains are robust, the paper provides a practical and well-motivated alternative to RMSE, with a clean derivation and a simple recipe for estimating p. The use of identical architectures for both loss comparisons is a strength, as is the multi-dataset demonstration that p is far from zero and varies with aggregation. The paper is honest about the non-significance of some downscaling differences. The principal concern is that the central nowcasting claim rests on a one-month evaluation without uncertainty quantification, and the 'statistically justified' framing is not tested against misspecification of p.
major comments (3)
- [§4.3, Fig. 6] The headline claim of ~2%→16% MAE improvement and ~2%→13% dry-hit-rate improvement rests entirely on one month (August 2018) with no error bars, confidence intervals, or significance tests, even though App. A.2 defines a three-month test period (Jun–Aug 2018). Given stochastic training and strong space–time autocorrelation in nowcast errors, the t+4 differences could be sampling noise. This is the paper's strongest claim and needs a full-test-period evaluation with block-bootstrap CIs and paired significance tests (or an explicit justification for the August-only choice).
- [§4.2, Fig. 4; Abstract] The abstract concludes that Tweedie loss 'improves wet-pixel MAE and extreme recall' in downscaling, and the introduction repeats this. Fig. 4b shows a non-significant MAE difference (0.415 vs 0.410) and the text states this is not significant; only extreme recall (0.504 vs 0.602, Fig. 4d) reaches significance, and only for a single month and a 99th-percentile threshold. The claim should be qualified and, if possible, the analysis extended to the full test year specified in App. A.3.
- [§4.1, App. A.4, §5] The Tweedie power p is estimated once from the marginal variance–mean power law (blockwise OLS, averaged over pixels) and then fixed for training. The paper itself defers p-sensitivity to future work. If p is biased or varies with intensity, storm type, or orography, the training objective is not the Tweedie likelihood the authors invoke as justification for the skill gains. Please add sensitivity tests (e.g., p ± 95% CI or profile-likelihood estimate) and report the exact p used for each experiment; otherwise the 'statistically justified' framing is not supported.
minor comments (5)
- [§2.1, Eq. (11)] The Taylor expansion for θ is not correct as written: for p=ε, θ = μ^{1-ε}/(1-ε) = μ[1 + ε(1 - ln μ)] + O(ε²), not μ[1 - (ε/2) ln μ] + O(ε²). The leading term is unaffected, but the displayed expression should be fixed.
- [§4.3 and Fig. 6b] The '~16%' improvement at t+4 is ambiguous: (0.669−0.578)/0.669 ≈ 14%, while (0.669−0.578)/0.578 ≈ 16%. The standard relative-to-RMSE convention gives ~14%; please align the abstract, main text, and figure caption on one definition and verify the numbers.
- [App. A.4] The actual values of p used for the Beijing hourly downscaling and Kolkata half-hourly nowcasting experiments are not reported. Since the method's reproducibility depends on this, please list the estimated p and its confidence interval for each training setup.
- [§4.2] The text says differences are 'bootstrap tested' but no details of the bootstrap procedure (resampling unit, number of replications, CI construction) are given. Please describe the procedure or cite a standard method.
- [§4.1, Fig. 2] The systematic 'weekly dip' in p is left unexplained. If it cannot be analyzed, at least label it as a possible finite-sample artifact and remove it from the bullet-point conclusions, or provide a targeted test.
Circularity Check
No significant circularity: the Tweedie loss is derived from first-principles distribution theory and the reported skill gains are measured on held-out data, not by construction.
full rationale
The paper's central derivation chain is self-contained. The Tweedie deviance is derived from the exponential dispersion family via the variance function V(μ)=μ^p (Eqs. 1–10), so the loss formula itself does not assume any empirical precipitation result. The index p is estimated from training data using the variance–mean power law (App. A.4) and then fixed for validation and testing, but the loss is used only as a training objective. The evaluation metrics that support the headline claims—wet-pixel MAE, dry hit rate, and extreme recall—are computed on held-out test periods (August 2018 for both downscaling and nowcasting) and are not algebraically equal to the Tweedie deviance or to the p-estimation procedure. Optimizing Tweedie deviance does not by construction improve MAE, hit rate, or recall; those are empirical outcomes. The extreme-recall improvement is bootstrap-tested, while the nowcasting results lack confidence intervals, but that is a statistical-evidence concern, not circularity. The only self-citation (Hunt and Turner, 2024) appears as an example of MSE being a default loss in AI weather models, and it is not load-bearing for any derived claim. No uniqueness theorem, no ansatz smuggled via citation, and no renaming of a known result as a new derivation are present. Therefore the paper does not exhibit significant circularity.
Assumptions & free parameters
free parameters (4)
- Tweedie power p =
1.18 (Reading 5-min) to ~1.85 (Beijing daily); exact values for downscaling and nowcasting tasks are estimated from trai
- Intercept c of the variance-mean power law =
Not reported
- Scaling factor s =
99th percentile of training target pixels
- Evaluation threshold tau =
0.1 mm
assumptions (5)
- standard math Tweedie family is an exponential dispersion family with variance function V(mu)=mu^p and unit deviance given by Eq. 10
- standard math The cumulant function identities in Eqs. 7-9 (E[Y]=kappa'(theta), theta'=1/V) hold
- domain assumption Precipitation at the target accumulation follows a Tweedie distribution with a single p, or at least has a variance-mean power law with that p
- domain assumption The variance-mean power law holds on the block-averaged statistics, and OLS on log-transformed values gives an unbiased estimate of p
- ad hoc to paper Evaluation on a single test month (August 2018) is representative of the models' skill
Cite this review
Pith. "Pith review of Stop using root-mean-square error as a precipitation target!." pith.science (2026). https://pith.science/paper/7CQ5WH7J
@misc{pith2026250908369,
author = {Pith},
title = {Pith review of: Stop using root-mean-square error as a precipitation target!},
year = {2026},
howpublished = {\url{https://pith.science/paper/7CQ5WH7J}},
note = {Machine review of arXiv:2509.08369}
}
abstract
Root-mean-square error (RMSE) remains the default training loss for data-driven precipitation models, despite precipitation being semi-continuous, zero-inflated, strictly non-negative, and heavy-tailed. This Gaussian-implied objective misspecifies the data-generating process because it tolerates negative predictions, underpenalises rare heavy events, and ignores the mass at zero. We propose replacing RMSE with the Tweedie deviance, a likelihood-based and differentiable loss from the exponential--dispersion family with variance function $V(\mu)=\mu^p$. For $1<p<2$ it yields a compound Poisson--Gamma distribution with a point mass at zero and a continuous density for $y>0$, matching observed precipitation characteristics. We (i) estimate $p$ from the variance--mean power law and show that precipitation across temporal aggregations is far from Gaussian, with the Tweedie power $p$ increasing with accumulation length towards a Gamma limit; and (ii) demonstrate consistent skill gains when training deep data-driven models with Tweedie deviance in place of RMSE. In diffusion-model downscaling over Beijing, Tweedie loss improves wet-pixel MAE and extreme recall ($\sim0.60$ vs $0.50$ at the 99th percentile). In ConvLSTM nowcasting over Kolkata, Tweedie loss yields improved wet-pixel MAE and dry-pixel hit rates, with improvements that compound autoregressively with lead time (for MAE, $\sim2%$ at $t{+}1$ growing to $\sim16%$ at $t{+}4$). Because the Tweedie deviance is continuous in $p$, it adapts smoothly across scales, offering a statistically justified, practical replacement for RMSE in precipitation-based learning tasks.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
AghaKouchak, A., and N. Nasrollahi, 2010: Semi- parametric and parametric inference of extreme value models for rainfall data.Water resources manage- ment,24 (6), 1229–1249. Agrawal, S., L. Barrington, C. Bromberg, J. Burge, C. Gazen, and J. Hickey, 2019: Machine learning for precipitation nowcasting from radar images.arXiv preprint arXiv:1912.12132. Ambu...
arXiv 2010
-
[2]
A comparison of distribu- tions of amounts.Water resources research,18 (5), 1461–1468. Xiang, Y., J. Ma, and X. Wu, 2020: A precipitation nowcasting mechanism for real-world data based on machine learning.Mathematical Problems in Engi- neering,2020 (1), 8408
work page 2020
-
[17]
Cao, Y., L. Chen, J. Wu, and J. Feng, 2025: Enhancing nowcasting with multi-resolution inputs using deep learning: Exploring model decision mechanisms.Geo- physical Research Letters,52 (4), e2024GL113
work page 2025
-
[35]
Suhaila, J., 2023: Tweedie models for Malaysia rain- fall simulations with seasonal variabilities.Journal of Water and Climate Change,14 (10), 3648–3670. Svensson, C., J. Hannaford, and I. Prosdocimi, 2017: Statistical distributions for monthly aggregations of precipitation and streamflow in drought indicator ap- plications.Water Resources Research,53 (2), 999–
work page 2023
-
[308]
Wernli, H., M. Paulat, M. Hagen, and C. Frei, 2008: SAL—A novel quality measure for the verification of quantitative precipitation forecasts.Monthly Weather Review,136 (11), 4470–4487. Wilks, D. S., 1989: Rainfall intensity, the weibull distri- bution, and estimation of daily surface runoff.Jour- nal of Applied Meteorology and Climatology,28 (1), 52–58. W...
work page 2008
-
[529]
Kochkov, D., and Coauthors, 2024: Neural general circulation models for weather and climate.Nature, 632 (8027), 1060–1066. Kostinski, A. B., and A. R. Jameson, 1997: Fluctuation properties of precipitation. Part I: On deviations of single-size drop counts from the Poisson distribution. Journal of the atmospheric sciences,54 (17), 2174–
work page 2024
-
[647]
Ebert, E. E., L. J. Wilson, B. G. Brown, P. Nurmi, H. E. Brooks, J. Bally, and M. Jaeneke, 2004: Ver- ification of nowcasts from the WWRP Sydney 2000 forecast demonstration project.Weather and forecast- ing,19 (1), 73–96. Gupta, V. K., and E. Waymire, 1998: Some mathemat- ical aspects of rainfall, landforms, and floods.Ad- vanced series in statistical sci...
work page 2004
-
[699]
Chiang, C.-H., and Coauthors, 2024: Climate down- scaling: a deep-learning based super-resolution model of precipitation data with attention block and skip connections.arXiv preprint arXiv:2403.17847. Cho, H.-K., K. P. Bowman, and G. R. North, 2004: A comparison of gamma and lognormal distributions for characterizing satellite rain rates from the tropical...
work page Pith review arXiv 2024
Show all 13 references
-
[931]
Yan, C.-W., S. Q. Foo, V. H. Trinh, D.-Y. Yeung, K.- H. Wong, and W.-K. Wong, 2024: Fourier amplitude and correlation loss: Beyond using l2 loss for skillful precipitation nowcasting.Advances in Neural Infor- mation Processing Systems,37, 100 007–100
2024
-
[1018]
Thom, H. C. S., 1958: A note on the gamma distribu- tion.Monthly weather review,86 (4), 117–122. Vandal, T., E. Kodra, S. Ganguly, A. Michaelis, R. Ne- mani, and A. R. Ganguly, 2017: DeepSD: Generating high resolution climate change projections through single image super-resol...
1958
-
[2186]
Kumar, B., K
Kozu, T., and Coauthors, 2001: Development of pre- cipitation radar onboard the Tropical Rainfall Mea- suring Mission (TRMM) satellite.IEEE transactions on geoscience and remote sensing,39, 102–116, doi: 10.1109/36.898669, URL http://dx.doi.org/10.1109/ 36.898669. Kumar, B., K...
2001 doi
-
[2281]
Martinez-Villalobos, C., and J. D. Neelin, 2019: Why do precipitation intensities tend to follow gamma dis- tributions?Journal of the Atmospheric Sciences, 76 (11), 3611–3631. Mittermaier, M. P., 2021: A “meta” analysis of the frac- tions skill score: The limiting case and imp...
2019
-
[3278]
Murata, A., S.-i. I. Watanabe, H. Sasaki, H. Kawase, and M. Nosaka, 2020: Assessing goodness of fit to a gamma distribution and estimating future projec- tion on daily precipitation frequency using regional climate model simulations over Japan with and with- out the influence ...
2020 arXiv
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.