REVIEW 3 major objections 5 minor 26 references
Fine Flood Forecasts: Incorporating local data into global models through fine-tuning
T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Fine-tuning a global LSTM on local basin data raises flood-forecast skill, with the biggest gains where the global model struggles.
desk verdict Useful, honest workshop paper on fine-tuning global LSTM flood forecasts per basin; the central numbers likely hold but the paper must document the evaluation split before those numbers can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is transfer learning by parameter fine-tuning of a 260K-parameter LSTM that ingests 365 days of atmospheric and catchment variables and outputs next-day streamflow. Global pretraining on 6,375 basins learns shared rainfall-runoff dynamics; fine-tuning then updates the network weights (either all weights or only the output head) on a single basin's record, with per-basin hyperparameter selection by validation NSE. A comparison against single-basin models trained from scratch isolates the value of the pretrained initialization.
What would settle it
Evaluate the fine-tuned and pre-trained models on a fixed test period that is explicitly disjoint from every validation decision, including the 50-trial hyperparameter sweep for each basin; if the mean NSE and KGE gains shrink to zero or reverse on that split, the paper's central claim fails.
Extended reading notes
Core claim
On its own terms, the paper's claim is that a globally pre-trained LSTM retains transferable skill that per-basin fine-tuning can sharpen: across 159 randomly sampled basins and 8 pretrained model instances, mean Nash-Sutcliffe efficiency rises from 0.473 to 0.541 and mean Kling-Gupta efficiency from 0.520 to 0.599, gains of 14% and 15%. The distribution of gains is left-skewed: median gains are smaller (NSE +0.042, KGE +0.026), and improvement correlates negatively with pretrained skill, explaining 11.1% of the variance, so the weakest basins benefit most; every country in the sample has a net positive change. Fine-tuning is not guaranteed to help: 22% of base-model/basin pairs degrade, and 14% of basins still degrade on average. The same setup also shows that training an LSTM from scratch on a single basin (mean NSE 0.358) is worse than both the global model and the fine-tuned model, which the paper interprets as confirming that large-sample pretraining sets the right initialization.
Load-bearing premise
The comparison assumes the test-period scores in Table 1 are computed on data that was not used for the per-basin hyperparameter sweeps; the paper never describes the validation/test split, and its footnote mentions only a 'pre-defined training set'.
Editorial extensions
If this is right
- National hydromet agencies can adapt a global pretrained model to their own basins without sharing proprietary data or training from scratch, lowering the compute and data barrier to operational ML flood forecasting.
- The same fine-tuning recipe likely transfers to other deep-learning hydrology architectures, since the paper notes the principle is model-agnostic.
- Basins underserved by global models, those with low pretrained skill, stand to gain the most, so fine-tuning could help close the flood-forecasting gap in developing regions.
- Because fine-tuning needs only historical data rather than near-real-time assimilation, it offers a cheaper route to skill gains than data assimilation, which previously produced an 8% median NSE improvement on 539 US basins.
Reading between the lines
- A natural next test is whether fine-tuning on local data and near-real-time data assimilation are additive; the paper compares the two approaches but does not combine them.
- The negative correlation between pretrained skill and fine-tuning gain suggests fine-tuning may act partly as bias correction for regions where global forcing or regional hydrology is poorly represented; one could test this by conditioning gains on aridity, basin area, and data quality.
- For operational adoption, agencies will need a protocol for re-fine-tuning as new observations arrive, since the paper's per-basin hyperparameter sweeps are expensive; a cheaper fixed-schedule fine-tune would be a testable simplification.
- Because 14% of basins still regress on average, a practical deployment rule such as only fine-tuning when validation NSE improves over the pretrained model would make the method safer; the paper does not propose such a rule.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a methodology for fine-tuning a globally pre-trained LSTM streamflow model on data from individual basins, using the Caravan dataset. It compares the fine-tuned model against the pre-trained model and against single-basin-trained models on a random sample of 159 basins, reporting that fine-tuning increases mean NSE from 0.473 to 0.541 and mean KGE from 0.520 to 0.599. The paper also reports a negative correlation between pre-trained skill and fine-tuning improvement, and provides a step-by-step guide for forecasters. The stated goal is to enable national agencies to take ownership of global ML flood-forecast models by adapting them with local data.
Significance. If the results hold, this work is practically significant: it offers a low-compute path for local agencies to improve global ML flood forecasts, addresses data-ownership concerns, and provides open-source code and a pre-trained model. The finding that fine-tuning helps most in basins where the global model performs worst is also useful for prioritizing deployment. The paper's strengths are its clear practical framing, the open-source release, and the multi-seed (8 seeds) experimental design. However, the central empirical claim depends entirely on a correctly held-out evaluation period, which the manuscript does not describe; the reported results also contain internal inconsistencies that must be resolved before the quantitative claims can be accepted.
major comments (3)
- [Section 2.2 and Section 3.1, Table 1] The manuscript never states the temporal boundaries of the training, validation, and test periods. Section 2.2 says per-basin hyperparameters are selected by optimizing validation NSE, but it does not say whether the evaluation period used for the NSE/KGE scores in Table 1 is disjoint from the fine-tuning training and validation periods. Given the 365-day input lookback and the strong autocorrelation of daily streamflow, any overlap between fine-tuning training data and the evaluation window can inflate skill because the model would have seen the observed flows it is asked to predict. The Table 1 footnote mentions a 'pre-defined training set' only for the single-basin baseline, not for the fine-tuned models. The authors must provide the exact date ranges for the training/validation/test split and a statement that the evaluation period is entirely unseen during fine-tuning (including hyperparameter selection). This is the load-bearing condition for the paper's central claim.
- [Table 1 footnote and Appendix C, Table 3] There are internal inconsistencies in the reported results. The footnote to Table 1 states that the single-basin model was trained on only 147 of the 159 basins, while the pre-trained and fine-tuned models appear to be evaluated on all 159 basins, so the three-way comparison in Table 1 is not on a common set of basins. Additionally, the standard errors for the same quantities differ between Table 1 and Table 3: for the fine-tuned model, KGE mean is 0.599 ± 0.042 in Table 1 but 0.599 ± 0.026 in Table 3; NSE mean is 0.541 ± 0.017 vs 0.541 ± 0.014; median NSE is 0.625 vs 0.627; median KGE is 0.709 vs 0.711. These discrepancies need to be reconciled, or the text should explicitly state what each interval represents (e.g., across seeds or across basins).
- [Section 3.1] The claim that fine-tuning provides a "significant increase" is not supported by any inferential statistic. The paper reports standard errors but no formal significance test (e.g., paired t-test or Wilcoxon signed-rank test across basins, or a test across the 8 seed instances). Given the large number of basins, the improvement may well be real, but the current use of the word "significant" is not backed by a stated test. The authors should add a formal test, or explicitly state that the improvement is consistent across all 8 seeds and all 159 basins, and clarify the aggregation procedure used to compute the reported means and intervals.
minor comments (5)
- [Figure 2 (right panel)] The text states that "for every country in the dataset, there is a net positive improvement," but the figure appears to show only a subset of countries. Please clarify whether this statement refers to all countries in the 159-basin sample and label the figure accordingly.
- [Section 3.2] The fraction of variance explained by the relationship between pre-trained skill and fine-tuning improvement is 11.1%; the text "improvement from fine-tuning is particularly large in these" overstates the strength of the relationship. Suggest tempering the language or reporting an effect-size measure.
- [Section 2.2] The text says "recursive neural networks (RNNs)" but should read "recurrent neural networks."
- [Abstract and Introduction] The phrase "short-perm predictions" appears to be a typo for "short-term predictions."
- [Appendix B.2 and Table 3] The caption for Table 3 says "mean ± std/√n" but does not define n (whether n=159 basins or n=8 seeds). Also, the text says 1272 data points (159×8), but then says the results are averaged over the fine-tuning values for each basin; please clarify the exact averaging order.
Circularity Check
No circularity is demonstrated; the fine-tuning comparison is an empirical evaluation against an external pre-trained baseline, and the only concern is an undocumented validation/test split, which is a statistical-transparency issue rather than a circular reduction.
full rationale
This paper makes no first-principles derivation; its central claims are empirical comparisons of model variants. The pre-trained global LSTM is an external artifact trained on 6375 basins, and the fine-tuned models are obtained by gradient descent on per-basin data, so the reported NSE/KGE improvements are not equal to any input by construction. The hyperparameter sweep selects configurations by validation NSE, but this is standard model selection; unless the Table 1 scores are computed on the same validation period, no circularity follows. The text does not state the temporal split of training, validation, and test periods, and the only explicit split reference is the Table 1 footnote mentioning a 'pre-defined training set' for the single-basin baseline, leaving the evaluation period unidentified. If the same validation set were reused for the headline numbers, the gains would be optimistically biased, but that would be a statistical validity problem, not a demonstrated circular reduction. The self-citations (e.g., Nearing et al. 2022/2024 and Kratzert et al. 2024) supply background and benchmarks, but the fine-tuning comparison itself is run in this paper and does not depend on those citations for its validity. Therefore, no circularity is demonstrated, and the score is 0.
Assumptions & free parameters
free parameters (3)
- Pretrained LSTM hyperparameters (hidden size 256, batch size 256, 40 epochs, LR schedule 5e-5/5e-6) =
Hidden size 256, LR 5e-5 (epochs 1-30), 5e-6 (epochs 31-40), 40 epochs
- Per-basin fine-tuning hyperparameters (epochs, learning rate, loss, modules) =
Selected per basin via 50 TPE trials on validation NSE; values not reported
- Random sample of 159 basins =
159 basins sampled from 6375; seed not stated
assumptions (5)
- domain assumption LSTM internal states can represent watershed hydrological dynamics such as storages and reservoirs
- ad hoc to paper There exists a valid temporal split such that the validation period used for hyperparameter selection does not overlap the test period used to report performance
- domain assumption Caravan streamflow and atmospheric data are reliable for all 6375 basins, including the 159 sampled
- domain assumption NSE and KGE are sufficient metrics for judging flood forecast skill
- domain assumption The single-basin trained model is a fair baseline for 'no global pretraining'
Cite this review
Pith. "Pith review of Fine Flood Forecasts: Incorporating local data into global models through fine-tuning." pith.science (2026). https://pith.science/paper/CZK2IOJ4
@misc{pith2026250412559,
author = {Pith},
title = {Pith review of: Fine Flood Forecasts: Incorporating local data into global models through fine-tuning},
year = {2026},
howpublished = {\url{https://pith.science/paper/CZK2IOJ4}},
note = {Machine review of arXiv:2504.12559}
}
read the original abstract
Floods are the most common form of natural disaster and accurate flood forecasting is essential for early warning systems. Previous work has shown that machine learning (ML) models are a promising way to improve flood predictions when trained on large, geographically-diverse datasets. This requirement of global training can result in a loss of ownership for national forecasters who cannot easily adapt the models to improve performance in their region, preventing ML models from being operationally deployed. Furthermore, traditional hydrology research with physics-based models suggests that local data -- which in many cases is only accessible to local agencies -- is valuable for improving model performance. To address these concerns, we demonstrate a methodology of pre-training a model on a large, global dataset and then fine-tuning that model on data from individual basins. This results in performance increases, validating our hypothesis that there is extra information to be captured in local data. In particular, we show that performance increases are most significant in watersheds that underperform during global training. We provide a roadmap for national forecasters who wish to take ownership of global models using their own data, aiming to lower the barrier to operational deployment of ML-based hydrological forecast systems.
Figures
Reference graph
Works this paper leans on
-
[1]
Enhancement of river flooding due to global warming
Haireti Alifu, Yukiko Hirabayashi, Yukiko Imada, and Hideo Shiogama. Enhancement of river flooding due to global warming. Scientific Reports, 12 0 (1): 0 20687, Nov 2022. ISSN 2045-2322. doi:10.1038/s41598-022-25182-6. URL https://doi.org/10.1038/s41598-022-25182-6
-
[2]
James Bergstra, Daniel Yamins, and David D Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 115--123. PMLR, 2013
work page 2013
-
[3]
The human cost of natural disasters 2015: A global perspective
Centre for Research on the Epidemiology of Disasters. The human cost of natural disasters 2015: A global perspective. Technical report, 2015. URL https://reliefweb.int/report/world/human-cost-natural-disasters-2015-global-perspective
work page 2015
-
[4]
C. Färber, H. Plessow, S. Mischel, F. Kratzert, N. Addor, G. Shalev, and U. Looser. Grdc-caravan: extending caravan with data from the global runoff data centre. Earth System Science Data Discussions, pages 1--28, 2024. doi:10.5194/essd-2024-427
-
[5]
Martin Gauch, Frederik Kratzert, Oren Gilon, Hoshin Gupta, Juliane Mai, Grey Nearing, Bryan Tolson, Sepp Hochreiter, and Daniel Klotz. In defense of metrics: Metrics sufficiently encode typical human preferences regarding hydrological model performance. Water Resources Research, 59 0 (6): 0 e2022WR033918, 2023. doi:https://doi.org/10.1029/2022WR033918. UR...
-
[6]
Hoshin V. Gupta, Harald Kling, Koray K. Yilmaz, and Guillermo F. Martinez. Decomposition of the mean squared error and nse performance criteria: Implications for improving hydrological modelling. Journal of Hydrology, 377 0 (1): 0 80--91, 2009. ISSN 0022-1694. doi:https://doi.org/10.1016/j.jhydrol.2009.08.003. URL https://www.sciencedirect.com/science/art...
-
[7]
Stéphane Hallegatte. A cost effective solution to reduce disaster losses in developing countries: Hydro-meteorological services, early warning, and evacuation. Policy Research Working Paper 6058, World Bank, Washington, DC, May 2012. © World Bank. License: CC BY 3.0 IGO
work page 2012
-
[8]
Feyera A. Hirpa, Peter Salamon, Hylke E. Beck, Valerio Lorini, Lorenzo Alfieri, Ervin Zsoter, and Simon J. Dadson. Calibration of the global flood awareness system (glofas) using daily streamflow data. Journal of Hydrology, 566: 0 595--606, 2018. ISSN 0022-1694. doi:https://doi.org/10.1016/j.jhydrol.2018.09.052. URL https://www.sciencedirect.com/science/a...
Show all 26 references
-
[9]
Long short-term memory
Sepp Hochreiter and J \"u rgen Schmidhuber. Long short-term memory. Neural Computation, 9 0 (8): 0 1735--1780, 1997. doi:10.1162/neco.1997.9.8.1735
1997 doi
-
[10]
Hrachowitz, H.H.G
M. Hrachowitz, H.H.G. Savenije, G. Blöschl, J.J. McDonnell, M. Sivapalan, J.W. Pomeroy, B. Arheimer, T. Blume, M.P. Clark, U. Ehret, F. Fenicia, J.E. Freer, A. Gelfan, H.V. Gupta, D.A. Hughes, R.W. Hut, A. Montanari, S. Pande, D. Tetzlaff, P.A. Troch, S. Uhlenbrook, T. Wagener...
2013
-
[11]
W. J. M. Knoben, J. E. Freer, and R. A. Woods. Technical note: Inherent benchmark or not? comparing nash--sutcliffe and kling--gupta efficiency scores. Hydrology and Earth System Sciences, 23 0 (10): 0 4323--4331, 2019. doi:10.5194/hess-23-4323-2019. URL https://hess.copernicu...
2019 doi
-
[12]
Kratzert, D
F. Kratzert, D. Klotz, C. Brenner, K. Schulz, and M. Herrnegger. Rainfall--runoff modelling using long short-term memory (lstm) networks. Hydrology and Earth System Sciences, 22 0 (11): 0 6005--6022, 2018. doi:10.5194/hess-22-6005-2018. URL https://hess.copernicus.org/articles...
2018 doi
-
[13]
Kratzert, D
F. Kratzert, D. Klotz, G. Shalev, G. Klambauer, S. Hochreiter, and G. Nearing. Towards learning universal, regional, and local hydrological behaviors via machine learning applied to large-sample datasets. Hydrology and Earth System Sciences, 23 0 (12): 0 5089--5110, 2019 a . d...
2019 doi
-
[14]
Kratzert, M
F. Kratzert, M. Gauch, D. Klotz, and G. Nearing. Hess opinions: Never train a long short-term memory (lstm) network on a single basin. Hydrology and Earth System Sciences, 28 0 (17): 0 4187--4201, 2024. doi:10.5194/hess-28-4187-2024. URL https://hess.copernicus.org/articles/28...
2024 doi
-
[15]
NeuralHydrology – Interpreting LSTMs in Hydrology, page 347–362
Frederik Kratzert, Mathew Herrnegger, Daniel Klotz, Sepp Hochreiter, and Günter Klambauer. NeuralHydrology – Interpreting LSTMs in Hydrology, page 347–362. Springer International Publishing, 2019 b . ISBN 9783030289546. doi:10.1007/978-3-030-28954-6_19. URL http://dx.doi.org/1...
2019 doi
-
[16]
Neuralhydrology --- a python library for deep learning research in hydrology
Frederik Kratzert, Martin Gauch, Grey Nearing, and Daniel Klotz. Neuralhydrology --- a python library for deep learning research in hydrology. Journal of Open Source Software, 7 0 (71): 0 4050, 2022. doi:10.21105/joss.04050. URL https://doi.org/10.21105/joss.04050
2022 doi
-
[17]
Caravan-a global community dataset for large-sample hydrology
Frederik Kratzert et al. Caravan-a global community dataset for large-sample hydrology. Scientific Data, 10 0 (1): 0 61, 2023
2023
-
[18]
T. Lees, S. Reece, F. Kratzert, D. Klotz, M. Gauch, J. De Bruijn, R. Kumar Sahu, P. Greve, L. Slater, and S. J. Dadson. Hydrological concept formation inside long short-term memory (lstm) networks. Hydrology and Earth System Sciences, 26 0 (12): 0 3079--3101, 2022. doi:10.5194...
2022 doi
-
[19]
Zwiers, and Gabriele C
Seung-Ki Min, Xuebin Zhang, Francis W. Zwiers, and Gabriele C. Hegerl. Human contribution to more-intense precipitation extremes. Nature, 470 0 (7334): 0 378--381, Feb 2011. ISSN 1476-4687. doi:10.1038/nature09763. URL https://doi.org/10.1038/nature09763
2011 doi
-
[20]
G. S. Nearing, D. Klotz, J. M. Frame, M. Gauch, O. Gilon, F. Kratzert, A. K. Sampson, G. Shalev, and S. Nevo. Technical note: Data assimilation and autoregression for using near-real-time streamflow observations in long short-term memory networks. Hydrology and Earth System Sc...
2022 doi
-
[21]
Global prediction of extreme floods in ungauged watersheds
Grey Nearing, Deborah Cohen, Vusumuzi Dube, Martin Gauch, Oren Gilon, Shaun Harrigan, Avinatan Hassidim, Daniel Klotz, Frederik Kratzert, Asher Metzger, Sella Nevo, Florian Pappenberger, Christel Prudhomme, Guy Shalev, Shlomo Shenzis, Tadele Yednkachw Tekalign, Dana Weitzner, ...
2024 doi
-
[22]
Roi-Cohen and E
O. Roi-Cohen and E. Morin. Global vs. regional training of deep learning streamflow prediction models using the caravan data set for interpreting high vs. low flow prediction (poster). Flood Forecasting Meets Machine Learning Google Workshop, January 22-23 2024. Virtual
2024
-
[23]
Rumelhart, Geoffrey E
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning internal representations by error propagation. Technical Report ICS 8506, Institute for Cognitive Science, University of California, San Diego, California, 1986
1986
-
[24]
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Lukasz Kaiser, Illia Polosukhin, Andreas Kaiser, Sharan Narang, Geoffrey Shazeer, and Yoshua Bengio. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Syst...
2017 arXiv
-
[25]
Tree-structured parzen estimator: Understanding its algorithm components and their roles for better empirical performance
Shuhei Watanabe. Tree-structured parzen estimator: Understanding its algorithm components and their roles for better empirical performance. ArXiv, abs/2304.11127, 2023
2023 arXiv
-
[26]
State of climate services 2021: Water
World Meteorological Organization . State of climate services 2021: Water. Report WMO-No. 1278, World Meteorological Organization, Geneva, 2021. URL https://library.wmo.int/index.php?lvl=notice_display&id=21963
2021
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.