Pith. sign in

REVIEW 3 major objections 5 minor 26 references

Fine Flood Forecasts: Incorporating local data into global models through fine-tuning

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Pith's one-line read Fine-tuning a global LSTM on local basin data raises flood-forecast skill, with the biggest gains where the global model struggles.

desk verdict Useful, honest workshop paper on fine-tuning global LSTM flood forecasts per basin; the central numbers likely hold but the paper must document the evaluation split before those numbers can be trusted. read the letter →

arxiv 2504.12559 v1 pith:CZK2IOJ4 submitted 2025-04-17 cs.LG physics.geo-ph

classification cs.LGphysics.geo-ph
keywords streamflowforecastingfloodpredictionfine-tuningtransferlearningLSTMglobalhydrologylocaldataNash-Sutcliffeefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that fine-tuning a globally pre-trained LSTM on data from individual river basins improves streamflow forecast accuracy, and that the largest gains appear exactly where the global model performs worst. This matters because national forecasters often have local data but cannot train global models; fine-tuning offers them a path to take ownership of a state-of-the-art model. The reported mean Nash-Sutcliffe efficiency rises from 0.473 to 0.541 (14%) and mean Kling-Gupta efficiency from 0.520 to 0.599 (15%) across 159 sampled basins, averaged over 8 pretrained model instances. The paper also confirms that single-basin training from scratch is worse than both global pretraining and fine-tuning. If the results hold, local agencies could deploy high-skill flood forecasts using their own proprietary data.

What carries the argument

The load-bearing mechanism is transfer learning by parameter fine-tuning of a 260K-parameter LSTM that ingests 365 days of atmospheric and catchment variables and outputs next-day streamflow. Global pretraining on 6,375 basins learns shared rainfall-runoff dynamics; fine-tuning then updates the network weights (either all weights or only the output head) on a single basin's record, with per-basin hyperparameter selection by validation NSE. A comparison against single-basin models trained from scratch isolates the value of the pretrained initialization.

What would settle it

Evaluate the fine-tuned and pre-trained models on a fixed test period that is explicitly disjoint from every validation decision, including the 50-trial hyperparameter sweep for each basin; if the mean NSE and KGE gains shrink to zero or reverse on that split, the paper's central claim fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper's claim is that a globally pre-trained LSTM retains transferable skill that per-basin fine-tuning can sharpen: across 159 randomly sampled basins and 8 pretrained model instances, mean Nash-Sutcliffe efficiency rises from 0.473 to 0.541 and mean Kling-Gupta efficiency from 0.520 to 0.599, gains of 14% and 15%. The distribution of gains is left-skewed: median gains are smaller (NSE +0.042, KGE +0.026), and improvement correlates negatively with pretrained skill, explaining 11.1% of the variance, so the weakest basins benefit most; every country in the sample has a net positive change. Fine-tuning is not guaranteed to help: 22% of base-model/basin pairs degrade, and 14% of basins still degrade on average. The same setup also shows that training an LSTM from scratch on a single basin (mean NSE 0.358) is worse than both the global model and the fine-tuned model, which the paper interprets as confirming that large-sample pretraining sets the right initialization.

Load-bearing premise

The comparison assumes the test-period scores in Table 1 are computed on data that was not used for the per-basin hyperparameter sweeps; the paper never describes the validation/test split, and its footnote mentions only a 'pre-defined training set'.

Editorial extensions

If this is right

  • National hydromet agencies can adapt a global pretrained model to their own basins without sharing proprietary data or training from scratch, lowering the compute and data barrier to operational ML flood forecasting.
  • The same fine-tuning recipe likely transfers to other deep-learning hydrology architectures, since the paper notes the principle is model-agnostic.
  • Basins underserved by global models, those with low pretrained skill, stand to gain the most, so fine-tuning could help close the flood-forecasting gap in developing regions.
  • Because fine-tuning needs only historical data rather than near-real-time assimilation, it offers a cheaper route to skill gains than data assimilation, which previously produced an 8% median NSE improvement on 539 US basins.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is whether fine-tuning on local data and near-real-time data assimilation are additive; the paper compares the two approaches but does not combine them.
  • The negative correlation between pretrained skill and fine-tuning gain suggests fine-tuning may act partly as bias correction for regions where global forcing or regional hydrology is poorly represented; one could test this by conditioning gains on aridity, basin area, and data quality.
  • For operational adoption, agencies will need a protocol for re-fine-tuning as new observations arrive, since the paper's per-basin hyperparameter sweeps are expensive; a cheaper fixed-schedule fine-tune would be a testable simplification.
  • Because 14% of basins still regress on average, a practical deployment rule such as only fine-tuning when validation NSE improves over the pretrained model would make the method safer; the paper does not propose such a rule.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a methodology for fine-tuning a globally pre-trained LSTM streamflow model on data from individual basins, using the Caravan dataset. It compares the fine-tuned model against the pre-trained model and against single-basin-trained models on a random sample of 159 basins, reporting that fine-tuning increases mean NSE from 0.473 to 0.541 and mean KGE from 0.520 to 0.599. The paper also reports a negative correlation between pre-trained skill and fine-tuning improvement, and provides a step-by-step guide for forecasters. The stated goal is to enable national agencies to take ownership of global ML flood-forecast models by adapting them with local data.

Significance. If the results hold, this work is practically significant: it offers a low-compute path for local agencies to improve global ML flood forecasts, addresses data-ownership concerns, and provides open-source code and a pre-trained model. The finding that fine-tuning helps most in basins where the global model performs worst is also useful for prioritizing deployment. The paper's strengths are its clear practical framing, the open-source release, and the multi-seed (8 seeds) experimental design. However, the central empirical claim depends entirely on a correctly held-out evaluation period, which the manuscript does not describe; the reported results also contain internal inconsistencies that must be resolved before the quantitative claims can be accepted.

major comments (3)
  1. [Section 2.2 and Section 3.1, Table 1] The manuscript never states the temporal boundaries of the training, validation, and test periods. Section 2.2 says per-basin hyperparameters are selected by optimizing validation NSE, but it does not say whether the evaluation period used for the NSE/KGE scores in Table 1 is disjoint from the fine-tuning training and validation periods. Given the 365-day input lookback and the strong autocorrelation of daily streamflow, any overlap between fine-tuning training data and the evaluation window can inflate skill because the model would have seen the observed flows it is asked to predict. The Table 1 footnote mentions a 'pre-defined training set' only for the single-basin baseline, not for the fine-tuned models. The authors must provide the exact date ranges for the training/validation/test split and a statement that the evaluation period is entirely unseen during fine-tuning (including hyperparameter selection). This is the load-bearing condition for the paper's central claim.
  2. [Table 1 footnote and Appendix C, Table 3] There are internal inconsistencies in the reported results. The footnote to Table 1 states that the single-basin model was trained on only 147 of the 159 basins, while the pre-trained and fine-tuned models appear to be evaluated on all 159 basins, so the three-way comparison in Table 1 is not on a common set of basins. Additionally, the standard errors for the same quantities differ between Table 1 and Table 3: for the fine-tuned model, KGE mean is 0.599 ± 0.042 in Table 1 but 0.599 ± 0.026 in Table 3; NSE mean is 0.541 ± 0.017 vs 0.541 ± 0.014; median NSE is 0.625 vs 0.627; median KGE is 0.709 vs 0.711. These discrepancies need to be reconciled, or the text should explicitly state what each interval represents (e.g., across seeds or across basins).
  3. [Section 3.1] The claim that fine-tuning provides a "significant increase" is not supported by any inferential statistic. The paper reports standard errors but no formal significance test (e.g., paired t-test or Wilcoxon signed-rank test across basins, or a test across the 8 seed instances). Given the large number of basins, the improvement may well be real, but the current use of the word "significant" is not backed by a stated test. The authors should add a formal test, or explicitly state that the improvement is consistent across all 8 seeds and all 159 basins, and clarify the aggregation procedure used to compute the reported means and intervals.
minor comments (5)
  1. [Figure 2 (right panel)] The text states that "for every country in the dataset, there is a net positive improvement," but the figure appears to show only a subset of countries. Please clarify whether this statement refers to all countries in the 159-basin sample and label the figure accordingly.
  2. [Section 3.2] The fraction of variance explained by the relationship between pre-trained skill and fine-tuning improvement is 11.1%; the text "improvement from fine-tuning is particularly large in these" overstates the strength of the relationship. Suggest tempering the language or reporting an effect-size measure.
  3. [Section 2.2] The text says "recursive neural networks (RNNs)" but should read "recurrent neural networks."
  4. [Abstract and Introduction] The phrase "short-perm predictions" appears to be a typo for "short-term predictions."
  5. [Appendix B.2 and Table 3] The caption for Table 3 says "mean ± std/√n" but does not define n (whether n=159 basins or n=8 seeds). Also, the text says 1272 data points (159×8), but then says the results are averaged over the fine-tuning values for each basin; please clarify the exact averaging order.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity is demonstrated; the fine-tuning comparison is an empirical evaluation against an external pre-trained baseline, and the only concern is an undocumented validation/test split, which is a statistical-transparency issue rather than a circular reduction.

full rationale

This paper makes no first-principles derivation; its central claims are empirical comparisons of model variants. The pre-trained global LSTM is an external artifact trained on 6375 basins, and the fine-tuned models are obtained by gradient descent on per-basin data, so the reported NSE/KGE improvements are not equal to any input by construction. The hyperparameter sweep selects configurations by validation NSE, but this is standard model selection; unless the Table 1 scores are computed on the same validation period, no circularity follows. The text does not state the temporal split of training, validation, and test periods, and the only explicit split reference is the Table 1 footnote mentioning a 'pre-defined training set' for the single-basin baseline, leaving the evaluation period unidentified. If the same validation set were reused for the headline numbers, the gains would be optimistically biased, but that would be a statistical validity problem, not a demonstrated circular reduction. The self-citations (e.g., Nearing et al. 2022/2024 and Kratzert et al. 2024) supply background and benchmarks, but the fine-tuning comparison itself is run in this paper and does not depend on those citations for its validity. Therefore, no circularity is demonstrated, and the score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the fine-tuning protocol and the evaluation split. No new physical entities or constants are introduced. The main free choices are the pretrained model hyperparameters and the per-basin hyperparameter sweep, both of which are standard model-selection choices; the lack of a documented validation/test split is the most consequential ledger item.

free parameters (3)
  • Pretrained LSTM hyperparameters (hidden size 256, batch size 256, 40 epochs, LR schedule 5e-5/5e-6) = Hidden size 256, LR 5e-5 (epochs 1-30), 5e-6 (epochs 31-40), 40 epochs
    Chosen by hand following Roi-Cohen and Morin (2024), not derived from first principles. These choices determine the quality of the pretrained base model and therefore the fine-tuning gains.
  • Per-basin fine-tuning hyperparameters (epochs, learning rate, loss, modules) = Selected per basin via 50 TPE trials on validation NSE; values not reported
    The TPE sweep is a model-selection procedure that fits hyperparameters to validation data. If the validation and test periods overlap, the reported improvements are optimistically biased.
  • Random sample of 159 basins = 159 basins sampled from 6375; seed not stated
    The paper does not report the seed or basin identifiers, so the exact sample is not reproducible from the text alone despite the random sampling claim.
assumptions (5)
  • domain assumption LSTM internal states can represent watershed hydrological dynamics such as storages and reservoirs
    Section 2.2 states LSTMs are suitable because their internal state space is similar to physical watershed systems, citing Kratzert et al. (2019b, 2018). This underpins the modeling approach.
  • ad hoc to paper There exists a valid temporal split such that the validation period used for hyperparameter selection does not overlap the test period used to report performance
    The paper refers to a 'pre-defined training set' in the Table 1 footnote but never specifies the split. If this assumption fails, the improvement claims are circular.
  • domain assumption Caravan streamflow and atmospheric data are reliable for all 6375 basins, including the 159 sampled
    Section 2.1 describes Caravan as globally sourced streamflow data mostly contributed by national hydromet agencies; data quality is assumed.
  • domain assumption NSE and KGE are sufficient metrics for judging flood forecast skill
    Section 2.3 states these are the main metrics in hydrology and cites Gauch et al. (2023) for the claim that they encode typical human preferences.
  • domain assumption The single-basin trained model is a fair baseline for 'no global pretraining'
    Section 3.1 compares against a model trained on a single basin with 50 TPE trials and hidden size in {16,32,64}; the comparison assumes this baseline is representative of basin-only training.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fine Flood Forecasts: Incorporating local data into global models through fine-tuning." pith.science (2026). https://pith.science/paper/CZK2IOJ4

@misc{pith2026250412559,
  author       = {Pith},
  title        = {Pith review of: Fine Flood Forecasts: Incorporating local data into global models through fine-tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CZK2IOJ4}},
  note         = {Machine review of arXiv:2504.12559}
}
read the original abstract

Floods are the most common form of natural disaster and accurate flood forecasting is essential for early warning systems. Previous work has shown that machine learning (ML) models are a promising way to improve flood predictions when trained on large, geographically-diverse datasets. This requirement of global training can result in a loss of ownership for national forecasters who cannot easily adapt the models to improve performance in their region, preventing ML models from being operationally deployed. Furthermore, traditional hydrology research with physics-based models suggests that local data -- which in many cases is only accessible to local agencies -- is valuable for improving model performance. To address these concerns, we demonstrate a methodology of pre-training a model on a large, global dataset and then fine-tuning that model on data from individual basins. This results in performance increases, validating our hypothesis that there is extra information to be captured in local data. In particular, we show that performance increases are most significant in watersheds that underperform during global training. We provide a roadmap for national forecasters who wish to take ownership of global models using their own data, aiming to lower the barrier to operational deployment of ML-based hydrological forecast systems.

Figures

Figures reproduced from arXiv: 2504.12559 by the authors.

Figure 1
Figure 1. Left: world map showing all 6375 in the Caravan dataset (grey), and our 159 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Correlation between fine-tuning improvements and pre-trained model skill across [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 16 canonical work pages

  1. [1]

    Enhancement of river flooding due to global warming

    Haireti Alifu, Yukiko Hirabayashi, Yukiko Imada, and Hideo Shiogama. Enhancement of river flooding due to global warming. Scientific Reports, 12 0 (1): 0 20687, Nov 2022. ISSN 2045-2322. doi:10.1038/s41598-022-25182-6. URL https://doi.org/10.1038/s41598-022-25182-6

  2. [2]

    Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures

    James Bergstra, Daniel Yamins, and David D Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 115--123. PMLR, 2013

  3. [3]

    The human cost of natural disasters 2015: A global perspective

    Centre for Research on the Epidemiology of Disasters. The human cost of natural disasters 2015: A global perspective. Technical report, 2015. URL https://reliefweb.int/report/world/human-cost-natural-disasters-2015-global-perspective

  4. [4]

    Färber, H

    C. Färber, H. Plessow, S. Mischel, F. Kratzert, N. Addor, G. Shalev, and U. Looser. Grdc-caravan: extending caravan with data from the global runoff data centre. Earth System Science Data Discussions, pages 1--28, 2024. doi:10.5194/essd-2024-427

  5. [5]

    In defense of metrics: Metrics sufficiently encode typical human preferences regarding hydrological model performance

    Martin Gauch, Frederik Kratzert, Oren Gilon, Hoshin Gupta, Juliane Mai, Grey Nearing, Bryan Tolson, Sepp Hochreiter, and Daniel Klotz. In defense of metrics: Metrics sufficiently encode typical human preferences regarding hydrological model performance. Water Resources Research, 59 0 (6): 0 e2022WR033918, 2023. doi:https://doi.org/10.1029/2022WR033918. UR...

  6. [6]

    Gupta, Harald Kling, Koray K

    Hoshin V. Gupta, Harald Kling, Koray K. Yilmaz, and Guillermo F. Martinez. Decomposition of the mean squared error and nse performance criteria: Implications for improving hydrological modelling. Journal of Hydrology, 377 0 (1): 0 80--91, 2009. ISSN 0022-1694. doi:https://doi.org/10.1016/j.jhydrol.2009.08.003. URL https://www.sciencedirect.com/science/art...

  7. [7]

    A cost effective solution to reduce disaster losses in developing countries: Hydro-meteorological services, early warning, and evacuation

    Stéphane Hallegatte. A cost effective solution to reduce disaster losses in developing countries: Hydro-meteorological services, early warning, and evacuation. Policy Research Working Paper 6058, World Bank, Washington, DC, May 2012. © World Bank. License: CC BY 3.0 IGO

  8. [8]

    Hirpa, Peter Salamon, Hylke E

    Feyera A. Hirpa, Peter Salamon, Hylke E. Beck, Valerio Lorini, Lorenzo Alfieri, Ervin Zsoter, and Simon J. Dadson. Calibration of the global flood awareness system (glofas) using daily streamflow data. Journal of Hydrology, 566: 0 595--606, 2018. ISSN 0022-1694. doi:https://doi.org/10.1016/j.jhydrol.2018.09.052. URL https://www.sciencedirect.com/science/a...

Show all 26 references
  1. [9]

    Long short-term memory

    Sepp Hochreiter and J \"u rgen Schmidhuber. Long short-term memory. Neural Computation, 9 0 (8): 0 1735--1780, 1997. doi:10.1162/neco.1997.9.8.1735

  2. [10]

    Hrachowitz, H.H.G

    M. Hrachowitz, H.H.G. Savenije, G. Blöschl, J.J. McDonnell, M. Sivapalan, J.W. Pomeroy, B. Arheimer, T. Blume, M.P. Clark, U. Ehret, F. Fenicia, J.E. Freer, A. Gelfan, H.V. Gupta, D.A. Hughes, R.W. Hut, A. Montanari, S. Pande, D. Tetzlaff, P.A. Troch, S. Uhlenbrook, T. Wagener...

  3. [11]

    W. J. M. Knoben, J. E. Freer, and R. A. Woods. Technical note: Inherent benchmark or not? comparing nash--sutcliffe and kling--gupta efficiency scores. Hydrology and Earth System Sciences, 23 0 (10): 0 4323--4331, 2019. doi:10.5194/hess-23-4323-2019. URL https://hess.copernicu...

  4. [12]

    Kratzert, D

    F. Kratzert, D. Klotz, C. Brenner, K. Schulz, and M. Herrnegger. Rainfall--runoff modelling using long short-term memory (lstm) networks. Hydrology and Earth System Sciences, 22 0 (11): 0 6005--6022, 2018. doi:10.5194/hess-22-6005-2018. URL https://hess.copernicus.org/articles...

  5. [13]

    Kratzert, D

    F. Kratzert, D. Klotz, G. Shalev, G. Klambauer, S. Hochreiter, and G. Nearing. Towards learning universal, regional, and local hydrological behaviors via machine learning applied to large-sample datasets. Hydrology and Earth System Sciences, 23 0 (12): 0 5089--5110, 2019 a . d...

  6. [14]

    Kratzert, M

    F. Kratzert, M. Gauch, D. Klotz, and G. Nearing. Hess opinions: Never train a long short-term memory (lstm) network on a single basin. Hydrology and Earth System Sciences, 28 0 (17): 0 4187--4201, 2024. doi:10.5194/hess-28-4187-2024. URL https://hess.copernicus.org/articles/28...

  7. [15]

    NeuralHydrology – Interpreting LSTMs in Hydrology, page 347–362

    Frederik Kratzert, Mathew Herrnegger, Daniel Klotz, Sepp Hochreiter, and Günter Klambauer. NeuralHydrology – Interpreting LSTMs in Hydrology, page 347–362. Springer International Publishing, 2019 b . ISBN 9783030289546. doi:10.1007/978-3-030-28954-6_19. URL http://dx.doi.org/1...

  8. [16]

    Neuralhydrology --- a python library for deep learning research in hydrology

    Frederik Kratzert, Martin Gauch, Grey Nearing, and Daniel Klotz. Neuralhydrology --- a python library for deep learning research in hydrology. Journal of Open Source Software, 7 0 (71): 0 4050, 2022. doi:10.21105/joss.04050. URL https://doi.org/10.21105/joss.04050

  9. [17]

    Caravan-a global community dataset for large-sample hydrology

    Frederik Kratzert et al. Caravan-a global community dataset for large-sample hydrology. Scientific Data, 10 0 (1): 0 61, 2023

  10. [18]

    T. Lees, S. Reece, F. Kratzert, D. Klotz, M. Gauch, J. De Bruijn, R. Kumar Sahu, P. Greve, L. Slater, and S. J. Dadson. Hydrological concept formation inside long short-term memory (lstm) networks. Hydrology and Earth System Sciences, 26 0 (12): 0 3079--3101, 2022. doi:10.5194...

  11. [19]

    Zwiers, and Gabriele C

    Seung-Ki Min, Xuebin Zhang, Francis W. Zwiers, and Gabriele C. Hegerl. Human contribution to more-intense precipitation extremes. Nature, 470 0 (7334): 0 378--381, Feb 2011. ISSN 1476-4687. doi:10.1038/nature09763. URL https://doi.org/10.1038/nature09763

  12. [20]

    G. S. Nearing, D. Klotz, J. M. Frame, M. Gauch, O. Gilon, F. Kratzert, A. K. Sampson, G. Shalev, and S. Nevo. Technical note: Data assimilation and autoregression for using near-real-time streamflow observations in long short-term memory networks. Hydrology and Earth System Sc...

  13. [21]

    Global prediction of extreme floods in ungauged watersheds

    Grey Nearing, Deborah Cohen, Vusumuzi Dube, Martin Gauch, Oren Gilon, Shaun Harrigan, Avinatan Hassidim, Daniel Klotz, Frederik Kratzert, Asher Metzger, Sella Nevo, Florian Pappenberger, Christel Prudhomme, Guy Shalev, Shlomo Shenzis, Tadele Yednkachw Tekalign, Dana Weitzner, ...

  14. [22]

    Roi-Cohen and E

    O. Roi-Cohen and E. Morin. Global vs. regional training of deep learning streamflow prediction models using the caravan data set for interpreting high vs. low flow prediction (poster). Flood Forecasting Meets Machine Learning Google Workshop, January 22-23 2024. Virtual

  15. [23]

    Rumelhart, Geoffrey E

    David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning internal representations by error propagation. Technical Report ICS 8506, Institute for Cognitive Science, University of California, San Diego, California, 1986

  16. [24]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Lukasz Kaiser, Illia Polosukhin, Andreas Kaiser, Sharan Narang, Geoffrey Shazeer, and Yoshua Bengio. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Syst...

  17. [25]

    Tree-structured parzen estimator: Understanding its algorithm components and their roles for better empirical performance

    Shuhei Watanabe. Tree-structured parzen estimator: Understanding its algorithm components and their roles for better empirical performance. ArXiv, abs/2304.11127, 2023

  18. [26]

    State of climate services 2021: Water

    World Meteorological Organization . State of climate services 2021: Water. Report WMO-No. 1278, World Meteorological Organization, Geneva, 2021. URL https://library.wmo.int/index.php?lvl=notice_display&id=21963

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.