{"id":"71c72c08-de31-4d3d-9ed5-1aa2fb2467b0","arxiv_id":"2411.19323","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An EnsCGP-initialized ResNet-18 with ECDF bias correction emulates LISFLOOD-FP flood depths in downtown Chicago with R2 over 0.96 and median depth errors around 1 cm.","lead":"A rainfall-to-flood-depth surrogate that primes a ResNet-18 with an ensemble-approximated conditional Gaussian process and corrects bias afterward predicts downtown Chicago flood depths in about 0.006 seconds per event. The authors report R2 above 0.96 against LISFLOOD-FP simulations, a speedup that could make large-scale flood risk assessment far more practical if it generalizes beyond the small test area.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pixel-wise R² and 1-cm median error are computed over a mostly dry 0.3-m-max crop; without wet-only metrics the abstract's R²>0.96 and ~1% relative-error claim is not supported.","rationale":"The reader's conditional verdict correctly identifies missing baselines, absence of code/data, and the simulation-as-truth limitation. My stress-test pass finds an additional, more immediate threat to the abstract's quantitative claim: the evaluation metric is dominated by a majority class of dry pixels. The paper itself supplies the ingredients: max depth ≈0.3 m in the crop (Sec. 5.1), pixel-wise R² over 170×130 maps (Eq. 8), and a reported median absolute error of ≈1 cm (Sec. 5.2). No wet-only statistics are reported and the relative-error definition is never stated. The internal inconsistency between the abstract's 'median relative errors ~1%' and the results' 'median error ~1 cm' is a red flag; the two cannot both be meaningful on depths of 0.1–0.3 m unless the relative error is computed against something other than local wet depth. This concern is testable without new simulations. It is more load-bearing than the simulation-bias objection for the headline claim, because even granting LISFLOOD-FP as perfect ground truth, the reported numbers do not yet demonstrate accurate flood-depth estimation on inundated pixels. The ECDF transfer issue is secondary but real, and the paper's own 2017 result (R²=0.89) is consistent with a failure of that transfer. For these reasons I do not change the reader's conditional verdict; I would make the wet-only metric check an explicit condition for acceptance.","tokens_in":10899,"tokens_out":7259,"duration_ms":64312,"concrete_test":"Ask the authors to release the per-event pixel-level emulations and LISFLOOD-FP fields (or, failing that, digitize the scatter plots in Figs. 6 and 10), and recompute, per test event: (1) R² restricted to pixels with simulated depth ≥1 cm; (2) median relative absolute error |ŷ−y|/y on those wet pixels; and (3) the same metrics on dry pixels alone. If wet-only R² falls below ~0.9 or wet-only median relative error exceeds ~5–10%, the abstract's headline numbers are artifacts of dry-pixel dominance. Also recompute 2017 metrics with and without ECDF correction to test whether the bias correction transfers to years with deeper events.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claim is not supported by the metrics as reported. The evaluation crop (Fig. 1b) has maximum simulated depth only about 0.3 m (Sec. 5.1), and Eq. 8 computes R² pixel-wise over the whole 170×130 domain. In pluvial flood maps of downtown Chicago, the large majority of pixels are dry or near zero in most events. A model that predicts near-zero depths everywhere therefore achieves a very high R² and a median absolute error at or near zero, because dry pixels are trivially exact. Section 5.2 reports 'median water depth error across all years is approximately 1 cm' (Fig. 11), but the abstract claims 'median relative errors ... about 1 percent'; no definition of the relative error is given. On the 0.1–0.3 m wet pixels, 1 cm is 3–10% error or more, and on shallower wet pixels it is much larger. The reported metrics hence do not distinguish skill on dry background from skill on inundated pixels. In addition, the ECDF correction is estimated from validation years and applied to test years; the paper itself reports degradation in 2017 when test depths exceed the validation range (R²=0.89, Sec. 5.2), so transfer of the correction is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a surrogate model for pluvial inundation in a downtown Chicago crop. The workflow first computes an EnsCGP initial guess from Daymet rainfall using PCA-truncated covariance (Eq. 5), then refines it with a ResNet-18, and finally applies an ECDF quantile-mapping bias correction estimated on validation years. The target labels are LISFLOOD-FP daily maximum flood depth maps. Evaluation is performed by training on years up to a target year and testing on the following year for 2013-2019. The paper reports average R^2 above 0.96, median depth errors near 1 cm, and a per-event runtime near 0.006 seconds, and the abstract claims median relative errors around 1%. The title's 'few data' claim is not operationalized: the abstract itself says training requires approximately 100 training iterations and extensive data.","tokens_in":11160,"tokens_out":7327,"duration_ms":57368,"significance":"If the accuracy claims were robust, the surrogate would be a useful fast emulator for flood risk screening. The evaluation has genuine strengths: test years are held out sequentially, the target is a well-defined hydrodynamic model output, the runtime gain over LISFLOOD-FP is substantial, and the metrics are computed on event-level test data. The EnsCGP priming idea is interesting and follows a prior downscaling application. However, the central claims as written are not supported: the 'few data' claim is untested, the reported R^2 is dominated by dry pixels, the relative error metric is undefined, and the ECDF correction's transfer to test years is questionable, particularly for 2017. These issues are fixable with additional analyses, so the paper is worthy of major revision rather than rejection.","major_comments":[{"comment":"The central 'few data' claim is not tested. The abstract states that training required 'approximately 100 training iterations and extensive data,' and §5.1 trains the EnsCGP on all events from February 1981 to December 2012 before testing on 2013. No experiment varies the training-set size or the number of simulated events to demonstrate behavior in the low-data regime, so the title and introduction's premise are unsupported as presented. An ablation that reduces the number of training years/events is needed to substantiate the data-efficiency claim.","section":"Abstract; §4.2; §5.1"},{"comment":"The headline accuracy metrics are dominated by dry pixels. Eq. (8) computes R^2 pixel-wise over the entire 170 by 130 crop, and §5.1 reports that the maximum simulated depth in the crop is about 0.3 m. Across mostly dry pluvial maps, a near-zero prediction achieves very high R^2 and a median error near zero, so these quantities do not measure skill on inundated pixels. The abstract's 'median relative errors ... about 1 percent' is further unsupported because no relative-error definition is given; the only error reported in §5.2 is a median absolute error of about 1 cm (Fig. 11), which is far larger than 1% on wet pixels with depths of 0.1-0.3 m. Please report wet-only metrics and define the relative error.","section":"Eq. (8); §5.1; §5.2"},{"comment":"The ECDF bias-correction transfer is not established. §4.3 says the correction is estimated on the validation period and applied to the testing period; §5.2 reports that in 2017, when test-set maxima reached 0.35 m versus a validation maximum near 0.15 m, R^2 fell to 0.89. Quantile mapping outside the fitted range is extrapolation and is not validated by the current protocol. The paper should either restrict the correction to the validation range, model the extrapolation, or report results with and without the ECDF correction to separate the network's skill from the correction's contribution.","section":"§4.3; §5.2"},{"comment":"The observational validation does not cover the surrogate domain. The NWIS comparison in Fig. 2 is for the full Chicago region of Fig. 1a, whereas the surrogate is applied to the downtown crop of Fig. 1b. The inference in §5.1 that surrogate results 'can be comparable with the observation' chains two separate comparisons (observation versus simulation over the full region, and simulation versus emulation over the crop) and does not establish accuracy of the emulated depths in the crop. This conclusion should be removed or supported by validation within the crop.","section":"§3; §5.1"},{"comment":"The coupling between EnsCGP and ResNet-18 is underspecified. Fig. 3 shows the EnsCGP initial guess entering the network, but the text does not say whether the guess is an extra input channel, an additive correction, a target for a residual branch, or a weight initialization. Without this detail the method cannot be reproduced or ablated, and the claimed benefit of priming is not isolated. Please specify the coupling and include an ablation with and without the EnsCGP guess.","section":"§4.1–4.2"}],"minor_comments":[{"comment":"The caption colors are inconsistent with the main text: the text states the black line is EnsCGP and the green line is ResNet-18, while the caption says 'GP error (green)' and 'ResNet (black)'.","section":"Fig. 5"},{"comment":"In Eq. (5), U_k^{-1} should be U_k^T because U is orthogonal; please define the dimensions of X, Y, and the matrices in Eqs. (1)–(5).","section":"Eq. (5)"},{"comment":"The caption contains a typo ('mater' should be 'meter') and should state the units and color scale for the depth maps.","section":"Fig. 7"},{"comment":"The term 'validation set' is ambiguous in the rolling-window setup; Fig. 9 should state explicitly whether the validation period is the 20% split within the training years used for early stopping.","section":"§5.2"},{"comment":"The phrase 'approximately 100 training iterations' conflates epochs with data-set size; please state the number of epochs, the number of training events, and what is meant by 'few data'.","section":"Abstract; §4.2"},{"comment":"Reference [51] for NWIS is listed as an arXiv preprint, which is not an appropriate citation for the USGS National Water Information System; please cite the data service directly.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be an early technical report. The central claims are overstated relative to the experiments; if the authors can add wet-only metrics and data-efficiency ablations, it may become suitable. The manuscript also has a mismatch between title and abstract and several citation formatting issues. Given the journal's standards, I recommend major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a straightforward application of EnsCGP-primed ResNet-18 plus ECDF bias correction to pluvial flood emulation in downtown Chicago. The components come from prior work (Saha & Ravela 2024), and the newness is the application. Held-out test years are external, and the LISFLOOD-FP validation against gauges (Fig. 2) is a real strength.\n\nThe big problem is the headline claim. The abstract says R2 > 0.96 with median relative errors about 1%. The evaluation domain is a 170x130 crop with max depth about 0.3 m, and R2 is computed pixel-wise over that entire grid. Most pixels are dry or near zero in every event. A model that predicts near-zero depths everywhere gets a very high R2 and a median absolute error around 1 cm, because the zero pixels are trivially correct. The text reports median absolute error ~1 cm, not relative error; on wet pixels (0.1-0.3 m) that is 3-10% or worse, and on shallower wet pixels it is larger. The stress-test note is right: the abstract's 'about 1%' is unsupported as stated.\n\nThe other soft spots follow from the same pattern. The 'few data' claim is not tested: there is no ablation that reduces the training set, and the paper itself says the training needed 'approximately 100 iterations and extensive data.' No standalone ResNet baseline or EnsCGP-alone comparison is shown, so the priming benefit is asserted, not demonstrated. The ECDF correction is fit on validation years and applied to test years; the 2017 degradation (R2 = 0.89) when test depths exceed the validation range is acknowledged but not investigated. No code or data are provided, so reproducibility is limited.\n\nI want to be fair: the pipeline is coherent, the writing is clear, held-out year evaluation is a reasonable check, and the authors are transparent about the small depths in this crop and about 2017. The flaw is in the reporting, not the architecture. If the authors recompute skill on wet pixels only, report relative errors properly, add a training-size ablation, and publish code, the paper could be a decent applied contribution. As it stands, the central claim does not have the support the abstract implies.\n\nAudience: people working on flood emulation and surrogate models. A serious referee could push it into a workable state, so I would send it to review, but I would want the revisions first. My own verdict is skeptical until the metrics are fixed.","headline":"A plausible flood-emulator pipeline whose headline accuracy claim is inflated by dry-pixel metrics; the 'few data' framing is not tested.","tokens_in":11683,"tokens_out":2271,"would_cite":false,"duration_ms":19521,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A rainfall-only surrogate model emulates LISFLOOD-FP flood depths for a downtown Chicago domain with average R2 above 0.96 and median relative errors around one percent, running in about 0.006 seconds per event.","keywords":["flood surrogate modeling","EnsCGP","ResNet-18","bias correction","LISFLOOD-FP","rainfall-driven inundation","Chicago","machine learning emulator"],"falsifier":"Running the trained surrogate on a set of extreme rainfall events that produce flood depths above the validation maximum (as in 2017) and checking whether the pixel-wise R2 against LISFLOOD-FP drops below the claimed 0.96 would test the generalization claim directly; an independent check would compare surrogate depths to observed gauge readings inside the crop.","tokens_in":10691,"feed_emoji":"🌊","tokens_out":4993,"duration_ms":40248,"temperature":0.7,"pith_summary":"This paper is trying to establish that a fast machine-learning surrogate for hydrodynamic flood simulation can be built from surprisingly little training data if the network is started from a statistical guess rather than from random weights. The specific claim is that priming a ResNet-18 with an ensemble-approximated conditional Gaussian process (EnsCGP), then applying an empirical-CDF bias correction, lets rainfall data alone reproduce LISFLOOD-FP flood depths for a downtown Chicago domain with average R2 above 0.96 and median relative errors around one percent, in about six milliseconds per event. The payoff would be that flood risk assessments, uncertainty quantification, and extreme-event screening, which currently need thousands of expensive hydrodynamic runs, could be done with a surrogate that is several orders of magnitude faster.","feed_headline":"Flood surrogate hits R2>0.96 in 0.006 seconds","feed_subtitle":"Rainfall-only emulator with EnsCGP priming and bias correction reproduces Chicago flood depths with ~1% median error.","key_machinery":"The central object is the ensemble-approximated conditional Gaussian process (EnsCGP), a covariance-based linear estimator, evaluated in a PCA-reduced space with k=7 at 95% cumulative energy, which produces a region-aware initial flood-depth guess in about 0.01 seconds. The ResNet-18 residual network then refines spatial detail, and the empirical cumulative distribution function (ECDF) quantile mapping removes systematic bias using the validation period. The input is Daymet rainfall only; the training labels are LISFLOOD-FP simulations.","core_discovery":"The paper claims that a surrogate flood model can learn the rainfall-to-inundation map for a small urban domain with only about 100 training passes, by first making a fast Gaussian-process guess (EnsCGP) and then letting a ResNet-18 refine the residual, with an ECDF quantile correction at the end. Tested on the downtown Chicago crop from 2013 through 2019, the surrogate yields average R2 greater than 0.96, median relative errors around 1 percent, a median absolute depth error near 1 cm, and a runtime of roughly 0.006 seconds per event against a 20-minute LISFLOOD-FP run.","pith_inferences":["The 'few data' framing is partly aspirational: the paper's own training used about 100 epochs and the full 1981 to 2012 record, so the data-efficiency claim needs a direct experiment with reduced training years to be confirmed.","If the ECDF bias correction is estimated from validation years that do not contain the largest events, the correction may not extrapolate to deeper floods; the 2017 case (test maximum 0.35 m versus validation maximum 0.15 m) already shows a dip to R2 = 0.89, suggesting a testable ceiling.","Because the paper only tests a crop with maximum depths around 0.3 m in 2013, claims about emulating 2.5 to 3 m floods are not yet supported by the reported experiments.","The priming-then-refine recipe could apply to other environmental emulation tasks where a cheap linear statistical guess is improved by a CNN, such as storm surge or landslide hazard mapping, but this is speculation beyond the paper."],"forward_implications":["If the surrogate generalizes as claimed, a full Monte Carlo flood risk assessment over thousands of synthetic rainfall events becomes computationally trivial: roughly 0.006 seconds per event instead of 20 minutes per hydrodynamic simulation.","The same EnsCGP-primed ResNet workflow could be retrained for other cropped regions (the paper lists future crops) using the same rainfall and simulation data pipeline, with only modest retraining cost.","Bias-corrected surrogate output can be compared directly with gauge observations, because the paper shows simulation-versus-observation agreement for the full region and surrogate-versus-simulation agreement for the crop, making the surrogate usable for real-world depth estimates.","The framework is designed to ingest CMIP6 and HighResMIP climate projections, so the paper expects it to produce flood hazard maps under future climate scenarios without rerunning the hydrodynamic model for every projection."],"supporting_citations":[{"why":"Supplies the ResNet-18 architecture that refines the EnsCGP guess.","marker":"[11]"},{"why":"Supplies the ensemble-approximated conditional Gaussian process formulation used for the initial flood-depth prediction.","marker":"[12]"},{"why":"Shows the statistical priming mechanism that, in prior downscaling work, reduced data requirements during training.","marker":"[15]"},{"why":"Provides the GPU-accelerated LISFLOOD-FP 8.1 solver whose simulations are the surrogate's training labels.","marker":"[46]"},{"why":"Supplies the Daymet 1-km rainfall data used as the surrogate's input.","marker":"[48]"},{"why":"Provides the observed stream and water-depth measurements used to validate the hydrodynamic model.","marker":"[51]"},{"why":"Provides the Hydrofunctions tool used to retrieve and process daily maximum water-depth data.","marker":"[52]"}],"fun_headline_variants":["Flood emulator: R2>0.96, 1% error, 0.006s per event","Surrogate flood model hits R2>0.96 with ~1% median error","Rainfall-to-flood surrogate: 0.006s runtime, R2>0.96","Few-data flood surrogate: R2>0.96, ~1% error, 0.006s","EnsCGP-primed ResNet predicts floods at 0.006s per event"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The surrogate's training labels come from LISFLOOD-FP simulations, so the whole accuracy claim rests on those simulations being close to real flood depths in the small downtown crop, and on the ECDF correction estimated from validation years staying valid in test years.","fun_headline_variants_meta":{"raw":{"variants":["Flood emulator: R2>0.96, 1% error, 0.006s per event","Surrogate flood model hits R2>0.96 with ~1% median error","Rainfall-to-flood surrogate: 0.006s runtime, R2>0.96","Few-data flood surrogate: R2>0.96, ~1% error, 0.006s","EnsCGP-primed ResNet predicts floods at 0.006s per event"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000334,"raw_usage":{"total_tokens":1838,"prompt_tokens":914,"completion_tokens":924,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":798}},"tokens_in":530,"tokens_out":924,"duration_ms":6889,"temperature":1.0,"reasoning_tokens":798,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:18:27.388021+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Running the trained surrogate on a set of extreme rainfall events that produce flood depths above the validation maximum (as in 2017) and checking whether the pixel-wise R2 against LISFLOOD-FP drops below the claimed 0.96 would test the generalization claim directly; an independent check would compare surrogate depths to observed gauge readings inside the crop.","supporting_citations":[{"cited_title":"Fast ensemble smoothing.Ocean Dynamics, 57(2):123– 134, 2007","cited_arxiv_id":null,"evidence_quote":"Supplies the ensemble-approximated conditional Gaussian process formulation used for the initial flood-depth prediction."},{"cited_title":"Statistical-physical adversarial learning from data and mod- els for downscaling rainfall extremes","cited_arxiv_id":null,"evidence_quote":"Shows the statistical priming mechanism that, in prior downscaling work, reduced data requirements during training."},{"cited_title":"Lisflood-fp 8.1: New gpu accelerated solvers for faster fluvial/pluvial flood simulations","cited_arxiv_id":null,"evidence_quote":"Provides the GPU-accelerated LISFLOOD-FP 8.1 solver whose simulations are the surrogate's training labels."},{"cited_title":"Daymet: Daily surface weather data on a 1-km grid for north america, version 2","cited_arxiv_id":null,"evidence_quote":"Supplies the Daymet 1-km rainfall data used as the surrogate's input."},{"cited_title":"National water information system (nwis).ArXiv preprint, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the observed stream and water-depth measurements used to validate the hydrodynamic model."},{"cited_title":"Hydrofunctions: A python package for working with hydrology data.ArXiv preprint, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the Hydrofunctions tool used to retrieve and process daily maximum water-depth data."}],"review_version":1}