{"id":"467a43c4-9bc9-4f4d-9bcd-799f7c41e04e","arxiv_id":"2501.00149","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Active learning with surrogate models identifies storm-tide producing tropical cyclones in a large downscaled catalog with high precision and far fewer hydrodynamic simulations than batch training.","lead":"This paper uses machine-learning surrogate models and an active sampling loop to find tropical cyclones that produce extreme storm tides, without simulating every storm. In a Bangladesh case study, the method reports 100% precision in retrieving damaging storms while using under 20% of the catalog for hydrodynamic simulation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Active learning precision is measured on the self-selected shrinking pool, not on a held-out or fresh catalog; 100% precision may not transfer.","rationale":"The batch experiments provide credible evidence that XGBoost surrogates can classify damaging TCs and generalize across climate scenarios. The active sampling experiment is the novel contribution, and its headline result depends entirely on the in-sample evaluation. The protocol is a closed loop: selection is based on the surrogate's predictions, and the selected examples become training data. This can induce confirmation bias; the model may become very precise on the subset it has chosen while remaining blind to other damaging TCs. Without a held-out evaluation or a cross-climate test for the active learner, the paper cannot distinguish between genuine search skill and overfitting to the active sampling trajectory. The batch generalization results do not rescue the active claim because the active learner's training distribution is not the same as the batch training distribution. Therefore, the central claim is conditional on an independent evaluation. The reader's weakest assumption identifies the same issue, and I agree with it. The recommended verdict is unchanged (conditional acceptance pending the held-out and cross-climate evaluation).","tokens_in":12374,"tokens_out":6500,"duration_ms":65053,"concrete_test":"Split the 4100-TC ERA5 catalog into an 80% active-search pool and a 20% held-out validation catalog. Run the Experiment IVb procedure using only the search pool, and at each iteration evaluate the current Ens-CGP surrogate on the held-out validation catalog (and on the 2000-TC SSP5-8.5 catalog) with the same 3 m threshold. Record precision and recall on both held-out sets at the iteration where 20% of the search pool has been simulated. If the held-out precision is substantially below the in-pool precision, or if the cross-climate precision is below the 80% level achieved by the batch surrogate, the 100% precision claim does not generalize and the abstract's claim should be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim is the 100% precision achieved by the Ens-CGP active sampler after evaluating 20% of the ERA5 catalog (Experiment IVb, Figure 8). The evaluation protocol, however, measures precision on the pool of unsimulated TCs that the surrogate itself is actively filtering, and the selected TCs are immediately added to the training set. Because the training set becomes dominated by storms the surrogate already predicts to be damaging, the model is optimized to be precise on the very distribution it is creating. There is no held-out subset of ERA5 and no SSP5-8.5 evaluation for the active learner; the abstract's 'generalizable to climate scenarios' is only demonstrated for batch-trained surrogates (Experiments II and III). If the active sampler exploits its own training distribution rather than learning a general hazard mapping, the reported 100% precision is a property of the self-selected set, not of a fresh catalog. The paper also does not report recall at the 20% point, so the practical value of the precision claim is ambiguous.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper addresses the computational bottleneck of identifying tropical cyclones (TCs) whose storm tides exceed a 3 m damage threshold at any of 54 Bangladesh coastal stations, where gold-standard labels come from expensive ADCIRC hydrodynamic simulations. The authors evaluate surrogate strategies in two modes. In batch mode, XGBoost trained on 75% of an ERA5-downscaled catalog (4100 TCs) achieves 81% precision and 84% recall on the held-out quarter; trained on all ERA5 and tested on 2000 EC-EARTH-3 SSP5-8.5 TCs it achieves 82% precision and 87% recall; and trained on only 20% of ERA5 it retains 80% precision and 84% recall when tested on SSP5-8.5. The headline contribution is an active, 'informative' learning loop (LASSE): starting from a 1% seed, a surrogate iteratively selects the most destructive predicted storms, which are then hydrodynamically simulated and added to the training set. With an ensemble-approximate conditional Gaussian process (Ens-CGP) surrogate, the paper claims 100% precision in retrieving destructive storms after evaluating roughly 20% of the catalog, with an adjustable operating point favoring precision or recall; an XGBoost-based active variant is presented as comparatively ineffective.","tokens_in":12558,"tokens_out":19034,"duration_ms":166541,"significance":"The batch results are internally consistent and practically useful: the ERA5-to-SSP5-8.5 transfer, retained with only 20% of the training fraction, is a genuine cross-climate generalization test and is presented with a clean 50-trial protocol. The paper also deserves credit for flagging its own main caveats (constant sea level in Section VI; ad-hoc threshold mechanism in Section VII), and the Ens-CGP operator in Eq. (8) is consistent with reduced-rank ridge regression. The active-learning contribution, if substantiated on independent evaluation, would be significant: locating rare destructive storms while simulating only about 20% of a catalog would directly reduce the cost of climate hazard screening. However, the headline 100%-precision claim currently rests on an in-pool evaluation without reported trial variance, without a held-out or future-climate test of the active sampler, and at an optimized operating point; its significance is therefore conditional on the additional evaluations requested below. If those evaluations support the claim, this would be a strong contribution to applied climate risk assessment.","major_comments":[{"comment":"The active-learning evaluation has no held-out test set. In Experiments IVa and IVb the surrogate labels the remaining unsimulated pool, the most destructive predictions are simulated and appended to the training set, and the reported precision and recall are computed on this self-selected, shrinking pool; no random subset of ERA5 is reserved, and the SSP5-8.5 catalog is never presented to the active sampler. Because each selected storm's label comes from the hydrodynamic simulation, the metric is not circular by construction, but the evaluation set is the algorithm's own evolving selection, so the reported precision has an unknown relationship to performance on an untouched catalog. Consequently, the abstract's claim that the informative sampling approach is 'generalizable to climate scenarios' is unsupported: generalization is demonstrated only for the batch-trained XGBoost of Experiments II and III, and even that claim is subject to the paper's own Section VI caveats (constant sea level, embedded SSP5-8.5 characteristics). The authors should hold out a stratified sample of ERA5 or run the trained sampler on the SSP5-8.5 catalog and report precision and recall on that untouched set, and should temper the generalization statement in the abstract until such evidence exists.","section":"§V.A–V.B, Figs. 7–8; Abstract"},{"comment":"No repetition count or spread is reported for the Ens-CGP active-learning experiment that produces the headline result. Experiments I and III state 50 trials and Experiment IVa states 10 trials, but Experiment IVb reports none, and Fig. 8 shows no error bars or seed-to-seed variability. Since the 1% seed set is drawn randomly and the early iterations of the loop depend on it, a single favorable run could produce the reported 100% precision. The authors should run the Ens-CGP sampler over multiple seeds (comparable to the 50-trial batch protocol), report the distribution of precision, recall, and accuracy versus evaluation fraction, and state how many of the trials reached 100% precision at the 20% point.","section":"§V.B, Fig. 8"},{"comment":"The abstract and Section I state, without qualification, that the active online learning system 'achieves 100% precision after evaluating only 20% of the dataset,' but Fig. 8 attributes 100% precision to the precision-tuned case, which is reached by varying the damage threshold, a mechanism Section VII itself describes as ad hoc. The threshold values used for the untuned, precision-tuned, and recall-tuned cases are not stated, and the recall achieved at the 20% evaluation point in the precision-tuned case is not quantified; the caption's 'maintaining progress toward total recall' is not backed by a number. The paper should state the exact operating points, report precision and recall jointly at 20% for each case, and qualify the abstract claim so that the tuned and untuned results are not conflated.","section":"§V.B.2, Fig. 8 caption, §VII"},{"comment":"The rarity of the target class is never quantified: the paper nowhere states the fraction of the 4100 ERA5 TCs whose maximum simulated storm tide exceeds 3 m, despite calling them 'rare destructive storms,' and neither figure includes a random-query baseline at matched simulation budget. The 'naive' curve in Fig. 7 is not such a baseline: it trains batch models on increasing random fractions and tests on the shrinking complement, so training-set size and test-set composition change together. Without the catalog base rate and a random-sampling active baseline, the improvement attributable to informative selection cannot be separated from the base rate, and the central efficiency claim ('100% precision ... using less than 20% of the simulations') lacks a quantitative benchmark.","section":"§IV–V, Figs. 7–8"}],"minor_comments":[{"comment":"Experiment IVa is described in §V.A as selecting the top 5% of TCs by predicted destructiveness, but §VI and the Conclusions describe the XGBoost selection as randomly sampling TCs predicted to exceed the threshold; these two descriptions of the same protocol should be reconciled.","section":"§V.A vs. §VI–VII"},{"comment":"The ridge parameter σ² is said only to be 'chosen empirically'; unlike the exhaustive XGBoost hyperparameter list in §IV, no value or sensitivity analysis is provided for this parameter, which directly controls the Ens-CGP operator in Eq. (8).","section":"§V.B.1, Eq. (8)"},{"comment":"The reported catalog sizes (4100 and 2000 tracks) are counts before applying the exclusion of TCs that never reach land or that make two or more landfalls; the effective numbers of tracks actually used in each experiment should be stated.","section":"§III.C"},{"comment":"The paper surveys adaptive Kriging surrogate models (refs. [16], [18], [47]) but gives no comparison against them on the same catalog; a standard adaptive Kriging or other published active-surrogate baseline would contextualize the claimed Ens-CGP gains.","section":"§II"},{"comment":"Minor typos: 'adaptive obswervations' in §II should be 'adaptive observations,' and the sentence beginning 'the training iteration is updated (j ← j + 1) after simulating the top 1% ...' in §V.B.2 is grammatically incomplete and should be rewritten.","section":"§II, §V.B.2"},{"comment":"Experiment II (ERA5 to SSP5-8.5) is reported as a single run with no trial count while Experiments I and III use 50 trials; a multi-seed estimate for Experiment II would make the cross-scenario comparison more robust.","section":"§IV.B, Fig. 4"},{"comment":"The paper provides no data or code availability statement; given that the ADCIRC outputs and downscaled track catalogs are the basis of all experiments, a reproducibility statement would be expected.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The manuscript relies heavily on the authors' own prior work for the surrogate method (Ens-CGP is cited to refs. [23], [32], [39], [43], [45], all from the same group, and the downscaling/hydrodynamic data follow refs. [7] and [30]). This is acceptable for an incremental methods paper, but the novelty boundary relative to the informative-learning papers refs. [33] and [43] should be drawn more explicitly so that the specific contribution (destructiveness-based query criterion for storm-tide extremes) is clear. The paper fits the journal's applied-climate scope. My main editorial concern is that the abstract presents the headline 100%-precision claim more strongly than the current evidence supports; the requested supplementary evaluations (multi-seed trials, held-out and SSP5-8.5 evaluation of the active learner, stated operating points, random baseline) should be completed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper does something genuinely useful—shows that a surrogate (XGBoost, Ens-CGP) can rank downscaled TCs by destructiveness and actively select which few to simulate, cutting the number of expensive ADCIRC runs needed to find the damaging storms. The batch-transfer results (ERA5 to SSP5-8.5) are clean and credible: precision 80-82% with 20% training data, repeated over 50 trials. That part is a solid, if modest, advance.\n\nThe genuinely new bit is the informative sampling loop: selecting the top predicted-destructive TCs, simulating those, retraining. The Ens-CGP variant with threshold tuning is a reasonable extension of active learning to a rare-event retrieval problem, and the authors correctly cite prior GP/Kriging surrogates and active learning literature. I don't see an obvious circularity in the algorithm design—querying on predicted destructiveness and validating with true hydrodynamic output is legitimate.\n\nThe soft spot is the evaluation of the active learner. The 100% precision claim (Figure 8, Experiments IVb) is computed on the pool of unsimulated TCs that the sampler itself is filtering, with selected storms immediately added to training. That measures precision on a self-selected, shrinking distribution, not on a fresh catalog. There is no held-out subset of ERA5 and no SSP5-8.5 test for the active loop, so the 'generalizable to climate scenarios' claim rests on the batch experiments only. The Ens-CGP trials also appear to lack stated repetition counts and variance, unlike the XGBoost batch trials, which had 50 repeats. And the recall at the 20% point isn't reported, so the practical value of the precision claim is ambiguous. These are fixable with a proper evaluation protocol—hold out a random subset of TCs and never let the active learner see their hydrodynamic outputs, plus report trial-to-trial variability.\n\nAlso minor: the paper leans on the authors' own Ens-CGP and downscaling work, but the cited results are used as methods, not as substitutes for evidence here. That's not a flaw.\n\nWho this is for: researchers working on surrogate-based storm surge or flood hazard assessment, especially in data-scarce coastal settings. It's worth a serious referee pass because the method is plausible and the batch evidence is solid, but the strong claims need verification. I'd recommend sending it to review with a request for a held-out evaluation of the active sampler and reproducible artifacts (code/data).","headline":"A useful active-learning extension for storm tide extremes, but the headline 100% precision is measured on the self-selected pool and needs a held-out evaluation before it carries weight.","tokens_in":13109,"tokens_out":1897,"would_cite":false,"duration_ms":18002,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By iteratively retraining a surrogate on the most destructive-looking unsimulated cyclones, the paper claims, a search can pull every retrieved storm from the rare set of storm-tide-producing cyclones using fewer than a fifth of the full…","keywords":["storm tide extremes","tropical cyclones","active learning","surrogate modeling","ensemble-approximated conditional Gaussian process","climate change","Bangladesh","extreme value sampling"],"falsifier":"Run the same precision-tuned Ens-CGP active sampler on a fully simulated, independently downscaled storm catalog that was not used in any way to train or select the data, and check whether 100% precision is reached before 20% of the new catalog has been simulated; if precision drops substantially on the unseen storms, the central claim is refuted.","tokens_in":12143,"feed_emoji":"🌊","tokens_out":9177,"duration_ms":76135,"temperature":0.7,"pith_summary":"The paper claims that a surrogate model can replace most of the expensive hydrodynamic simulations needed to find the rare tropical cyclones that cause destructive storm tides. It shows that an active, iterative learning loop—rank unsimulated storms by predicted destructiveness, simulate the top few, retrain the surrogate—can retrieve these rare storms with 100% precision while evaluating under 20% of a large catalog of downscaled cyclones. The paper further claims that the surrogate generalizes from storms downscaled from present-climate reanalysis to storms from a high-emissions future climate scenario, retaining roughly 80% precision. If these claims hold, screening very large synthetic storm catalogs for climate risk assessment becomes computationally tractable.","feed_headline":"Surrogate sampling hits 100% precision for storm tide extremes","feed_subtitle":"An active-learning loop finds damaging cyclones in huge catalogs using under 20% of the simulations.","key_machinery":"The central object is the ensemble-approximated conditional Gaussian process (Ens-CGP), a surrogate that maps a 165-dimensional vector of cyclone parameters (latitude, longitude, maximum wind speed, minimum sea-level pressure, and radius of maximum wind at 33 hourly steps around landfall) to a 54-element vector of peak storm tides at stations along the Bangladesh coast. The model is the linear estimator $M = C_{yx}(C_{xx} + \\sigma^2 I)^{-1}$, computed cheaply with a reduced-rank singular-value decomposition of the training ensemble. In the informative sampling loop, the surrogate predicts storm tides for all unsimulated cyclones, ranks them by destructiveness—the number of stations where the predicted tide exceeds the 3 meter threshold—and the top 1% are simulated and added to the training set before the surrogate is refit. Adjusting the threshold biases the loop toward precision or recall.","core_discovery":"The central claim is that an ensemble-approximated conditional Gaussian process (Ens-CGP), retrained online on the most destructive unsimulated cyclones, identifies tropical cyclones that generate storm tides above 3 meters at any of 54 Bangladesh coastal stations with 100% precision after evaluating about 20% of the catalog. In the precision-tuned configuration, perfect precision is reached early in the iterative process while recall continues to climb toward total recall, and a recall-tuned configuration achieves total recall. The paper also reports that a batch-trained surrogate using only 20% of present-climate storm data reaches about 80% precision when tested on future-climate storms, supporting the generalization claim. All results treat water levels relative to mean sea level and do not include sea-level rise.","pith_inferences":["A stronger test than the paper's residual-pool evaluation would hold out an entire climate-model ensemble, run the active loop only on the rest, and score on the held-out storms; this would separate genuine generalization from self-confirmation inside the selection loop.","The destructiveness score could be replaced by expected economic damage or by exceedance of a high quantile of the tide distribution, which would directly optimize what risk managers care about rather than an arbitrary 3-meter cut.","Adding sea-level rise as an input parameter or post-processed offset would test whether the perfect-precision claim survives under the higher baseline water levels expected in the future climate.","The fixed 54-station output vector could be extended to continuous coastline maps, letting the sampler choose which locations are informative rather than assuming all stations are equally important."],"forward_implications":["Large synthetic storm catalogs can be screened for extreme storm tides with only about a fifth of the full hydrodynamic simulations, making tail-risk estimates feasible for regions with sparse historical storm records.","A surrogate trained on present-climate storms carries its skill to future-climate scenarios, so the active search can be run on downscaled storm sets from different climate models and emission scenarios.","The threshold knob lets users trade precision for recall, so the same machinery can serve both conservative risk screening and exhaustive hazard discovery.","Because the surrogate replaces storm-tide simulations, the active loop can be redeployed to other hazard variables, such as inundation depth from rainfall, without changing the learning framework."],"supporting_citations":[{"why":"Defines the Ens-CGP surrogate model that carries the informative sampling loop.","marker":"[39]"},{"why":"Provides the Bangladesh ADCIRC hydrodynamic setup, the 54 coastal stations, and the 3-meter damage threshold.","marker":"[30]"},{"why":"Supplies the downscaled TC climate and the Bangladesh storm set used as training and test data.","marker":"[7]"},{"why":"Establishes the statistical-deterministic downscaling method that generates the synthetic cyclone catalogs.","marker":"[9]"},{"why":"Provides the ERA5 reanalysis used to downscale present-climate storms for training.","marker":"[13]"},{"why":"Provides the EC-EARTH-3 climate model output used to produce future-climate storms for testing.","marker":"[6]"},{"why":"Defines the SSP5-8.5 high-emissions scenario used to generate the future-climate test set.","marker":"[27]"},{"why":"Supplies the XGBoost surrogate used in the batch experiments and the first active-learning variant.","marker":"[2]"}],"fun_headline_variants":["Active learning finds storm tide extremes with 100% precision using 20% of simulations","Surrogate search spots destructive cyclones with perfect precision early","Less than 20% of simulations yield 100% precision for storm tide extremes","Online learning pinpoints damaging cyclones from minimal simulations","Generalizable active learning achieves 100% precision on storm tides"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that precision computed on the shrinking pool of unsimulated cyclones, after the surrogate has been retrained on the very storms it selected as most destructive, is a faithful measure of how the search would perform on a fresh, independent catalog of storms.","fun_headline_variants_meta":{"raw":{"variants":["Active learning finds storm tide extremes with 100% precision using 20% of simulations","Surrogate search spots destructive cyclones with perfect precision early","Less than 20% of simulations yield 100% precision for storm tide extremes","Online learning pinpoints damaging cyclones from minimal simulations","Generalizable active learning achieves 100% precision on storm tides"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000167,"raw_usage":{"total_tokens":1215,"prompt_tokens":860,"completion_tokens":355,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":263}},"tokens_in":476,"tokens_out":355,"duration_ms":3922,"temperature":1.0,"reasoning_tokens":263,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:58:34.305131+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same precision-tuned Ens-CGP active sampler on a fully simulated, independently downscaled storm catalog that was not used in any way to train or select the data, and check whether 100% precision is reached before 20% of the new catalog has been simulated; if precision drops substantially on the unseen storms, the central claim is refuted.","supporting_citations":[{"cited_title":"Tractable Non-Gaussian Representations in Dynamic Data Driven Coherent Fluid Mapping , pages 29–46","cited_arxiv_id":null,"evidence_quote":"Defines the Ens-CGP surrogate model that carries the informative sampling loop."},{"cited_title":"Morss, Kerry A","cited_arxiv_id":null,"evidence_quote":"Provides the Bangladesh ADCIRC hydrodynamic setup, the 54 coastal stations, and the 3-meter damage threshold."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the downscaled TC climate and the Bangladesh storm set used as training and test data."},{"cited_title":"Sequential design of experiments","cited_arxiv_id":null,"evidence_quote":"Establishes the statistical-deterministic downscaling method that generates the synthetic cyclone catalogs."},{"cited_title":"Tropical cyclone risk in bangladesh","cited_arxiv_id":null,"evidence_quote":"Provides the ERA5 reanalysis used to downscale present-climate storms for training."},{"cited_title":"From these predictions, TCs exceeding the dam- age threshold (e.g., > 3 meters) are ranked based on their destructiveness","cited_arxiv_id":null,"evidence_quote":"Provides the EC-EARTH-3 climate model output used to produce future-climate storms for testing."},{"cited_title":"Adcirc: an advanced three- dimensional circulation model for shelves, coasts, and estuaries","cited_arxiv_id":null,"evidence_quote":"Defines the SSP5-8.5 high-emissions scenario used to generate the future-climate test set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the XGBoost surrogate used in the batch experiments and the first active-learning variant."}],"review_version":1}