{"id":"99335295-38cf-4289-945b-b8615e33d3e4","arxiv_id":"2411.10108","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"STCO-FS combines spatial clustering with evolutionary feature selection to identify region-specific heatwave drivers and their relevant time lags from ERA5 data.","lead":"This paper introduces STCO-FS, a framework that clusters climate variables and uses an evolutionary algorithm to select which variables and time lags detect heatwave days in the Adda river basin, Italy. The selected drivers include short-term local variables and sub-seasonal teleconnection indices, and the best classifier reaches an F1 score of about 0.79 on a held-out period.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set threshold selection invalidates reported driver set and F1; the paper's central 'key drivers' claim currently rests on an optimistic evaluation.","rationale":"The reader's weakest assumption focused on the conceptual leap from predictive skill to physical driver importance. My stress-test identifies a more concrete, load-bearing methodological flaw that compounds that conceptual issue: the agreement threshold is chosen by looking at test F1, and the same test set is then used to report final model performance. This makes the headline F1 of 0.7906 and the selected 146-feature driver set optimistic estimates, and it means the paper's central evidence for the framework's success does not currently come from a valid out-of-sample evaluation. The paper's self-admitted selection of DOY and CO2 with arbitrary lags reinforces that the selected variables are not necessarily physical drivers, but the threshold leakage is the decisive technical problem because it affects both the quantitative result and the feature set. The issue is correctable with a nested holdout design, so the appropriate verdict remains CONDITIONAL rather than REJECT: the framework is coherent and the code is available, but the current demonstration needs a clean validation before the central claims can be accepted.","tokens_in":15294,"tokens_out":6783,"duration_ms":75997,"concrete_test":"Perform a strictly nested evaluation: on 1951-2009, for each threshold compute LR CV F1 (inner 5-fold), pick the threshold with maximum CV F1, freeze the feature set, train GB, and evaluate exactly once on the untouched 2010-2020 test set. Also run the same outer protocol with a baseline LR containing only DOY, CO2, and local T2M. If the CV-chosen threshold is not 0.85, or the held-out GB F1 is materially below 0.7906, or the simple baseline matches it, then the reported driver set and F1 are leakage-inflated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.1 is not a clean validation. The authors use the training period to rank the 150,000 candidate solutions by 5-fold CV, but the final feature set is chosen by sweeping an agreement threshold and selecting 0.85 because it maximizes the test-set F1 in Table 3 and Figure 7. All classifiers in Table 4 are then evaluated on that same 2010-2020 test set, so the reported GB F1 of 0.7906 is optimistically biased and the 146-feature driver set is partly fit to the test data. This is the central evidence for the framework's skill and for the claimed key drivers. The paper's own observation that DOY and CO2 are widely selected with arbitrary lags (Section 4.1) further shows that F1-optimized selection does not by itself identify physical drivers. Thus the headline claim is not currently supported by a clean out-of-sample evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents STCO-FS, a two-stage framework for identifying short-term heatwave drivers. Meteorological fields from ERA5 are first reduced in spatial dimension via K-means clustering (k=5 per field), and the resulting 71 candidate driver time series (clusters plus local variables and climate indices) are then subjected to a wrapper feature selection driven by the PCRO-SL evolutionary algorithm, where the fitness is the 5-fold cross-validated F1 of a logistic regression for daily heatwave occurrence over the Adda river basin. The final driver set is obtained by keeping time steps selected in at least 85% of the best 10% of 150,000 candidate solutions, and this set is then used to train ten ML classifiers. The best test F1 is 0.7906 (gradient boosting). The paper reports that regional 2m temperature, local soil moisture, precipitation, Z500 patterns, NAO, IOD, and tropical Pacific SST are key short-term and sub-seasonal-to-seasonal drivers.","tokens_in":15522,"tokens_out":2282,"duration_ms":24745,"significance":"If the central evaluation were clean, the contribution would be useful: the framework is modular, code is promised (GitHub link), and it combines spatial clustering with evolutionary temporal-lag selection, which is a sensible way to reduce a large spatio-temporal search space. The frequency-map analysis over 150,000 solutions is a nice attempt to move beyond a single optimization run. However, the claimed skill and the identified driver list rest on an evaluation that is not fully out-of-sample: the agreement threshold is chosen by inspecting test-set F1. In addition, no baseline or ablation is provided, so the reported F1 cannot be attributed to the framework's driver identification rather than to the simple presence of contemporaneous local temperature and calendar-day features. The paper acknowledges that DOY and CO2 are widely selected with arbitrary lags, which independently confirms that F1-maximizing selection does not by itself yield physically interpretable drivers. The central claim, as stated in the abstract and conclusions, is therefore not yet supported by the evidence as presented.","major_comments":[{"comment":"The final driver set is chosen by sweeping the agreement threshold and selecting 0.85 because it maximizes the test-set F1 (Table 3, Figure 7). This makes the reported test F1 of 0.7615 for the logistic regression, and by extension the feature set used in Section 4.2, optimistically biased: the test set has been used for model selection, not only for evaluation. The authors state in Section 4 that the test set is \"reserved exclusively for evaluating model performance,\" which is contradicted by this threshold selection. A clean evaluation would require selecting the threshold inside cross-validation on the training period, or reporting both the training-selected threshold and the resulting test F1, with the threshold fixed before any test-set inspection.","section":"Section 4.1, Table 3 and Figure 7"},{"comment":"The headline test F1 of 0.7906 is reported without any baseline or ablation, and the test period has a very different positive-class rate (15.1%) from the training period (5.1%). Without a climatological or persistence baseline, a no-driver model that always predicts the majority class, or a model using only local T2M and day-of-year, the reader cannot tell whether the framework's driver-selection procedure adds skill. An ablation (e.g., using all 71 variables with no selection, random feature subsets, or the 0.5-threshold set) would be needed to support the claim that STCO-FS identifies skilful drivers.","section":"Section 4.2, Table 4; Section 2.2"},{"comment":"The framework defines 'driver importance' operationally as the contribution to logistic-regression F1 under 5-fold CV. The paper's own observation that DOY and global CO2 are widely selected, but that 'the selection of a specific lag time for each is considered arbitrary,' shows that F1-optimized selection does not by itself separate physically meaningful drivers from seasonal or trend confounders. The conclusions nevertheless state that the framework 'identifies key heatwave drivers' and 'important time frames.' This is an overinterpretation: the output is a set of statistically useful predictors for a nowcasting classifier, not established causal drivers. The text should either restrict its claims to 'predictive features' or provide an explicit discussion of why F1-based selection is expected to track physical driver importance.","section":"Section 3.2 and Section 4.1"},{"comment":"The frequency map and threshold analysis are based on the best 10% of 150,000 solutions, but the paper does not report any uncertainty across the ten independent PCRO-SL runs. For instance, it is unclear whether the 0.85-threshold feature set is stable across runs or dominated by a single run's best solution. Reporting the distribution of F1 across runs, and the overlap of the selected features across runs, would substantiate the claim of robustness and guard against overfitting to the stochastic optimization path.","section":"Section 4.1, Figure 6 and Figure 8"}],"minor_comments":[{"comment":"The abstract says 'key immediate (short-term) HW drivers' while the method is a detection/nowcasting exercise with h=0; the conclusions later mention 'nowcasting error metrics of 0.8363,' but the test F1 in Table 4 is 0.7906. Please clarify which metric is being reported and reconcile the numbers.","section":"Abstract / Conclusions"},{"comment":"The preprocessing states that anomalies are computed by removing the local seasonal cycle, but the day-of-year variable is still included as a candidate driver. Since HW occurrence is defined only for May-August and has a strong seasonal cycle, DOY can easily become a high-F1 predictor. Please discuss why DOY is not treated as a confounder to be removed or controlled rather than a candidate 'driver'.","section":"Section 2.2"},{"comment":"The choice k=5 for K-means is acknowledged as arbitrary, but the sensitivity of the final driver set and the test F1 to k is not examined. A sentence noting that k is a free parameter to be tuned per application is not sufficient; at least one sensitivity experiment (e.g., k=3 and k=7) would help.","section":"Section 3.1"},{"comment":"Figures 4 and 5 are not fully self-explanatory: Figure 4 shows one 'potential solution' but the color scale for blue/red is not defined in the caption in a way that distinguishes selected/unselected time steps, and Figure 5 does not state whether the red points are the best 10% by CV F1 or by some other criterion. Please expand the captions.","section":"Figure 4 and Figure 5"},{"comment":"Reference [9] is used for the heatwave definition, but the precise percentile threshold and duration criterion used to construct the binary target from the cumulative normalized TX exceedances are not given in the text. Please specify these details, as the target definition directly affects all reported F1 values.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper has a genuine methodological contribution in structure, but the evaluation protocol is the main obstacle: the test set has been used to choose the agreement threshold, and the paper's own discussion of DOY/CO2 undercuts the physical-driver interpretation. The authors should be asked to re-run the threshold selection inside cross-validation, add baselines/ablations, and soften the causal language. If the revised evaluation still shows strong out-of-sample skill, the paper could become acceptable for this journal. There is also a citation-pattern concern: several references appear generic or placeholder-like (e.g., PACS/MSC codes '0000, 1111'), and the ERA5 reference is cited as 'ERA5 hourly data... from 1979 to present' although the paper uses data from 1950."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The method is genuinely new: combining K-means spatial clustering with an evolutionary wrapper that optimizes lag and window per variable, then using a consensus frequency map across runs to stabilize selection. That integration is not in the cited literature, and the paper describes the pipeline clearly, releases code, and is upfront about several limitations. For those reasons it deserves a serious look.\n\nThe soft spots are real and central. The agreement threshold (0.85) is chosen by maximizing F1 on the test set (Table 3, Figure 7), and the same test set is then used to report all classifier scores in Table 4. That is test-set leakage, and it means the headline GB F1 of 0.7906 is optimistically biased. A proper evaluation would hold out a validation set for threshold selection or use nested CV. Without that, the 146-feature driver set is partly fit to the test period.\n\nThere are also no baselines or ablations, so we do not know whether this complex pipeline beats, say, a simple logistic regression on local temperature with a fixed lag, or whether the evolutionary search adds value over standard feature selection. The train/test positive-class rates differ a lot (5.1% vs 15.1%), which further clouds the comparison. And the conceptual step from predictive features to physical drivers is overclaimed: day-of-year and CO2 are widely selected with arbitrary lags, as the authors themselves note. That is seasonality and trend, not a mechanism. The paper would be stronger if it framed the output as candidate predictors for hypothesis generation, not confirmed drivers.\n\nWhat is good: the consensus analysis across 10 runs and 150,000 solutions is a thoughtful way to reduce noise from a stochastic optimizer, and the authors are honest about the limits of their physical interpretation. The framework is modular and could be ported to other extremes.\n\nWho is this for? Researchers working on feature selection for climate extremes, especially those who want a template for combining spatial reduction with temporal lag optimization. The current version is not a clean validation of the method's skill or of the specific drivers, but the flaws are correctable.\n\nRecommendation: send to peer review, but expect a major revision with a proper validation scheme, baselines, and a more careful interpretation of what 'driver' means here.","headline":"Interesting feature-selection framework, but the central evaluation is compromised by tuning the agreement threshold on the test set, so take the reported F1 and the 'key drivers' list with a grain of salt.","tokens_in":16077,"tokens_out":2519,"would_cite":false,"duration_ms":26364,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Clustering plus evolution identifies heatwave drivers at 0.79 F1.","keywords":["heatwaves","driver identification","spatio-temporal feature selection","clustering","evolutionary algorithms","teleconnections","Adda river basin","extreme event detection"],"falsifier":"Shuffle the heatwave labels in time and rerun the full STCO-FS pipeline; if similar driver sets with comparable F1 scores still emerge, the method is tracking seasonality or trends rather than event-specific precursors, and the driver-identification claim collapses.","tokens_in":15151,"feed_emoji":"🌡️","tokens_out":9986,"duration_ms":90360,"temperature":0.7,"pith_summary":"This paper proposes a general framework, STCO-FS, for discovering which climate variables drive extreme events such as heatwaves, and demonstrates it on the agricultural Adda river basin in northern Italy. The framework reduces the spatial dimension of gridded climate data by clustering nodes with similar behavior, then uses an ensemble evolutionary algorithm to select, for each candidate driver, the time lag and averaging window that best separate heatwave from non-heatwave days. On the Adda basin, the selected drivers split naturally into short-term local conditions (regional 2m temperature, soil moisture, precipitation) and sub-seasonal-to-seasonal teleconnections (NAO, IOD, tropical Pacific SST). The best classifier, a gradient-boosting model, reaches a test F1 score of 0.7906, and the authors argue that the same pipeline can be transferred to other regions and other types of extremes.","feed_headline":"Clustering plus evolution identifies heatwave drivers at 0.79 F1","feed_subtitle":"The framework clusters climate fields and evolves lagged feature sets, reaching 0.79 test F1 on heatwave days.","key_machinery":"The central machinery is a two-stage spatio-temporal feature-selection pipeline. First, K-means clustering groups the nodes of each ERA5 variable field into five regions of similar temporal behavior, reducing 12 gridded variables to 60 cluster-mean time series; adding 11 unclustered local variables and climate indices gives 71 candidate drivers. Second, the PCRO-SL ensemble evolutionary algorithm encodes each candidate solution as, for every driver, a time lag (0-180 days), a sequence length (1-60 days), and a binary include/discard flag. The fitness of a solution is the five-fold cross-validated F1 score of a fast logistic-regression nowcast of heatwave days, so the search favors compact lagged feature sets. Robustness is enforced by running the optimizer ten times, keeping the best 10% of all 150,000 solutions, and retaining only time-lag steps that appear in at least 85% of those good solutions.","core_discovery":"On its own terms, the paper discovers that the key short-term heatwave drivers for the Adda river basin are: regional 2m temperature over the cluster containing the basin, local soil moisture and precipitation with lags up to about 20 days; sub-seasonal predictors including OLR over the western-central Pacific (20-30 days), Z500 over the eastern Mediterranean (30-50 days), NAO and IOD (20-55 days); and seasonal predictors including Z500 over the North Atlantic (70-85 days) and tropical Pacific SST (90-100 days). The paper claims these features are robust because they appear in nearly all of the best-performing solutions out of 150,000 evaluated, and that the final feature set, selected at an 85% agreement threshold, yields a nowcasting F1 of 0.7906 with a gradient-boosting classifier. It presents STCO-FS as a general method for extreme-event driver detection, not a one-off analysis of this basin.","pith_inferences":["Because 'driver' is defined operationally by nowcast F1, the selected set may include variables that track seasonality or warming trends rather than cause heatwaves; re-running the search with a causal objective would tell which of the identified features are genuinely mechanistic.","The agreement threshold that fixes the final feature set is tuned on the test period (Table 3), so the reported 0.7906 F1 is likely optimistic; a fully nested validation with the threshold chosen inside cross-validation would give a fairer estimate of out-of-sample skill.","The framework's output is a hypothesis generator: the specific Z500 clusters and NAO lags it selects point to precursor wave-train configurations that could be tested in dynamical model experiments.","A transfer test would strengthen the generality claim: applying STCO-FS to a different basin or a different extreme and checking whether the selected driver geography matches known physical teleconnections would show whether the method discovers mechanisms or only region-specific correlations."],"forward_implications":["The identified short-term drivers (regional temperature, soil moisture, precipitation) give a concrete input list for heatwave nowcast and early-warning systems in the Po Valley region.","The sub-seasonal and seasonal drivers (NAO, IOD, tropical Pacific SST, North Atlantic Z500) specify lead times at which remote conditions are informative, which can guide the design of sub-seasonal-to-seasonal forecast models.","Because the pipeline is modular with respect to target and predictor data, the same two-stage search can be applied to other extreme events (e.g., droughts, heavy rainfall) to generate candidate driver sets for those hazards.","The framework's frequency-based threshold provides a principled way to separate consistently selected drivers from noise, so users can trade off the size of the feature set against detection skill."],"supporting_citations":[{"why":"Provides the ERA5 reanalysis fields from which all gridded predictor variables and the target time series are extracted.","marker":"[34]"},{"why":"Supplies the heatwave definition used to build the binary target of heatwave days.","marker":"[9]"},{"why":"Defines PCRO-SL, the ensemble evolutionary optimizer that searches lag, window, and inclusion flags for each driver.","marker":"[35]"},{"why":"Supplies the K-means algorithm used to reduce each gridded variable into five spatially coherent clusters.","marker":"[54]"},{"why":"Provides the logistic-regression classifier whose cross-validated F1 is the fitness function during driver selection.","marker":"[60]"},{"why":"Supports the physical role of soil moisture and precipitation in European summer heat, validating the short-term local drivers selected.","marker":"[48]"},{"why":"Establishes the link between regional circulation anomalies (blocking/ridges) and heat extremes, which the selected Z500 clusters are interpreted against.","marker":"[22]"},{"why":"Documents the influence of NAO and atmospheric blocking on European heatwaves, supporting the NAO selection on sub-seasonal scales.","marker":"[43]"},{"why":"Links ENSO to summer heat extremes in northern continents, supporting the tropical Pacific SST selection at seasonal lead times.","marker":"[39]"}],"fun_headline_variants":["Clustering+evolution framework IDs heatwave drivers at 0.79 F1","Heatwave trigger detection: STCO-FS hits 0.79 F1","Cluster-evolve method finds heatwave drivers with 0.79 F1","New spatio-temporal tool identifies heatwave drivers, F1 0.79"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a variable that improves a binary classifier's F1 score for nowcasting heatwave days is genuinely a 'driver' of heatwaves; if predictive skill and physical causation come apart, the selected features could simply be seasonal markers or trend-correlated variables.","fun_headline_variants_meta":{"raw":{"variants":["Clustering+evolution framework IDs heatwave drivers at 0.79 F1","Heatwave trigger detection: STCO-FS hits 0.79 F1","Cluster-evolve method finds heatwave drivers with 0.79 F1","New spatio-temporal tool identifies heatwave drivers, F1 0.79"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001316,"raw_usage":{"total_tokens":5340,"prompt_tokens":905,"completion_tokens":4435,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":4348}},"tokens_in":521,"tokens_out":4435,"duration_ms":30922,"temperature":1.0,"reasoning_tokens":4348,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:57:35.862829+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Shuffle the heatwave labels in time and rerun the full STCO-FS pipeline; if similar driver sets with comparable F1 scores still emerge, the method is tracking seasonality or trends rather than event-specific precursors, and the driver-identification claim collapses.","supporting_citations":[{"cited_title":"Hersbach, B","cited_arxiv_id":null,"evidence_quote":"Provides the ERA5 reanalysis fields from which all gridded predictor variables and the target time series are extracted."},{"cited_title":"Russo, J","cited_arxiv_id":null,"evidence_quote":"Supplies the heatwave definition used to build the binary target of heatwave days."},{"cited_title":"P´ erez-Aracil, C","cited_arxiv_id":null,"evidence_quote":"Defines PCRO-SL, the ensemble evolutionary optimizer that searches lag, window, and inclusion flags for each driver."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the K-means algorithm used to reduce each gridded variable into five spatially coherent clusters."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the logistic-regression classifier whose cross-validated F1 is the fitness function during driver selection."},{"cited_title":"Stefanon, F","cited_arxiv_id":null,"evidence_quote":"Supports the physical role of soil moisture and precipitation in European summer heat, validating the short-term local drivers selected."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the link between regional circulation anomalies (blocking/ridges) and heat extremes, which the selected Z500 clusters are interpreted against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the influence of NAO and atmospheric blocking on European heatwaves, supporting the NAO selection on sub-seasonal scales."},{"cited_title":"Luo, N.-C","cited_arxiv_id":null,"evidence_quote":"Links ENSO to summer heat extremes in northern continents, supporting the tropical Pacific SST selection at seasonal lead times."}],"review_version":1}