{"id":"f6d4d8cf-48e0-490c-99b9-929d4916039c","arxiv_id":"2507.00845","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Adding echo top height to a 3D U-Net rainfall nowcaster improves detection of very light rain but worsens intensity bias and does not help heavier rain forecasts.","lead":"This study tested whether adding echo top height, a radar measurement of storm cloud height, improves deep learning rainfall nowcasts in the Netherlands. It found that the extra data helped only for very light rain and introduced a systematic underestimation of heavier rainfall.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The null result rests on a top-1000-event, single-year test set; high-threshold skill differences need bootstrap/stratified confirmation before concluding ETH does not help.","rationale":"The paper is an honest, carefully hedged empirical study, and the reader's CONDITIONAL verdict is appropriate. The main load-bearing concern is not internal inconsistency but the strength of the null claim relative to the evaluation sample. The authors deliberately select extreme events, restrict training to the post-2016 radar era, and test on a single year; this is a reasonable proof-of-concept design, but it limits how strongly one can state that ETH does not improve high-intensity nowcasting. The high-threshold metrics are especially fragile because the rarest categories have near-zero FSS, so small differences are hard to interpret without event-level uncertainty. The proposed check is feasible with the already-released models and data, and would either confirm the conclusion on a more representative sample or show that it is an artifact of the curated test set. Because the reader already conditioned the verdict on exactly this kind of dataset concern, no verdict change is needed; the value of the stress-test pass is to sharpen the condition into a concrete, runnable evaluation.","tokens_in":17908,"tokens_out":7652,"duration_ms":99453,"concrete_test":"Use the released code and KNMI data to evaluate the already-trained model pairs on the full set of 2022 observations (not just the top-1000 starts), and additionally stratify by event type (e.g., widespread vs convective using areal coverage and ETH distribution). Compute FSS/ETS with bootstrap confidence intervals over independent events rather than pixels. If ETH remains worse at ≥1 mm/h and more negatively biased in the full/stratified sample, the dataset-selection concern is resolved; if the ordering changes, the conclusion must be restricted to the current curated subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central null claim is conditional on a dataset construction (§4.1) that may not support the high-intensity sub-claim. Training uses only the top-1000 observations per year ranked by summed squared reflectivity (~1.5% of available data), is restricted to after the Oct-2016 radar upgrade, and is tested on 2022 only. The FSS matrices in Fig. 8 show near-zero scores at 5 and 10 mm/h for both configurations, so the ETS/FSS differences at high thresholds are estimated from very few positive pixels and carry no uncertainty quantification. If the selection overrepresents isolated convective cells and underrepresents widespread moderate-rain systems, the conclusion that ETH does not help at higher intensities could be a property of this curated sample rather than of ETH itself. The negative-bias result is likewise measured on this sample. This does not make the paper internally inconsistent—the authors hedge appropriately—but it makes the strongest negative claim less secure than the headline suggests.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether adding echo top height (ETH) as an auxiliary input channel to a deterministic 3D U-Net improves radar-based precipitation nowcasting in the Netherlands. The authors compare eight models trained with ETH and eight without, using pixel-wise metrics (MSE, MAE, ME), threshold-based metrics (precision, recall, ETS), the Fractions Skill Score (FSS), and three qualitative case studies. The central finding is that ETH improves skill only at the lowest rain-rate threshold (0.1 mm/h), does not improve high-intensity nowcasts, and introduces an additional negative bias, leading the authors to conclude that ETH does not provide convincing proof-of-concept evidence for nowcasting. The paper includes code links, animations, and a candid discussion of limitations.","tokens_in":18065,"tokens_out":4122,"duration_ms":47583,"significance":"If the result holds, it is a useful and reproducible negative result for the nowcasting community: it suggests that naively adding ETH as an input channel is insufficient, and it provides a benchmark for future work on auxiliary radar-derived variables. The paper deserves credit for training multiple models under different seeds and data splits, for reporting multi-scale FSS along with threshold metrics, and for making code and visualizations publicly available. The negative-bias finding and the case-specific behavior are also of operational interest. The main limitations, as discussed in the report, concern the dataset construction and the lack of uncertainty quantification, which weaken the strength of the null conclusion.","major_comments":[{"comment":"The dataset description is internally inconsistent about the training sample size: selecting the top-1000 observations per year for 2008–2022 gives 15,000 starting points, but the text then restricts training to data from October 2016 onward, which would leave roughly 6,000 starting points. Please clarify whether the 15,000 figure includes pre-2016 data and how the restriction was applied. This matters because the composition of the training set and the representativeness of the 2022-only test set are load-bearing for the paper's null claim.","section":"§4.1"},{"comment":"The central null claim rests on comparisons of MSE, MAE, ME, and ETS between the two model groups, but no statistical significance tests or confidence intervals are reported. Given that the paper concludes that ETH 'introduced an additional negative bias' and is 'detrimental' at higher intensities, the authors should provide paired bootstrap or permutation tests across the eight model pairs (or across events) for the key metrics. Without such tests, the reported mean differences—many of which are small relative to the standard deviations in Figures 5 and 7—cannot be distinguished from sampling noise.","section":"§5.1 and §5.2"},{"comment":"The FSS matrices in Figure 8 show essentially zero scores at thresholds of 5 and 10 mm/h for both configurations at all lead times. The statement that ETH does not improve spatial predictive skill for heavier rainfall is therefore based on a near-floor comparison in which the observed fields contain very few positive pixels; this does not robustly support the conclusion. Please either restrict the high-threshold FSS analysis to events with meaningful rain coverage, supplement it with a metric that remains interpretable at low base rates, or explicitly quantify the uncertainty on the zero and near-zero differences.","section":"§5.3"}],"minor_comments":[{"comment":"In the first paragraph of Section 5, 'Ta gain additional insight' should read 'To gain additional insight'.","section":"§5"},{"comment":"In the final paragraph of Section 3.1, 'it it is possible' should read 'it is possible'.","section":"§3.1"},{"comment":"In the first paragraph of Section 4, 'for it's simplicity' should read 'for its simplicity'.","section":"§4"},{"comment":"In the paragraph introducing the case studies, 'with a without ETH' should read 'with and without ETH', and 'ploted' should be 'plotted'.","section":"§5.4"},{"comment":"The green/red color scheme used for the two model groups is difficult to distinguish for color-blind readers; consider using more separable colors or line styles.","section":"Figures 5–7"}],"recommendation":"major_revision","confidential_remarks":"The paper is an honest, reproducible negative result with a reasonable experimental design, but the dataset-size inconsistency and the absence of uncertainty quantification on the key differences undermine the strength of the null conclusion. I believe these issues can be fixed within a major revision, so I do not recommend rejection. The paper fits the journal's scope as a critical evaluation of a proposed input variable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is an honest, well-structured empirical null result. The authors test whether adding echo top height (ETH) as an input channel improves a 3D U-Net nowcaster on Dutch radar data, and by their own metrics it doesn't—except at 0.1 mm/h. It also adds a negative bias. That's a useful result for the nowcasting community, and it fills a small but real gap: previous ETH work was on QPE and lightning, not on deep learning nowcasting.\n\nWhat's good: eight seeds per condition, held-out 2022 test set, multiple metrics (MSE/MAE/ME, precision/recall/ETS, FSS), open code and data links, and the discussion is appropriately hedged. The architecture description is clear, and the case studies are used as illustration, not as proof.\n\nThe soft spots are real but not fatal. The dataset is aggressively filtered: top 1000 observations per year by summed squared reflectivity (~1.5% of data), training only from October 2016 because of a radar upgrade, and testing only on 2022. That makes the high-intensity sub-claims fragile. At 5 and 10 mm/h, FSS is near zero for both configurations and you're working with very few positive pixels. There's no significance testing, and the FSS matrices have no uncertainty bounds. A bootstrap over events or a stratified analysis by event type would substantially strengthen the negative conclusion. The selection may also overrepresent isolated convective cells, so the result that ETH doesn't help at higher intensities is conditional on this sample. The authors don't overclaim—their conclusion is closer to 'no convincing proof-of-concept'—but the title asks a general question, and the data can't fully answer it.\n\nAlso minor: the code link has no commit hash or container, so reproducibility is good but not pinned.\n\nBottom line: worth sending to peer review. The central claim is broadly supported by the evidence they present, and the limitations are stated. A referee should ask for uncertainty quantification and a more stratified test evaluation, but this is a legitimate contribution that will save other groups wasted effort. I'd bring it to a reading group focused on negative results in applied ML.","headline":"A careful, well-scoped empirical null result on adding echo top height to a deterministic nowcaster; the negative conclusion is broadly supported, but the filtered single-year test set and missing uncertainty estimates keep it from being the last word.","tokens_in":18628,"tokens_out":2282,"would_cite":true,"duration_ms":26565,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding echo top height to radar nowcasts helps only for the lightest rain.","keywords":["precipitation nowcasting","echo top height","radar reflectivity","deep learning","3D U-Net","fractions skill score","Netherlands"],"falsifier":"Evaluate the same with- and without-ETH comparison on a multi-year test set that includes widespread moderate-rain events, or retrain on a sample not truncated to the top 1000 reflectivity frames per year; if ETH models then beat no-ETH models on Equitable Threat Score at 1 mm/h or 2.5 mm/h without a more negative mean error, the paper's negative verdict would be overturned.","tokens_in":17669,"feed_emoji":"🌧️","tokens_out":8179,"duration_ms":76041,"temperature":0.7,"pith_summary":"This paper asks whether adding echo top height (ETH) — the highest altitude at which radar detects precipitation — as an extra input channel improves deep-learning nowcasts of rainfall. It trains a 3D U-Net on Dutch radar data with and without ETH, comparing eight model instances per configuration. The answer is mostly no: ETH improves skill at the 0.1 mm/h threshold, but at 1 mm/h and above the scores are equal or worse, and the ETH models systematically underestimate rainfall. The paper concludes that ETH carries physically relevant information but, in this proof-of-concept setup, does not deliver convincing value for nowcasting, and it warns that auxiliary variables can add bias and variance rather than skill.","feed_headline":"Echo top heights help radar nowcasts only for very light rain","feed_subtitle":"Adding storm-top data cuts skill at higher rain rates and deepens the models' dry bias.","key_machinery":"The central object is echo top height (ETH), a two-dimensional field giving, at each grid point, the maximum altitude at which the radar detects reflectivity above a 7 dBZ threshold; it is a compact stand-in for the vertical extent of precipitation. The architecture that carries the argument is a single-pass 3D U-Net (an encoder–decoder convolutional network with skip connections) that treats time as the third dimension and takes radar reflectivity and ETH as separate input channels, predicting 18 future frames in one forward pass to avoid recursive bias accumulation. The evaluation machinery is a set of verification scores — MSE, MAE and mean error, precision/recall/ETS at four thresholds, and Fractions Skill Score at five thresholds and three spatial scales — chosen so that any genuine reduction in nowcast uncertainty from ETH should show up as less blurring and better spatial skill.","core_discovery":"The central claim is that, on this dataset and architecture, echo top height does not provide convincing proof-of-concept value for precipitation nowcasting. Models given ETH as a separate input channel achieve higher Fractions Skill Score and Equitable Threat Score only at the lowest rain-rate threshold (0.1 mm/h) and at short lead times; at thresholds of 1 mm/h and above, the two configurations score similarly, with the no-ETH models slightly ahead on ETS. The ETH models also show larger run-to-run variance and a more negative mean error, meaning they tend to predict less rain than observed. Three case studies show the effect is event-dependent: ETH helps when the storm has high echo tops, but hurts or confuses the model when rainfall is intense yet echo tops are low, and the added input can amplify error variance even when average error improves.","pith_inferences":["A natural next test is to stratify the test set by observed echo top height; the case studies suggest ETH helps in high-ETH convection, so a conditioned evaluation might show positive skill that the aggregate metrics hide.","The negative bias may be a side effect of the pointwise MSE loss combined with an extra input that correlates with but does not uniquely determine surface rain; an architecture that decouples intensity from occurrence, or a loss that penalizes underestimation asymmetrically, might change the balance.","Because the radar system changed in 2016 and the test year is a single year, the result is a snapshot of one country's radar regime; transfer to other climates or radar networks could plausibly reverse the verdict.","ETH artifacts from the radar sweep geometry (circular bands) are visible even in the cropped domain; cleaning these artifacts by masking unreliable pixels is a cheap preprocessing fix that could alter the comparison."],"forward_implications":["Adding ETH as an extra channel is technically feasible in a 3D U-Net and slightly improves detection of very light rain (0.1 mm/h), especially at short lead times.","At rain rates of 1 mm/h and above, ETH does not improve Fractions Skill Score or Equitable Threat Score; at higher thresholds the no-ETH models are equal or better.","ETH inputs systematically increase the negative bias (mean error), so flood- and warning-oriented applications would see more underestimated intensities.","The effect of ETH is case-dependent: it can preserve intensity and reduce blurring when echo tops are high, but can degrade forecasts when intense rain is accompanied by low echo tops.","A deterministic U-Net with a pointwise loss is a workable diagnostic benchmark for testing auxiliary variables, but the paper argues that generative or probabilistic approaches and better ETH preprocessing may be needed to unlock ETH's potential."],"supporting_citations":[{"why":"Supplies the U-Net baseline architecture that this study extends to 3D convolutions and additional input channels.","marker":"[1]"},{"why":"The Dutch echo-top dataset that provides the ETH input field studied throughout.","marker":"[9]"},{"why":"The Dutch reflectivity dataset used to build the rainfall inputs and targets.","marker":"[10]"},{"why":"The standard reflectivity-to-rain-rate conversion applied to both model inputs and verification targets.","marker":"[13]"},{"why":"Prior work showing 3D volumetric radar inputs can improve nowcasting, motivating the 3D architecture choice.","marker":"[16]"},{"why":"Defines the Fractions Skill Score used as the spatial verification metric in the central comparison.","marker":"[19]"},{"why":"The national data platform through which the radar and echo-top records are accessed.","marker":"[20]"},{"why":"Prior evidence that adding echo-top data improves machine-learning rainfall estimation, the main positive precedent the study tests.","marker":"[28]"}],"fun_headline_variants":["Echo top data only aids nowcasts for the lightest rain","Storm-top heights fail to boost deep learning rain nowcasts","Echo top heights add little to deep-learning nowcasts","Adding echo top heights worsens rain nowcast bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result assumes that the curated sample — roughly the top 1000 events per year by summed reflectivity, restricted to the post-2016 radar period, cropped to where 7 km echo tops are detectable, with 2022 as the only test year — fairly represents the range of rainfall intensities where ETH could matter.","fun_headline_variants_meta":{"raw":{"variants":["Echo top data only aids nowcasts for the lightest rain","Storm-top heights fail to boost deep learning rain nowcasts","Echo top heights add little to deep-learning nowcasts","Adding echo top heights worsens rain nowcast bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000713,"raw_usage":{"total_tokens":3196,"prompt_tokens":921,"completion_tokens":2275,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":2207}},"tokens_in":537,"tokens_out":2275,"duration_ms":19301,"temperature":1.0,"reasoning_tokens":2207,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:05:30.246841+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the same with- and without-ETH comparison on a multi-year test set that includes widespread moderate-rain events, or retrain on a sample not truncated to the top 1000 reflectivity frames per year; if ETH models then beat no-ETH models on Equitable Threat Score at 1 mm/h or 2.5 mm/h without a more negative mean error, the paper's negative verdict would be overturned.","supporting_citations":[{"cited_title":"0: a convolutional neural net- work for radar-based precipitation nowcasting","cited_arxiv_id":null,"evidence_quote":"Supplies the U-Net baseline architecture that this study extends to 3D convolutions and additional input channels."},{"cited_title":"https://dataplatform","cited_arxiv_id":null,"evidence_quote":"The Dutch echo-top dataset that provides the ETH input field studied throughout."},{"cited_title":"knmi.nl/dataset/radar-tar-refl-composites-1-0 (2024)","cited_arxiv_id":null,"evidence_quote":"The Dutch reflectivity dataset used to build the rainfall inputs and targets."},{"cited_title":"Jour- nal of Atmospheric Sciences 5(4), 165 – 166 (1948)","cited_arxiv_id":null,"evidence_quote":"The standard reflectivity-to-rain-rate conversion applied to both model inputs and verification targets."},{"cited_title":"In: CD- CEO@ IJCAI","cited_arxiv_id":null,"evidence_quote":"Prior work showing 3D volumetric radar inputs can improve nowcasting, motivating the 3D architecture choice."},{"cited_title":"Monthly Weather Review 136(1), 78–97 (2008)","cited_arxiv_id":null,"evidence_quote":"Defines the Fractions Skill Score used as the spatial verification metric in the central comparison."},{"cited_title":"https: //dataplatform.knmi.nl/ (2025), https://dataplatform.knmi.nl/, accessed: 2025- 04-22","cited_arxiv_id":null,"evidence_quote":"The national data platform through which the radar and echo-top records are accessed."},{"cited_title":"Advances in Atmospheric Sciences 40(6), 1043–1057 (2023)","cited_arxiv_id":null,"evidence_quote":"Prior evidence that adding echo-top data improves machine-learning rainfall estimation, the main positive precedent the study tests."}],"review_version":1}