{"id":"37d4d5ca-19f8-4d27-81dd-0e046c3ba3a3","arxiv_id":"2412.05825","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Masked-model pre-training plus probabilistic density labels improves rainfall post-processing CSI and heavy-rain detection on the Korean RDAPS test set.","lead":"This paper presents a machine-learning method that post-processes numerical weather prediction outputs to estimate rainfall probability, combining self-supervised pre-training with a soft-labeling scheme. It reports improved detection of heavy rain over six baselines on a South Korean regional test set.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SSLPDL's 'surpasses all baselines' claim rests on one unreplicated test month (Aug 2022) deliberately selected for heavy rainfall; the CSI10/mIoU gains may be noise and the reported 9.9% mIoU improvement is not the gain over the best baseline.","rationale":"The paper's contribution is useful and the pipeline is coherent, but the strongest empirical claim is not yet supported at the precision stated. The reader correctly identified the single-month, unreplicated test set as the weakest assumption; my independent reading agrees. Testing on August 2022 is reasonable for a heavy-rain regime, but it cannot establish a general claim of superiority over baselines across seasons and years, especially when rare-event CSI values are small and no uncertainty is reported. The mIoU arithmetic further weakens the abstract's wording: the improvement over the best baseline is about 5.9%, not 9.9%, unless the reference is an unspecified average. I also found a secondary, more technical issue: the probabilistic density labels in Eq. (3) and Algorithm 1 do not sum to 1 for rainfall values inside an interval, only at thresholds, despite the paper's assertion that they do. This matters for the method's theoretical framing but is not the primary reason the superiority claim is insecure. The conditional verdict is therefore appropriate; no verdict change is needed, but the requested re-evaluation should be a prerequisite for acceptance.","tokens_in":13109,"tokens_out":12551,"duration_ms":118195,"concrete_test":"Retrain SSLPDL and the strongest baseline (Swin-Unet or PostRainBench) under the identical protocol with at least 5 random seeds, and evaluate on every available month from March through November for 2022 and 2023 (or at least on June-September 2023, which is out of the training period). For each seed and month compute CSI10 and mIoU, then report the mean and 95% bootstrap confidence interval of the difference Ours - best baseline. If the lower bound of the CI for CSI10 or mIoU is ≤0, or if the sign flips in non-August months, the 'surpasses all baselines' claim is not supported. Additionally, recompute the 9.9% mIoU gain against the best baseline and state the reference used.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 4.3 (Table 3) is that SSLPDL outperforms all six baselines on RDAPS. For that claim to be reliable, the test evaluation must be representative and stable. It is neither: the test set is a single month, August 2022 (Section 4.1), chosen because over 50% of Korea's annual precipitation occurs in August, and the results are reported once with no random seeds, confidence intervals, or significance tests. Table 3 aggregates only six forecast hours (25-30 h). Heavy-rain CSI10 is a rare-event metric: 0.096 vs 0.060, so a shift of a few correctly detected cells in one month can flip the comparison. The abstract's 'over 9.9% mIoU improvement' is also not vs the best baseline: against Swin-Unet/PostRainBench at 0.389, the improvement is (0.412-0.389)/0.389 ≈ 5.9%; the 9.9% figure only roughly matches an average of the learned baselines, which is not what the text says. A secondary internal issue is that Eq. (3)/Algorithm 1's density labels do not sum to 1 except when the rainfall is exactly at a threshold, contradicting the claim that the probability sum is 1; this does not by itself overturn the empirical comparison but needs correction or justification. Without multi-season, multi-year, repeated-seed evidence, the headline superiority claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SSLPDL, a two-stage framework for post-processing numerical weather prediction (NWP) precipitation forecasts. In the first stage, a masked autoencoding-style pre-training task reconstructs masked atmospheric variables using an InternImage encoder with deformable convolutions, intended to capture dependencies among variables. In the second stage, the pre-trained encoder is transferred to a precipitation segmentation task with a UperNet decoder, and the loss combines one-hot labels with newly proposed probabilistic density labels. Experiments on the Korean RDAPS dataset compare SSLPDL with six baselines, reporting improvements in CSI and mIoU, including heavy-rain CSI10 of 0.096, and the paper claims extended forecast lead times. Code is made publicly available.","tokens_in":13473,"tokens_out":9493,"duration_ms":84977,"significance":"The topic is practically important: post-processing high-resolution regional NWP forecasts with self-supervised representation learning is a plausible route toward better rare-event precipitation detection, and the paper proposes a concrete, potentially reusable algorithmic combination of masked pre-training, deformable convolution, and density-based soft labels. The release of code and the use of a real operational NWP dataset are strengths. If the reported gains were confirmed on a broader evaluation, the method would be useful for operational forecasting. However, the current evidence base is too narrow to establish the headline claims: the main comparison rests on a single test month with no uncertainty quantification, the abstract's improvement figure is overstated relative to the best baseline, and the proposed density labels are not normalized as claimed.","major_comments":[{"comment":"The central claim that SSLPDL 'outperforms all the baselines on the RDAPS test set' is supported by results from a single test month, August 2022, with no repeated runs, confidence intervals, or significance tests. Heavy-rain CSI10 is a rare-event metric, and the difference between the reported 0.096 and the best baseline 0.060 can be sensitive to a small number of correctly detected cells in one month. Please provide multi-season and multi-year evaluation, or at minimum bootstrap confidence intervals and multiple seeded runs, before the superiority claim can be considered established.","section":"Section 4.1 and Table 3"},{"comment":"The abstract's claim of 'improving mIoU by over 9.9%' is not supported relative to the strongest baseline. In Table 3, the best baseline mIoU is 0.389 (Swin-Unet and PostRainBench), while SSLPDL reaches 0.412; the relative improvement is approximately (0.412-0.389)/0.389 = 5.9%. The 9.9% figure only roughly matches the gain against the average of all six baselines (average mIoU about 0.375). Please state the comparison basis explicitly and avoid the implication that SSLPDL improves on the best baseline by 9.9%.","section":"Abstract and Section 4.3"},{"comment":"The statement that 'the probability sum for each class equals 1' is incorrect except at threshold values. For a rainfall amount gamma in [tau_{i-1}, tau_i), the label y*_i equals (1-alpha)*(1 - (gamma-tau_{i-1})/(tau_i-tau_{i-1})) + alpha/N, while all other entries are alpha/N, so the total is (1-alpha)*(1 - (gamma-tau_{i-1})/(tau_i-tau_{i-1})) + alpha, which is less than 1 unless the density factor is exactly 1. Since the cross-entropy in Eq. (2) uses these values as targets, either normalize y* or revise the claim that the labels form a probability distribution; as written, the labeling scheme is mis-specified.","section":"Section 3.4, Eq. (3), Algorithm 1"},{"comment":"The text says that 'the larger the masking ratio in the pre-training, the better the results,' but the table does not show a monotonic trend: for pre-training, the 75% ratio gives CSI0.1 = 0.389, while the 90% ratio gives CSI0.1 = 0.356. Please correct either the narrative or the table so that the ablation result is reported accurately.","section":"Section 4.4, Table 4"},{"comment":"The text reports an overall 4.2% improvement over RDAPS and a 7.1% gain in July, but the data labels in Figure 5 appear to contradict this. If the first row of labels (48.0, 45.9, 48.1, 46.2, 48.3, 46.5, 47.9, 46.5, 41.9) is RDAPS and the second row (39.1, 41.2, 38.9, 47.1, 45.8, 47.3, 46.4, 48.5, 46.3) is SSLPDL, SSLPDL is lower in five of nine months and the overall average is lower, not 4.2% higher. If the order is reversed, the July gain is approximately 5.5%, not 7.1%. Please reconcile the data labels, the series order, and the reported percentages.","section":"Section 4.4, Figure 5"}],"minor_comments":[{"comment":"The phrase 'surpasses other precipitation forecasting models' is broader than what is demonstrated; the comparison is with post-processing baselines on the RDAPS test set, and the claims should be scoped accordingly.","section":"Abstract and Section 1"},{"comment":"The notation f_Theta, f_phi, and f_theta is confusing; the sentence 'involving reconstructing f_phi from the NWP forecasts to map the target object function f_theta' should be rewritten for clarity.","section":"Section 3.1"},{"comment":"The 'Labeling' column is unclear: probabilistic density labeling produces soft vectors, not discrete class assignments, so the meaning of 'data proportion after probabilistic density labeling' needs a definition, for example whether the percentages are sums of probability mass over the training set.","section":"Table 2"},{"comment":"The notation for the parenthetical deltas is ambiguous; please state explicitly that the plain number is the score without probabilistic density labeling and the parenthetical is the change, since rows such as PostRainBench CSI10 = .063 (▼.035) only cohere with Table 3 (where the labeled score is .028) under that reading.","section":"Table 5"},{"comment":"References [37] and [38] are the same paper (PostRainBench); please deduplicate the entry and cite it consistently.","section":"References"},{"comment":"The caption reports percentage improvements per lead time (5.2%, 8.2%, 5.0%, 8.4%, 9.9%, 4.9%), but the baseline for these percentages is not specified and they are not directly tied to values in Table 3; please clarify the calculation.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The core idea is reasonable and the code availability is commendable, but the empirical evaluation is currently too thin for the strength of the claims. The most important requested changes are a broader test period with uncertainty quantification, a corrected statement of the mIoU improvement, and a fix to the normalization claim in Eq. (3). These are revision-level issues rather than fundamental flaws, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this paper stitches together three off-the-shelf ideas — masked autoencoder pre-training, an InternImage deformable-conv backbone, and a soft labeling trick — and gets decent numbers on a Korean regional NWP post-processing task. The combination is new in this specific form, code is promised, and the heavy-rain CSI10 improvement (0.096 vs 0.060 best baseline) is the kind of gain that would matter operationally if it held. But the evidence that it holds is thin.\n\nWhat it does well: the pre-training design is sensible (3D patches, reconstructing masked atmospheric variables, transfer to segmentation), the probabilistic density labeling is a simple and reasonable alternative to one-hot for thresholded rain categories, and the ablation structure is thorough — sampling, α, β, masking ratios. The authors also include a monthly analysis (Figure 5) showing consistent improvement over raw NWP output across most of a year, which is a point in their favor.\n\nSoft spots, in rough order of seriousness. First, the central claim that SSLPDL 'surpasses all baselines' is based on a single test month (August 2022), chosen for heavy rain, with no error bars or repeated runs. Rare-event metrics like CSI10 can jump around a lot from a few cells; one month is not enough to establish superiority. Second, the abstract says 'over 9.9% mIoU improvement' compared to six baselines, but against the best baseline (Swin-Unet/PostRainBench at 0.389) the gain is about 5.9%. The 9.9% only works if you average the baselines, which isn't what the text says. Third, the masking-ratio ablation text claims 'the larger the masking ratio, the better' but Table 4 shows 90% is worse than 75% (CSI0.1 0.356 vs 0.389). Fourth, the density labels don't actually sum to 1 for interior intervals, despite the claim — the sum ranges from α to 1 depending on the fractional position. That's a mathematical slip, not necessarily fatal, but it needs correction.\n\nOne thing the stress-test note got wrong: Table 5 is not internally inconsistent. Read as base score (no density labeling) + delta from applying the labeling, every row reconciles with Table 3 (e.g., PostRainBench CSI10 0.063 − 0.035 = 0.028). The presentation is confusing, but the numbers are consistent.\n\nBottom line: the method is plausible, the writing is clear, and the code promise is a plus. But the headline superiority claim is not yet supported. It deserves a serious referee, not a desk reject, because the recipe is worth testing against multi-season, multi-seed data. If the authors broaden the evaluation and fix the reporting, this could become a useful paper.\n\nRecommendation: send to peer review, but with a clear request for multi-month, multi-seed evaluation and significance testing before any acceptance decision.","headline":"A plausible SSL pre-training recipe for precipitation post-processing, but the headline 'surpasses all baselines' rests on one unreplicated month and a misread improvement figure.","tokens_in":13978,"tokens_out":4843,"would_cite":false,"duration_ms":42935,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Self-supervised rainfall post-processor beats six baselines on heavy-rain detection.","keywords":["self-supervised learning","rainfall probability estimation","numerical weather prediction","precipitation post-processing","class imbalance","probabilistic density labeling","masked modeling","heavy rain forecasting"],"falsifier":"Recompute CSI10 and mIoU for SSLPDL and every baseline on a multi-year, multi-season test set with several random seeds: if the heavy-rain CSI advantage over the best baseline, 0.096 versus 0.060 on August 2022, does not persist in other months or shrinks inside run-to-run variance, the central outperformance claim is falsified.","tokens_in":12901,"feed_emoji":"🌧️","tokens_out":11885,"duration_ms":109336,"temperature":0.7,"pith_summary":"The paper asks whether a machine-learned post-processor can turn raw numerical weather prediction (NWP) output into reliable rainfall probability maps, especially for rare heavy-rain events. It proposes SSLPDL, which first pre-trains an encoder by reconstructing randomly masked chunks of sixteen atmospheric variables, then fine-tunes that encoder for pixel-level rainfall segmentation. To stop the model collapsing onto the overwhelmingly common 'no rain' class, it replaces hard one-hot labels with probability-density labels that spread probability across neighboring rainfall classes. On a one-month regional test set, SSLPDL reports the best scores among seven compared approaches, with heavy-rain CSI10 of 0.096 and mean overlap (mIoU) of 0.412 against a best baseline of about 0.389. If that advantage holds beyond the single test month, the method offers a practical way to extend the usable warning time of existing NWP forecasts.","feed_headline":"Rainfall post-processor tops six baselines on heavy rain","feed_subtitle":"Masked self-supervised learning plus smooth density labels corrects NWP bias and extends lead time to 30 hours.","key_machinery":"The machinery is a two-stage pipeline. In pre-training, patches of the 16-variable NWP input are randomly masked and fed through an encoder built from deformable convolution layers, convolution kernels with learned sampling offsets that let each output position pull features from adaptive neighboring locations, and the model is trained to reconstruct the masked patches by minimizing mean squared error over the masked set. This forces the encoder to represent dependencies among temperature, humidity, wind, pressure, and precipitation rather than memorize fixed pixels. In the downstream stage, the pre-trained encoder is attached to a segmentation decoder and trained with a weighted cross-entropy loss that mixes the one-hot label $y_i$ with a probabilistic density label $y_i^*$, defined for rainfall $\\gamma$ between thresholds $\\tau_{i-1}$ and $\\tau_i$ as $y_i^* = (1-\\alpha)\\frac{\\tau_i - \\gamma}{\\tau_i - \\tau_{i-1}} + \\frac{\\alpha}{N}$, with $\\alpha$ controlling smoothing strength and $N$ the number of classes. The mixture ratio $\\beta$ in the loss is set to 0.25 and the fine-tuning masking ratio to 25%, which the ablation identifies as the best configuration.","core_discovery":"The central claim is that masking-based self-supervised pre-training over atmospheric variables, combined with a smooth density-based labeling scheme, produces a precipitation post-processor that corrects NWP bias better than models trained directly on the downstream task. Specifically, SSLPDL reports the best scores among all six compared baselines on every rainfall interval, and its largest advantage is in heavy rain above 10 mm: CSI10 of 0.096 versus a best baseline around 0.060, with mIoU of 0.412 versus 0.389. Because the evaluation uses forecasts from 25 to 30 hours ahead, the authors present the method as extending the lead time at which NWP rainfall guidance remains reliable. The paper also claims the probabilistic density labeling is the component that prevents the rare heavy-rain class from being overwhelmed by the dominant no-rain class.","pith_inferences":["If the reported gain reproduces across multiple years, the same pre-training-plus-density-label recipe should transfer to other regional NWP systems and other rainfall thresholds, because neither the masking nor the labeling depends on the specific Korean grid.","The probabilistic density labeling is a general answer to class imbalance in gridded geophysical segmentation; hail, fog, and storm-surge probability mapping, all of which suffer from rare extreme pixels, are natural places to test it.","A decisive check would be calibration: reliability diagrams and Brier scores would show whether the smooth labels yield well-calibrated rainfall probabilities or merely better segmentation rankings.","Because the current evidence is a single August with no error bars, the specific magnitude of the heavy-rain gain, 0.096 versus 0.060, should be read as provisional until run-to-run and year-to-year variance is measured."],"forward_implications":["Heavy-rain detection improves by a large margin: CSI10 rises from about 0.060 for the strongest baseline to 0.096, with mIoU reaching 0.412 versus 0.389.","The gains are not limited to the rare class: the method reports the best CSI0.1 and mIoU across all rainfall intervals, so the pre-training does not sacrifice common-rain performance to catch heavy rain.","The approach works as a post-processor on forecasts already made 25 to 30 hours ahead, so it can extend the usable lead time of existing NWP output without waiting for a finer-resolution run.","Probabilistic density labeling outperforms both one-hot labels and standard label smoothing across sampling strategies, with the largest improvements concentrated in the 10 mm class.","Monthly analysis shows consistent improvement over raw NWP output from March through November, with the largest gain, about 7.1% mIoU, in the high-variability July monsoon period."],"supporting_citations":[{"why":"Supplies the masked-autoencoder pre-training objective that SSLPDL adapts to three-dimensional atmospheric fields.","marker":"[15]"},{"why":"Provides the deformable-convolution backbone used for adaptive spatial aggregation in the encoder.","marker":"[42]"},{"why":"Motivates the masked spatiotemporal pre-training strategy for video-like weather data.","marker":"[39]"},{"why":"Anchors the masked-autoencoder approach in weather-specific forecasting and multi-variable reconstruction.","marker":"[26]"},{"why":"One of the six benchmark post-processing models whose scores SSLPDL is compared against.","marker":"[33]"},{"why":"One of the six benchmark neural weather models used as a baseline.","marker":"[36]"},{"why":"Supplies the post-processing benchmark and one of the models that the method must beat.","marker":"[38]"},{"why":"Defines the standard label-smoothing technique that probabilistic density labeling is designed to improve on.","marker":"[28]"}],"fun_headline_variants":["Self-supervised post-processor tops six baselines on heavy rain","Density labels and masked learning extend rainfall lead time","SSLPDL corrects NWP bias, boosts CSI on heavy rain","Rare rain events get a self-supervised boost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the single test month, August 2022, with one run and no confidence intervals, represents the method's true performance; if that month is unrepresentative, the claim that SSLPDL surpasses all baselines is unsupported.","fun_headline_variants_meta":{"raw":{"variants":["Self-supervised post-processor tops six baselines on heavy rain","Density labels and masked learning extend rainfall lead time","SSLPDL corrects NWP bias, boosts CSI on heavy rain","Rare rain events get a self-supervised boost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000231,"raw_usage":{"total_tokens":1480,"prompt_tokens":937,"completion_tokens":543,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":473}},"tokens_in":553,"tokens_out":543,"duration_ms":6050,"temperature":1.0,"reasoning_tokens":473,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:18:16.719863+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute CSI10 and mIoU for SSLPDL and every baseline on a multi-year, multi-season test set with several random seeds: if the heavy-rain CSI advantage over the best baseline, 0.096 versus 0.060 on August 2022, does not persist in other months or shrinks inside run-to-run variance, the central outperformance claim is falsified.","supporting_citations":[{"cited_title":"Internimage: Exploring large-scale vi- sion foundation models with deformable convolutions","cited_arxiv_id":null,"evidence_quote":"Provides the deformable-convolution backbone used for adaptive spatial aggregation in the encoder."},{"cited_title":"Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training","cited_arxiv_id":null,"evidence_quote":"Motivates the masked spatiotemporal pre-training strategy for video-like weather data."},{"cited_title":"Convolutional lstm network: A machine learning approach for precipitation nowcasting","cited_arxiv_id":null,"evidence_quote":"One of the six benchmark post-processing models whose scores SSLPDL is compared against."},{"cited_title":"When does label smoothing help? Advances in neural in- formation processing systems, 32, 2019","cited_arxiv_id":null,"evidence_quote":"Defines the standard label-smoothing technique that probabilistic density labeling is designed to improve on."}],"review_version":1}