Pith. sign in

REVIEW 3 major objections 4 minor 2 references

Analogue Forecast System for Daily Precipitation Prediction Using Autoencoder Feature Extraction: Application in Hong Kong

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An enhanced analogue forecast system correctly predicted over half of Hong Kong's heavy-rain days at Day 1, more than double the old system.

desk verdict A genuinely operational autoencoder-based analogue forecast system, but the headline skill claim is compromised by overlapping optimization and verification periods. read the letter →

arxiv 2501.02814 v1 pith:E6NAKXL4 submitted 2025-01-06 physics.ao-ph cs.LG

Yee Chun Tsoi , Yu Ting Kwok , Ming Chun Lam , Wai Kin Wong This is my paper · ORCID
classification physics.ao-phcs.LG
keywords AnalogueforecastDailyprecipitationAutoencoderfeatureextractionHeavyrainHongKongNumericalweatherpredictionMachinelearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that replacing hand-crafted weather-pattern comparisons with autoencoder-learned feature vectors makes analogue forecasting markedly better for daily rain classes in Hong Kong. The new system compresses 60 gridded meteorological fields into compact vectors, optimises a weight for each field, and blends the 25 closest historical analogues into a rain-class forecast. Over a three-year verification period, more than half of observed heavy-rain days were correctly identified on Day 1, more than double the old system's count, and the heavy-rain advantage persisted through Day 9. If true, forecasters gain earlier, more consistent guidance on the damaging heavy-rain events, using the same global model output they already receive.

What carries the argument

The load-bearing component is the convolutional autoencoder. For each day, 60 fields (ten variables at six pressure levels) are remapped to a one-degree grid, normalised, and multiplied by a Gaussian weight $G(r_k)=\exp(-r_k^2/(1.2 r_0)^2)$ centred on Hong Kong; the encoder compresses each field into a vector, and the decoder is trained to reconstruct the input, so each vector is meant to be a compact representation of the weather pattern. Similarity between a forecast day and an archive day is scored by the mean squared error of the extracted vectors, combined with per-feature weights optimised to minimise the rain-class error of the best matching analogue. The final rain class comes from a weighted mean of the 25 closest analogues, with each analogue's contribution scaled by the ratio of its rain class's occurrence in the 25-member sample to its past population frequency.

What would settle it

Re-score the enhanced and existing systems on a period that contributed no days to weight optimisation, for example forecasts issued from May 2022 onward, and compare heavy-rain CSI and POD; if the enhanced system no longer beats the existing system on this independent sample, the central claim collapses. A cleaner experiment is to re-optimise the weights using only 2017–2018 days and verify only on 2021–2022 days.

Watch

Extended reading notes

Core claim

The paper's central claim is that autoencoder-based feature extraction improves the analogue method for daily precipitation prediction. During the May 2019 to April 2022 verification, the enhanced AFS correctly predicted over half of the observed heavy-rain days at Day 1, with a critical success index near 0.4 and a probability of detection above 0.5, and it held a consistent heavy-rain edge over the existing AFS from Day 1 through Day 9 while keeping comparable performance for lighter rain classes. The authors interpret this as evidence that the autoencoder captures the synoptic patterns relevant to local rain more effectively than the previous hand-defined similarity scores, and that the optimised per-feature weights plus the ensemble of 25 analogues produce forecasts that are more accurate and more stable.

Load-bearing premise

The load-bearing premise is that the 2019–2022 verification period fairly represents future operational performance, even though days in 2019–2020 also helped set the optimised feature weightings; if that overlap inflates the apparent skill, the reported superiority may not hold on genuinely new forecasts.

Editorial extensions

If this is right

  • Heavy-rain forecasting is the largest gain: at Day 1 the enhanced AFS reaches a critical success index near 0.4 and a probability of detection above 0.5, outperforming the existing AFS, forecaster bulletins, and direct model output for that class.
  • The advantage is not a one-day effect: the enhanced AFS keeps higher heavy-rain skill from Day 1 through Day 9 and nearly eliminates Day 1 forecasts that miss the observed rain class by more than one step.
  • The 25-analogue ensemble gives forecasters a traceable set of historical scenarios and a range of plausible rainfall amounts, not just a single class, reducing run-to-run fluctuation.
  • False alarms of light rain on dry days increase slightly, so users should expect a small wet bias accompanying the improved heavy-rain detection.
  • The system has been running in real-time operations since May 2022, meaning its reported behaviour can be checked against daily use.
  • The paper itself warns against overfitting the empirical scoring equation to the verification period; the same caution applies to the overlapping optimisation and verification years, though the paper does not directly address that overlap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An independent test on forecasts issued after the weights were frozen, say from May 2022 onward, would give a cleaner estimate of the operational gain than the 2019–2022 verification.
  • Because the autoencoder is trained to reconstruct fields rather than to separate rain classes, a supervised or contrastive training objective could make the features even more precipitation-specific.
  • The pipeline should transfer to other regions with a long reanalysis archive and deterministic model output; the main requirements are enough historical cases and a sensible seasonal restriction on analogues.
  • Under climate change, the fixed historical archive may lack good analogues for unprecedented extremes, so the system would benefit from an explicit flag when no analogue is close enough.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper describes an enhanced analogue forecast system (AFS) for daily precipitation prediction in Hong Kong. The system uses an autoencoder trained on ERA5 reanalysis to compress 60 meteorological fields into feature vectors, applies Optuna-optimised feature weightings to select analogues from ERA5, and forms an ensemble of the 25 most similar analogues to produce a rain-class forecast. The enhanced AFS is verified over May 2019 to April 2022 and compared with the existing HKO AFS, forecaster bulletins, and ECMWF direct output. The authors report that the enhanced AFS outperforms the existing AFS, especially for heavy rain, with a Day-1 CSI near 0.4, POD above 0.5, and 38 of 73 heavy-rain days correctly predicted.

Significance. If the reported skill holds, the enhanced AFS is a practically valuable upgrade to an operational precipitation guidance system, with a clear methodological advance in using autoencoder feature extraction and a detailed, reproducible description of the workflow. The paper also provides a useful comparison against existing operational practice. However, the central verification claim is compromised by the overlap between the optimisation period (2017-2020) and the verification period (May 2019-April 2022), which affects roughly 20 of the 36 verification months. The reported superiority over the existing AFS is therefore partially in-sample, and the absence of uncertainty quantification further weakens the quantitative claims. The strengths are the complete system description, the use of a long ERA5 archive (1979-2020), the real-time operational deployment, and the transparent confusion matrices.

major comments (3)
  1. [Sections 2.1.2, 2.2.3, 3.1] The verification period (May 2019-April 2022) overlaps with the feature-optimisation period (2017-2020) by approximately 20 months. Section 2.2.3 states that Optuna minimises the MSE between the rain class of each day in 2017-2020 and the rain class of its best analogue, using Day-1 HRES forecasts; hence days from May 2019 through December 2020 directly participated in selecting the 60 weightings in Table 3. The Day-1 heavy-rain statistics (38/73 hits, CSI ≈ 0.4, POD > 0.5) and the 'more than double' improvement over the existing AFS are therefore partly in-sample, and the comparison with the existing AFS (whose parameters were not re-optimised on this period) is biased in favour of the enhanced AFS. To establish the headline claim, the authors should either report skill on the subset of the verification period outside the optimisation period (e.g., January 2021-April 2022), or apply a cross-validated/nested optimisation and show that the conclusion is unchanged.
  2. [Section 3.1, Table 5] The paper reports point-value verification metrics without confidence intervals or significance tests. The central comparison rests on only 73 heavy-rain events over three years; with this sample size the difference between 38 and 16 hits, or the CSI difference near 0.4, may not be statistically robust. Please provide interval estimates (e.g., bootstrap or a proper score test) for at least the Day-1 heavy-rain metrics, and ideally for the other lead times and rain classes.
  3. [Sections 2.2.3, 2.2.4, 4.2] Several key design choices are justified only as 'empirical' or 'based on tested results': the Gaussian radius r0 = 7 in Eq. (1), the weighted-score formula with the sample-to-population ratio threshold in Eq. (2), and the fixed ensemble size of 25 analogues. The paper presents no sensitivity analysis or ablation for these choices. This matters because Section 4.2 itself warns against overfitting the 'empirical equation' to the verification period; a sensitivity study would show whether the reported heavy-rain advantage is robust to these choices or an artefact of the specific settings.
minor comments (4)
  1. [Section 3.1] The text contains a typo: 'predication of moderate rain' should be 'prediction of moderate rain'.
  2. [Section 3.1 and Figures 6-8] The acronym 'ECMWF' is misspelled as 'ECWMF' in the sentence 'the averaged ECWMF direct model output' and in the figure captions.
  3. [Figures 6-8] The phrase 'Confusion Comparison' in the figure titles appears to be a typo; it should likely read 'Comparison' or 'Performance Comparison'.
  4. [Table 5] The row label 'Heavy ≤' is ambiguous; it would be clearer to use 'Heavy or above' to match the rain-class grouping in Table 2.

Circularity Check

1 steps flagged · score 6.0 of 10

Feature-weight optimisation on 2017-2020 overlaps the May 2019-Apr 2022 verification period, so the enhanced AFS's reported superiority over the existing AFS is partly in-sample rather than an independent skill estimate.

  1. fitted input called prediction [Sections 2.1.2, 2.2.3, 3.1; Table 1 and Table 3]
    "Day 1 of the forecast data from 2017 to 2020 are used for feature optimisation, while three years of data from May 2019 to April 2022 are used for subsequent forecast verification. ... The objective is to minimise the MSE between the rain classes (Table 2) of each day during the optimisation period and its corresponding best analogue, which is selected from the data archive as the case with least weighted MSE deduced by the feature weightings from the current run."

    The 60 feature weightings in Table 3 are fit by Optuna to make the best analogue's rain class match the observed rain class on each day in 2017-2020. The verification period stated in Section 2.1.2 is May 2019 to April 2022, so approximately 20 of the 36 verification months (May 2019 through December 2020) are also optimisation days. For those days, the same daily rainfall observations serve both as optimisation targets and as verification truth. The Day-1 heavy-rain hits, CSI near 0.4, POD above 0.5, and the 'more than double' improvement over the existing AFS reported in Section 3.1 therefore partly measure in-sample fit. The comparison is also asymmetric: the existing AFS was not re-optimised on this same period.

full rationale

The central claim — consistent superior performance of the enhanced AFS over the existing AFS, especially for heavy rain — is compromised by a training/verification overlap. Section 2.2.3 says Optuna minimises the MSE between the observed rain class of each day in 2017-2020 and the rain class of that day's best analogue, with the best analogue selected by the weighted MSE defined by the feature weightings. Section 2.1.2 says Day 1 forecast data from 2017-2020 are used for feature optimisation while verification spans May 2019 to April 2022. The intersection is about 20 months of the 36-month verification period, so the reported Day 1 heavy-rain skill is partly in-sample. The comparison with the existing AFS is also asymmetric, because the existing AFS parameters were not re-optimised on the same period. The paper itself cautions in Section 4.2 that 'care must be taken not to overfit the empirical equation to cases in the verification period,' but it does not disclose the feature-weighting overlap as a limitation. The autoencoder is not flagged as circular because it is trained without rain labels and only performs unsupervised reconstruction; the circularity is confined to the Optuna feature-weight fitting and the overlapping verification sample. Because the paper's strongest quantitative claim reduces, for a substantial fraction of the verification days, to optimising weights on the same days that are then counted as forecast successes, the score is 6 rather than 0. It is not a complete tautology, since analogue selection, the 25-member ensemble, and the weighted-mean rainfall calculation still add untrained structure.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The system rests on the analogue assumption, the exchangeability of ECMWF HRES forecasts with ERA5 reanalysis after remapping and normalisation, and a set of fitted choices: 60 feature weights, Gaussian radius r0, ensemble size 25, search window width, and the empirical weighted-score formula. No new physical entities are introduced. The fitted parameters are disclosed in tables or text, but their effect on the headline verification is not isolated.

free parameters (6)
  • Gaussian radius r0 = 7 degrees
    r0 in Equation 1 is set empirically so the weight halves at about 7 degrees from Hong Kong; it changes the similarity metric on which all analogue selection is based.
  • Per-feature weightings (60 values) = Table 3 values
    Optimised by Optuna to minimise rain-class MSE between the best analogue and the observed class over 2017 to 2020; directly controls analogue ranking.
  • Number of analogues in ensemble = 25
    Chosen from tested results (Section 4.1), not derived; affects the final rain class through the weighted mean rainfall.
  • Seasonal search window = forecast month +/- 2 months
    Imposed to preserve seasonality; the width is a design choice without sensitivity analysis.
  • Weighted score formula parameters = exponent 2 and sample/population ratio threshold 1
    Equation 2 is described as obtained empirically; the functional form and cutoff are free choices.
  • Autoencoder architecture and training hyperparameters = not fully specified
    Architecture is said to be defined arbitrarily by testing combinations and minimising MSE (Section 2.2.2); the final configuration determines extracted features.
assumptions (6)
  • domain assumption Historical recurrence: similar synoptic patterns in the reanalysis archive imply similar local daily rainfall outcomes.
    Central premise of any analogue forecast; introduced in Section 1 with Lorenz 1969 and the phrase 'history repeats itself'.
  • domain assumption ECMWF HRES deterministic forecasts are sufficiently representative of the expected synoptic conditions for analogue matching.
    The system matches forecasts to reanalysis as if forecast fields were exchangeable with analysed fields; acknowledged as a limitation in Section 4.
  • domain assumption Averaging rainfall over seven gauges gives a representative territory-wide daily rainfall.
    Used for labels and verification (Section 2.1.3); no sensitivity analysis is provided.
  • domain assumption 1 degree average pooling preserves the precipitation-relevant structure of 0.125 and 0.25 degree fields.
    All inputs are remapped to 1 by 1 degrees; the paper notes information loss is possible in Section 4.3.
  • domain assumption Min-max normalisation using archive-wide minima and maxima does not leak future information into historical comparisons.
    Section 2.2.1.2; if normalisation statistics are computed over the full archive including the verification period, forecast vectors are transformed using future data.
  • domain assumption An autoencoder trained only to reconstruct fields produces features whose MSE distance is a useful similarity measure for precipitation outcomes.
    Used in Sections 2.2.2 to 2.2.4; the reconstruction objective does not include rainfall information, so the link between feature-space distance and rain is assumed rather than derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Analogue Forecast System for Daily Precipitation Prediction Using Autoencoder Feature Extraction: Application in Hong Kong." pith.science (2026). https://pith.science/paper/E6NAKXL4

@misc{pith2026250102814,
  author       = {Pith},
  title        = {Pith review of: Analogue Forecast System for Daily Precipitation Prediction Using Autoencoder Feature Extraction: Application in Hong Kong},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E6NAKXL4}},
  note         = {Machine review of arXiv:2501.02814}
}
read the original abstract

In the Hong Kong Observatory, the Analogue Forecast System (AFS) for precipitation has been providing useful reference in predicting possible daily rainfall scenarios for the next 9 days, by identifying historical cases with similar weather patterns to the latest output from the deterministic model of the European Centre for Medium-Range Weather Forecasts (ECMWF). Recent advances in machine learning allow more sophisticated models to be trained using historical data and the patterns of high-impact weather events to be represented more effectively. As such, an enhanced AFS has been developed using the deep learning technique autoencoder. The datasets of the fifth generation of the ECMWF Reanalysis (ERA5) are utilised where more meteorological elements in higher horizontal, vertical and temporal resolutions are available as compared to the previous ECMWF reanalysis products used in the existing AFS. The enhanced AFS features four major steps in generating the daily rain class forecasts: (1) preprocessing of gridded ERA5 and ECMWF model forecast, (2) feature extraction by the pretrained autoencoder, (3) application of optimised feature weightings based on historical cases, and (4) calculation of the final rain class from a weighted ensemble of top analogues. The enhanced AFS demonstrates a consistent and superior performance over the existing AFS, especially in capturing heavy rain cases, during the verification period from 2019 to 2022. This paper presents the detailed formulation of the enhanced AFS and discusses its advantages and limitations in supporting precipitation forecasting in Hong Kong.

Figures

Figures reproduced from arXiv: 2501.02814 by the authors.

Figure 1
Figure 1. Spatial domain of the gridded data. Also shown is the schematic representation of the Gaussian weight function G(rk) applied, where rk is the distance (in degrees) from Hong Kong (HK). 2.1.2 Forecast Data The parameters and spatial information of the ECMWF deterministic model (HRES) data used in this study are identical to that of the reanalysis data. The dataset is formed by forecast data with a model base time of … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 1 canonical work pages

  1. [2]

    The data formats, time periods, and their corresponding purpose at various stages of the model training and operation are summarised in Table 1

    DATA AND METHODOLOGY 2.1 Data Three sets of data are used for the enhanced AFS, namely, (a) the gridded reanalysis data (ERA5), (b) the gridded forecast data from the deterministic model of the ECMWF (HRES), and (c) the daily rainfall observation. The data formats, time periods, and their corresponding purpose at various stages of the model training and o...

    work page 2014

  2. [3]

    1 / weighted MSE) between each forecast and reanalysis pair during operational forecasts

    These optimised weightings are applied to calculate the weighted MSE and similarity (i.e. 1 / weighted MSE) between each forecast and reanalysis pair during operational forecasts. Table 2: Definition of rain class by daily rainfall in this study. Classes Heavy, Very Heavy, Torrential, and Extreme are grouped together for model optimisation and verificatio...

    arXiv 2019

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.