REVIEW 3 major objections 4 minor 1 references
Operational convection-permitting COSMO/ICON ensemble predictions at observation sites (CIENS)
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The CIENS dataset pairs more than twelve years of operational convection-permitting ensemble forecasts with station observations at 170 German sites.
desk verdict Worth reviewing as a dataset contribution, but the abstract's flagship non-stationarity use case hinges on model-version metadata that is nowhere mentioned. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the CIENS station-level forecast-observation pairing: 55 model variables are mapped or spatially aggregated onto 170 station locations and aligned with hourly lead times from 0 to 21 hours for both daily runs, with six observed variables for verification. This machinery makes the model output directly usable for post-processing and verification without grid handling, and the long record across model updates turns the archive into a test bed for changing forecast-error distributions.
What would settle it
A single calculation could settle the central premise: compare CIENS station-mapped forecasts with raw nearest-grid-point model output for a sample of stations and lead times, and check whether residuals jump at known station relocation or instrument-change dates.
Extended reading notes
Core claim
The central claim is that CIENS is a benchmark dataset in which operational convection-permitting ensemble forecasts, rather than reanalysis or frozen model runs, are mapped to 170 synoptic stations over more than twelve years, with six observed variables attached for direct verification. Because the archive spans several upgrades of the underlying numerical weather prediction model, it exposes the non-stationarity that real forecast post-processing must handle instead of hiding it behind a fixed model version. A use case on ensemble post-processing illustrates that machine-learning forecasting models benefit from the rich set of available model predictors.
Load-bearing premise
The dataset's worth rests on the station-mapped forecasts and paired observations staying consistent and unbiased over twelve years, since any interpolation bias or break in the observation record is inherited by every downstream study.
Editorial extensions
If this is right
- Users can train and evaluate ensemble post-processing methods directly on forecast-observation pairs without having to handle model grid geometry.
- The multi-year, multi-model-version span lets researchers test whether post-processing methods remain accurate when the underlying numerical model changes.
- The 55-variable predictor set plus spatial aggregates enables machine-learning post-processing to use far more information than standard ensemble moments.
- The paired observations support verification of probabilistic forecasts at the point scale for six standard variables.
- Because the data format matches operational forecast use, results obtained on CIENS should transfer more readily to real forecasting practice than idealized benchmarks.
Reading between the lines
- If CIENS becomes a standard benchmark, model-update boundaries offer natural test splits: methods that degrade around those boundaries can be identified and replaced, turning operational model churn into a research asset.
- The station-mapping step may introduce small-scale representativeness errors, especially at short lead times, so users studying kilometer-scale processes should compare the mapped values with raw grid output.
- The long record could also be used to estimate how station relocations and observation quality-control changes affect apparent forecast skill, a confound the paper does not itself quantify.
- Similar station-mapped archives for other national weather services would allow cross-country comparisons of post-processing transferability and model-change robustness.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents CIENS, a station-matched dataset built from DWD's operational convection-permitting ensemble forecasts (COSMO/ICON), containing 55 forecast variables mapped to 170 German synoptic stations, hourly lead times from 0 to 21 hours for the 00 UTC and 12 UTC runs, and station observations for pressure, temperature, precipitation, wind speed, wind direction, and wind gusts, covering December 2010 to June 2023. The authors emphasize the dataset's long temporal extent across multiple NWP model updates as its key distinguishing feature, and they illustrate a machine-learning ensemble post-processing use case that benefits from the rich predictor set.
Significance. If the dataset is as described and its construction is fully documented, CIENS would be a valuable community resource: it provides a long, station-matched, multi-variable ensemble forecast record spanning several operational model generations, which is well suited for post-processing, verification, and studies of forecast non-stationarity. The included machine-learning use case is a constructive demonstration of the dataset's potential. However, the provided manuscript text is severely corrupted in the version I received, and the abstract alone omits several load-bearing details, so the dataset's significance cannot currently be fully assessed from the materials available to me.
major comments (3)
- [Abstract, distinguishing feature] The paper's flagship claim is that the long temporal extent 'encompasses multiple updates to the underlying numerical weather prediction model' and supports investigations into how forecasting methods account for such changes. This use case requires a machine-readable mapping from each forecast run to the operational model version, or at least a documented update chronology aligned with DWD's operational change log. The abstract enumerates 55 variables, 170 stations, lead times, and two daily runs, but it does not mention model-version identifiers or update timestamps anywhere in the dataset description. Please state explicitly whether such metadata is included; without it, the central advertised capability is not actually supported by the dataset as described.
- [Abstract, station mapping] The abstract states that forecasts are 'mapped to the locations of synoptic stations' and that additional spatially aggregated forecasts from surrounding grid points are available for a subset of variables, but it does not specify the interpolation scheme, the treatment of station elevation versus model orography, or the definition of 'surrounding grid points'. For convection-permitting output, the grid-to-point mapping strongly affects apparent forecast errors and any downstream post-processing benchmark. The corrupted full text I received does not allow me to locate such a description; please add a precise specification of the mapping and, ideally, an estimate of its representativeness uncertainty.
- [Observations and quality control] The dataset supplies twelve years of station observations for six variables at 170 locations, but the abstract provides no information about quality control, homogeneity adjustments, station relocations, instrument changes, or missing-data flags. Any inhomogeneity in the observational record will be inherited by every post-processing or verification study built on CIENS. Please include a dedicated observation-processing section, with station metadata and per-variable quality flags, so that users can assess and account for observational inhomogeneities.
minor comments (4)
- [Abstract] There is a typo in 'Since the forecast are mapped to the observed locations'; it should read 'Since the forecasts are mapped'.
- [Full text] The manuscript text in the version provided to me is in an unreadable character encoding (mojibake), so I could not verify the section structure, equations, tables, or any statements beyond the abstract. Please ensure the arXiv source compiles correctly and contains readable text.
- [Data availability] The abstract does not state where the CIENS dataset can be accessed, under what license, or in which file format. A dataset paper should include a persistent identifier and a clear data-availability statement.
- [Model configuration] Please specify exactly which operational models and configurations are included across the 2010-2023 period (e.g., COSMO-DE versus ICON-D2 ensemble), including ensemble size and perturbation strategy, since the period straddles a major model transition.
Circularity Check
No significant circularity: this is a dataset release paper whose central claim is an existence claim about archived operational forecasts and observations, not a derived scientific prediction.
full rationale
The paper's load-bearing assertion is that the CIENS dataset exists and contains operational convection-permitting ensemble forecasts mapped to station locations, paired with station observations over a long period. That is an empirical curation claim, not a quantity derived from a fitted model or from a self-citation chain. The described use case trains and evaluates a machine-learning post-processing model on the dataset, which is a benchmark demonstration rather than a prediction that is forced by construction; any honest evaluation would use held-out data and standard baselines. No equation was presented in the readable portion that defines a target variable in terms of the same target variable, and no fitted parameter is renamed as a prediction. Concerns about interpolation choices, observation homogeneity, or missing model-version metadata are correctness or completeness risks, not circularity. Accordingly, the score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption Archived COSMO/ICON ensemble output used for CIENS matches the operational forecast product without undocumented corrections.
- domain assumption Station observations for pressure, temperature, precipitation, wind speed, wind direction, and gusts are quality-controlled and homogeneous from December 2010 to June 2023.
- domain assumption Mapping ensemble forecasts to station locations preserves the statistical properties of the ensemble, including spread.
Cite this review
Pith. "Pith review of Operational convection-permitting COSMO/ICON ensemble predictions at observation sites (CIENS)." pith.science (2026). https://pith.science/paper/I3AX3O5T
@misc{pith2026250803845,
author = {Pith},
title = {Pith review of: Operational convection-permitting COSMO/ICON ensemble predictions at observation sites (CIENS)},
year = {2026},
howpublished = {\url{https://pith.science/paper/I3AX3O5T}},
note = {Machine review of arXiv:2508.03845}
}
read the original abstract
We present the CIENS dataset, which contains ensemble weather forecasts from the operational convection-permitting numerical weather prediction model of the German Weather Service. It comprises forecasts for 55 meteorological variables mapped to the locations of synoptic stations, as well as additional spatially aggregated forecasts from surrounding grid points, available for a subset of these variables. Forecasts are available at hourly lead times from 0 to 21 hours for two daily model runs initialized at 00 and 12 UTC, covering the period from December 2010 to June 2023. Additionally, the dataset provides station observations for six key variables at 170 locations across Germany: pressure, temperature, hourly precipitation accumulation, wind speed, wind direction, and wind gusts. Since the forecast are mapped to the observed locations, the data is delivered in a convenient format for analysis. The CIENS dataset complements the growing collection of benchmark datasets for weather and climate modeling. A key distinguishing feature is its long temporal extent, which encompasses multiple updates to the underlying numerical weather prediction model and thus supports investigations into how forecasting methods can account for such changes. In addition to detailing the design and contents of the CIENS dataset, we outline potential applications in ensemble post-processing, forecast verification, and related research areas. A use case focused on ensemble post-processing illustrates the benefits of incorporating the rich set of available model predictors into machine learning-based forecasting models.
Reference graph
Works this paper leans on
-
[1]
����� ������� ������ ��� ���� ������� ����� � � ��� ��������� ����� �� ��������� ���������� ����������� ������������������ �������� ������ � ������� ������ �� �� � ��� �� �� �� ������ � � ���������� �� �������� ���������� �� �������� �� �� ������ ������� �������� �������� ��� ���� ������ � ����� �� �������� � ������������� ��������� ���������� �� ������� ...
work page Pith review arXiv 2025
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.