{"id":"db631988-33f7-4084-b360-06e7dd7221ed","arxiv_id":"2509.11047","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Training an autoregressive diffusion weather forecaster on 20% of ERA5 data selected uniformly by calendar month matches full-data CRPS/RMSE and improves the spread-skill ratio on the 2018 test year.","lead":"Training a diffusion weather model on one fifth of the historical record, sampled evenly by calendar month, matches full-data forecasting accuracy and improves ensemble calibration in a 2018 test. If confirmed with proper error bars, this gives an immediate 80% reduction in training data for expensive autoregressive weather diffusion models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SSR advantage may be an artifact of unequal training budget: Section 4.1 asserts only the data subset varies but never reports optimizer steps/epochs, so the 20% models may have seen each sample ~5x more often.","rationale":"The reader's weakest_assumption is exactly the one I would raise: the missing training budget. The strongest claim is empirical and comparative, so the only way to attribute the SSR result to temporal stratification is to hold the optimization procedure constant. The paper's own Section 4.1 asserts 'only variable is subset' without specifying steps/epochs. This is a genuine hole, not a disagreement with consensus. The tables provide some indirect support: across five variables, stratified time consistently has the best SSR, and CRPS/RMSE are close to full-data. That consistency makes the result plausible but does not rule out the confound, because the same budget artifact would affect all 20% runs. I considered other concerns (single test year, no error bars, no code), but the single-year limitation is disclosed and the metric pattern is broad; the budget ambiguity is more direct. The appendix also says extended sampling variants 'were lacking' and omits the promised ws10 table, minor reporting gaps. An honest non-finding is not appropriate here because the missing training schedule is verifiable and central. I agree with the reader's conditional verdict and would not shift it.","tokens_in":7755,"tokens_out":5021,"duration_ms":61525,"concrete_test":"Obtain the training configuration and retrain the Full Data and Stratified Time models under a matched optimization budget: fix batch size and the total number of optimizer steps for both models, so the subset model completes ~5x more epochs per sample. Use the 2016-17 validation CRPS for early stopping and report three seeds. If the stratified model's 5-day z500 SSR is no longer closer to 1 than Full Data in this matched-steps condition, the paper's central 'outperforms full data on SSR' claim is not established; if it persists, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that month-stratified 20% training matches full-data CRPS/RMSE and beats it on SSR. Section 4.1 says 'the only variable is the subset of training data used,' but the paper never states the number of optimizer steps, epochs, batch size, learning-rate schedule, or convergence criterion. If all runs used a fixed number of parameter updates (the common default), then each sample in a 20% subset is seen about five times more often than in full-data training. The reported SSR improvements (e.g., 0.89 vs 0.84 for z500 at 5 days; 0.93 vs 0.89 for t850 at 5 days) could then be an increased-epochs effect or an overfitting/regularization artifact rather than a property of temporal stratification. Because the same confound affects all 20% baselines, it could also explain why random/k-means/greedy methods show modest SSR gains over full data in several rows. This is not an internal logical inconsistency, but it means the headline result is not identifiable from the text: without the optimization budget, the comparison does not isolate data selection. The authors disclose the single-test-year limitation and report consistent multi-variable tables, but this missing control undermines the causal attribution that stratified sampling causes better calibration.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies data selection for training an autoregressive diffusion model for global ensemble weather forecasting. Using ERA5/WeatherBench data (5.625° resolution), it compares several 20% training-subset strategies—random, k-means, greedy diverse, spatial, herding, and stratified time sampling—against full-data training over 1979–2015, with evaluation on the 2018 test year. Metrics are CRPS, RMSE, and spread/skill ratio (SSR) at 5-day and 10-day lead times. The central claim is that simple month-stratified time sampling matches full-data CRPS/RMSE and improves SSR, despite using only 20% of the training data. The paper explicitly acknowledges the single-test-year limitation and that the evaluated samplers are static heuristics rather than learned or adaptive methods.","tokens_in":8112,"tokens_out":3830,"duration_ms":52198,"significance":"If the central claim is correct, the paper provides a useful and inexpensive baseline for data-efficient training of diffusion-based ensemble weather forecasters. A simple temporal coverage prior that matches or improves calibration at 20% data would be of practical interest and would motivate model-aware sampling. The paper is also honest about several limitations, and the evaluation is not circular: metrics are computed on an independent 2018 test period against ERA5, and no fitted constant is defined in terms of the reported outcomes. However, the empirical support is currently too weak to establish the headline claim: there is no stated optimization budget, no repeated-seed variability information, and only one test year.","major_comments":[{"comment":"The paper states that \"the only variable is the subset of training data used,\" but it never reports the number of training iterations/epochs, batch size, learning-rate schedule, or convergence criterion. If all models are trained for the same number of optimizer steps (the common default), then a 20% subset sees each sample roughly five times more often than the full-data model. The reported SSR improvements (e.g., Table 1: 0.89 vs 0.84 for z500 at 5 days; Table 2: 0.93 vs 0.89 for t850 at 5 days) could then be an increased-epochs-per-sample effect, or an overfitting/regularization artifact, rather than a property of temporal stratification. Because the confound affects all 20% baselines, it also weakens comparisons among sampling methods. Please specify the optimization budget, or better, run controlled comparisons at both equal total updates and equal epochs per sample.","section":"Section 4.1"},{"comment":"Every table reports a single value per method with no standard deviations, no confidence intervals, and no significance tests, while diffusion training and inference are stochastic and each method appears to be run once. Differences such as Table 1's z500 5-day SSR 0.89 vs 0.84, or Table 4's u10 5-day SSR 0.94 vs 0.92, may be within run-to-run noise. The claim that stratified time \"outperforms\" full-data training on SSR is not statistically supported. Please provide results over multiple seeds (at least three, preferably more) and report intervals or error bars, and use an appropriate test for paired metric differences.","section":"Tables 1–5"},{"comment":"The authors acknowledge that experiments are limited to a single test year (2018). Since weather exhibits substantial interannual variability, the central claim that stratified 20% training matches or beats full-data training across metrics cannot be established from one year alone. This is not merely a presentation issue: the conclusion is about general forecast skill. Please evaluate on additional independent years (e.g., the 2016–2017 validation years or a multi-year test period) and report per-year and aggregate results.","section":"Section 5"}],"minor_comments":[{"comment":"The baselines labeled \"Spatial\" and \"Herding\" appear in all tables but are described only in one sentence under \"Other Baselines.\" No algorithms, distances, or implementation details are given. Please move them into the main methods or remove them from the tables.","section":"Section 3.2 / Tables 1–5"},{"comment":"Section 4.5 says full results for ws10 are in Appendix A.1, but the appendix contains tables only for t2m, u10, and v10; no ws10 table appears. Either add the ws10 results or correct the reference.","section":"Appendix A.1"},{"comment":"The appendix states that several hybrid stratified variants were tested but \"performance results however were lacking and thus were not included.\" Omitting the numerical results of experiments that were actually run is a reporting gap; it prevents readers from verifying the claim and risks selective reporting. At minimum, include the results in a supplementary table or specify the quantitative criterion used to exclude them.","section":"Appendix A.2"},{"comment":"The diffusion model configuration is described only by reference to prior work. Please state the architecture size, number of diffusion steps, noise schedule, ensemble size at inference, and sampling procedure, since these affect reproducibility and runtime.","section":"Section 4.1"},{"comment":"The tables are inconsistent in formatting: some entries have two decimals and others one (e.g., Table 1 CRPS 242.66 vs 335.2); the full-data row is labeled \"Full Data\" in Tables 1–3 and \"AR-24h (full data)\" in Tables 4–5. Please standardize.","section":"Tables 1–5"}],"recommendation":"major_revision","confidential_remarks":"The core idea is timely, and the empirical setup has some good properties (independent test year, no circular metric). However, the missing training budget is a load-bearing confound for the central causal claim, and the single-seed/single-year evaluation makes the headline results fragile. The manuscript would currently be more appropriate for a workshop than a journal; the authors should be given the opportunity to add controlled training-budget comparisons, repeated-seed intervals, and multi-year evaluation. I also note the appendix's admitted omission of failed hybrid variants; that should be addressed during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Kevin, quick take on arXiv:2509.11047. The genuinely new thing here is the evaluation: nobody has looked at simple data-subset selection for autoregressive diffusion ensemble weather forecasting. The authors test random, k-means, greedy diversity, and stratified time sampling under a fixed 20% budget using the standard WeatherBench setup, and they report metrics on an independent 2018 test year. The month-stratified result is plausible—weather is strongly seasonal, so uniform month coverage is a sensible prior—and the tables are internally consistent. Giving credit where it's due: the paper is honest about the single-test-year limitation in Section 5, and the related work is fair.\n\nThe soft spot is the one the stress-test flags. Section 4.1 says 'the only variable is the subset of training data used,' but the paper never reports the number of optimizer steps, epochs, batch size, or convergence criterion. If all runs use the same number of parameter updates, the 20% models see each sample about five times more often than the full-data model. That alone can improve calibration metrics like SSR, so the claim that stratified sampling causes the better spread/skill ratio is not identifiable from what's written. This affects all the 20% baselines equally, and it may not change the relative ordering among them, but it breaks the comparison to full data. That's a real confound, not a manufactured one.\n\nThe other weaknesses are the usual ones: one seed per method, no error bars or significance tests, and a single test year. The authors disclose the test year, but not the seed or step details. The appendix's extended hybrid sampling variants are mentioned only as failures—fine, but it means we don't know how much was tried.\n\nOverall: this is a plausible benchmark study that deserves a serious referee, but the headline needs an additional experiment or at least a clear statement of the optimization budget. If the authors fix that and add error bars, the result that 20% stratified data roughly matches full-data RMSE/CRPS while improving SSR would be a useful data point for the ML-weather community. Right now it's a good motivation for future work, not a demonstrated causal claim.\n\nRecommendation: send to peer review, with a request for seeds, the training budget, and ideally a second test year.","headline":"Useful first benchmark of data selection for diffusion weather forecasting, but the SSR headline is not yet identifiable because the training budget is unreported.","tokens_in":8540,"tokens_out":3016,"would_cite":false,"duration_ms":36755,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 20% month-stratified subset of weather data trains a diffusion forecaster as well as the full dataset, and improves ensemble calibration.","keywords":["diffusion models","ensemble weather forecasting","data selection","stratified sampling","data efficiency","ERA5","spread-skill ratio","autoregressive forecasting"],"falsifier":"Train the full-data and stratified-20% models with matched total optimization steps and matched number of epochs per sample, then compare SSR and CRPS on an independent test year other than 2018 (for example, 2019 or 2020). If the stratified model no longer matches or beats the full-data model on SSR, the paper's central claim about data selection would be weakened.","tokens_in":7647,"feed_emoji":"🌦️","tokens_out":2065,"duration_ms":26431,"temperature":0.7,"pith_summary":"The paper tries to show that curated data selection can make diffusion-based ensemble weather forecasting much cheaper without losing quality. Using only 20% of the training data, a simple strategy that samples evenly from each calendar month matches the full-data model's accuracy and often produces better-calibrated ensembles. This matters because training autoregressive diffusion models on decades of global weather data is computationally heavy, and high-quality climate data is often scarce or costly. If true, the result opens the door to practical data-efficient training for scientific forecasting models.","feed_headline":"20% of weather data trains a forecaster as well as all of it","feed_subtitle":"Month-balanced sampling matches full-data accuracy and improves ensemble calibration in diffusion forecasting.","key_machinery":"The central mechanism is stratified time sampling: partitioning the training period by calendar month and drawing an equal number of samples from each month, enforcing seasonal coverage. This is applied to an autoregressive 24-hour diffusion model that is rolled out iteratively to produce 10-day ensemble forecasts. The paper's argument depends on this sampling strategy, alongside the fixed 20% data budget and the evaluation metrics CRPS (probabilistic accuracy and sharpness), RMSE (ensemble-mean error), and SSR (spread-skill ratio, calibration).","core_discovery":"Stratified time sampling—selecting a fixed number of training examples uniformly from each calendar month—achieves performance comparable to full-data training across the five main ERA5 variables (z500, t850, t2m, u10, v10) while using only 20% of the training data. On the spread/skill ratio (SSR), a calibration metric where values closer to 1 are better, the stratified 20% model matches or slightly beats the full-data model at both 5-day and 10-day lead times, while staying within roughly 3–6% on CRPS and RMSE. The paper argues that temporal representativeness provides a strong domain prior for weather, and that simple static sampling heuristics can serve as a baseline for future adaptive d","pith_inferences":["The reported SSR advantage could partly reflect an implicit extra-epochs effect: if the same training-step budget is used for all methods, the 20% model revisits each sample about five times more often, which could improve calibration independent of the sampling strategy—this is not addressed in the paper.","Since only the 2018 test year is used, the seasonal-coverage benefit might be exaggerated for a single year; testing on additional years (e.g., 2019–2020) would clarify whether the advantage is robust to interannual variability.","The failure of hybrid strategies that combine stratified time with spatial clustering or entropy suggests that heuristics designed on top of temporal stratification can hurt, implying that gains come mostly from even seasonal coverage rather than additional feature-space diversity—an inference that could be tested by ablations.","A practical extension would be to combine stratified temporal sampling with online selection based on forecast error or uncertainty, potentially reaching the same performance with even less than 20% data."],"forward_implications":["Training an autoregressive diffusion weather forecaster on 20% of the data can cut storage, preprocessing, and training cost by roughly 80% while preserving forecast accuracy and improving ensemble calibration.","Stratified temporal sampling is a strong, simple baseline that future adaptive or model-aware sampling methods should be compared against.","The result suggests that for strongly seasonal signals like weather, data coverage across regime space matters more than the total volume of data.","Because the 20% models match full-data CRPS and RMSE, data-efficient training is feasible even when data access is restricted or expensive."],"fun_headline_variants":["Month-balanced sampling: 80% less data, same forecast skill","Stratified sampling cuts training data 80% with no accuracy loss","Weather AI: 20% data, 100% skill with smart sampling","Diffusion forecaster thrives on 20% data via month sampling","Data-efficient weather AI: better calibration with 20% data"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The comparison assumes that fixing model hyperparameters isolates the effect of data selection, but the paper never states the number of training steps or epochs; if the step budget is the same for all methods, the 20% model sees each sample roughly five times more often, so part of the advantage could come from repetition rather than stratified selection.","fun_headline_variants_meta":{"raw":{"variants":["Month-balanced sampling: 80% less data, same forecast skill","Stratified sampling cuts training data 80% with no accuracy loss","Weather AI: 20% data, 100% skill with smart sampling","Diffusion forecaster thrives on 20% data via month sampling","Data-efficient weather AI: better calibration with 20% data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000459,"raw_usage":{"total_tokens":2107,"prompt_tokens":686,"completion_tokens":1421,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":430,"completion_tokens_details":{"reasoning_tokens":1328}},"tokens_in":430,"tokens_out":1421,"duration_ms":11919,"temperature":1.0,"reasoning_tokens":1328,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T17:10:26.824652+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the full-data and stratified-20% models with matched total optimization steps and matched number of epochs per sample, then compare SSR and CRPS on an independent test year other than 2018 (for example, 2019 or 2020). If the stratified model no longer matches or beats the full-data model on SSR, the paper's central claim about data selection would be weakened.","supporting_citations":[],"review_version":1}