{"id":"2990e3f6-15c9-4c6f-b47e-fbab8d753e46","arxiv_id":"2509.00653","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A curated 12-km, 43-channel, 2000-2019 regional reanalysis dataset with standardized splits and UNet/Stormer/GraphCast/Hi baselines for data-driven weather forecasting over India.","lead":"IndiaWeatherBench is a new benchmark for machine-learning weather forecasting over India, built on 20 years of 12-kilometer reanalysis data with standardized training and test splits. It provides baselines in four model architectures and two ways of feeding boundary conditions, so different forecasting methods can be compared fairly.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation uses ground-truth auxiliary inputs (true boundary/ERA5 state) so reported skill and rankings reflect perfect boundary information, not operational forcing; this is disclosed but limits the benchmark's value as a common forecasting yardstick.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: using ground-truth auxiliary inputs during evaluation. My independent reading confirms this is the most consequential threat to the central claim. For IndiaWeatherBench to be a common yardstick, its evaluation protocol must produce results that at least approximately reflect the forecasting task users care about. Perfect boundary/global-condition forcing is an idealized task; the paper even concedes operational support is absent. This alone justifies a conditional verdict: the benchmark asset (curated IMDAA data, splits, metrics, code) is genuinely useful, but the headline baseline results should be interpreted as idealized, not operational. I considered other issues—the inconsistent parameter counts (25–30M in Section 5 vs. 30–35M in Section 9.1), the absence of uncertainty quantification across training runs, and the limited precipitation evaluation—but these are secondary: they affect reproducibility or completeness, whereas the ground-truth auxiliary protocol affects the validity of every quantitative result in the paper. The proposed test would settle the concern by degrading the auxiliary input to something a user could actually obtain in forecast mode. If rankings hold, the concern is largely mitigated; if they change, the paper's numeric conclusions require re-framing. This does not warrant rejection because the disclosure is honest and the benchmark infrastructure remains valuable, but it does warrant keeping the reader's conditional verdict.","tokens_in":18065,"tokens_out":7270,"duration_ms":97536,"concrete_test":"Re-run the Section 5.1 baselines (Figures 1–3) with S_t replaced by a realistic operational proxy instead of ground truth: e.g., for coarse conditioning, use ERA5 forecasts from a public global ML model such as GraphCast (or at minimum a 6-hour persistence of the true ERA5 state) as S_t for the 2019 test year. Then recompute RMSE and the relative model rankings at lead times 6–120 h. If rankings or error gaps change materially (e.g., Stormer's coarse-conditioning collapse disappears, or Hi's heatwave advantage in Figure 4 reverses), the ground-truth-auxiliary protocol is the cause and the benchmark's reported comparisons are not transferable to operational forcing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that IndiaWeatherBench provides a common yardstick for data-driven regional forecasting—requires that the reported baseline numbers and rankings meaningfully represent forecasting skill. But the protocol feeds ground-truth auxiliary inputs at every autoregressive step: Section 4.1 states \"we use the ground-truth weather state for S_t during training and evaluation,\" and Section 6 lists as a limitation that the benchmark \"relies on ground-truth auxiliary inputs during evaluation and does not yet support real-time forecasting from operational global models.\" This is not merely an operational nicety. For boundary forcing, the 10-pixel true boundary acts as a perfect lateral boundary condition; for coarse-resolution conditioning, true ERA5 is an oracle global analysis sequence that never drifts. In reality, S_t would come from a global forecast model and would contain errors that propagate into the regional domain. Consequently, Figures 1–3 and the extreme-event analysis (Figures 4) are conditional on this idealized forcing: they overstate achievable skill and, more importantly, may mis-rank models by how well they exploit perfect rather than imperfect auxiliary information. For example, a model that aggressively relies on boundary corrections could look strong with true boundaries but degrade in operational settings where boundary errors accumulate. Thus the benchmark's core usefulness as a common yardstick for regional forecasting is undermined exactly where the paper is least secure. The limitation is openly acknowledged, but it is load-bearing because it directly qualifies every headline result in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces IndiaWeatherBench, a dataset and benchmark for data-driven regional weather forecasting over India. It curates 20 years (2000-2019) of IMDAA reanalysis at 0.12° (~12 km) resolution over a 256x256 grid, with 43 variables including single-level, pressure-level, and static fields, and provides standardized train/validation/test splits (2000-2017/2018/2019). The benchmark defines a regional forecasting task with two auxiliary-input strategies: boundary forcing (10-pixel true boundary) and coarse-resolution conditioning (ERA5, interpolated). Four architectures (UNet, Stormer, GraphCast, Hi) are trained with deterministic MSE on increments and one UNet diffusion model for probabilistic forecasting; metrics are RMSE, ACC, CRPS, SSR. Results show ranking changes across conditioning strategies (Stormer best under boundary forcing, worst under coarse conditioning) and a heatwave case study. Data, code, and evaluation pipelines are open-sourced.","tokens_in":18361,"tokens_out":7018,"duration_ms":88086,"significance":"If the protocol is accepted, IndiaWeatherBench fills an evident gap by providing a common, high-resolution regional benchmark for India, analogous to WeatherBench. The dataset construction, splits, and metric definitions are careful, and the open release of raw/preprocessed data and model code is a concrete contribution. However, because auxiliary inputs (boundary and coarse-resolution context) are ground-truth in all experiments (Sections 4.1 and 5), the reported numbers quantify performance under perfect external forcing rather than operational forecasting. This limits the benchmark's value as a direct yardstick for real forecasting skill and may bias model rankings; the paper needs to address this gap before the central claim is fully supported. With that addressed, the benchmark would be a valuable resource.","major_comments":[{"comment":"The evaluation protocol uses ground-truth S_t throughout autoregressive rollout. Section 4.1 states \"we use the ground-truth weather state for S_t during training and evaluation,\" and Section 5 says boundary values are \"extracted from the ground truth\" while the coarse input is ERA5. For lead times beyond 6h, this supplies perfect lateral boundary/global forcing at every intermediate valid time, i.e., an oracle not available in operational forecasting. Absolute skill and model rankings in Figs. 1-3 and 4 are therefore conditional on perfect external information; a model that aggressively uses boundary corrections may look strong here yet degrade with realistic boundary errors. This is load-bearing for the \"common yardstick\" claim. Please add at least one sensitivity analysis with perturbed/persistent/forecast-derived boundary inputs (or a global-model forecast as S_t) to show rankings ar","section":"Sections 4.1, 5"},{"comment":"There is an inconsistency in the claimed parameter constraint: Section 5 (Training and evaluation details) says \"constrain the total parameter count of each baseline model to lie between 25M and 30M,\" while Appendix 9.1 says \"kept the parameter count of each architecture from 30 to 35 million.\" Since the paper claims fair comparison across architectures, the actual constraint must be stated unambiguously and verified in the released configurations. Please reconcile and report the actual parameter counts for each model.","section":"Sections 5 and 9.1"},{"comment":"The paper claims to provide a benchmark with \"diverse baselines\" across architectures and training objectives, including probabilistic forecasting. However, Section 10.3 states \"we only benchmark UNet with boundary forcing for probabilistic forecasting.\" Thus CRPS/SSR results are available for only one baseline, and no probabilistic comparison across architectures or conditioning strategies is provided. This undercuts the breadth of the benchmark as presented. Please add probabilistic results for at least one additional architecture, or explicitly scope the central claims to deterministic baselines while presenting the probabilistic component as a protocol definition and worked example.","section":"Section 10.3 and Abstract/Introduction"}],"minor_comments":[{"comment":"Diffusion-LAM is cited as [OLL23], but that reference is the Hi-LAM paper. The separate citation [Lar+25] appears later; please fix the citation for Diffusion-LAM.","section":"Section 2"},{"comment":"The x-axis is labeled 05-26 to 05-31, while the text says the period is May 25 to June 1. Please align the figure and text.","section":"Figure 4b"},{"comment":"The model name is spelled inconsistently as \"Graphcast\" and \"GraphCast\"; please standardize.","section":"Throughout"},{"comment":"Results are based on a single training run per configuration, with no error bars or seed variability. Given that some ranking differences appear small, a comment on this limitation or a multi-seed check for at least the central comparisons would strengthen the conclusions.","section":"Section 5"},{"comment":"Data is hosted on Google Drive; consider providing a persistent DOI or an institutional archive to ensure long-term access and reproducibility.","section":"Section 3.2 / data availability"}],"recommendation":"major_revision","confidential_remarks":"The ground-truth auxiliary input issue is the principal obstacle. If the authors can add a sensitivity experiment with imperfect boundary/global forcing, or clearly re-scope the benchmark claims, and reconcile the parameter-count inconsistency, I would support acceptance. The dataset and code release are valuable and in scope for the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Know this: IndiaWeatherBench is a genuinely useful asset. The authors ship a curated 12-km, 43-channel, 20-year subset of IMDAA with clean splits, two data formats, and four baseline architectures under two boundary conditioning strategies. That combination did not exist for India, and it will lower the entry cost for ML groups who want to work on regional forecasting without wrangling raw reanalysis.\n\nThe curation is where the paper earns its keep. The raw IMDAA is notoriously hard to access and use; the authors standardize it, document it, and release code and data. The baselines are reasonable and include deterministic and diffusion-based probabilistic variants, plus climatology and persistence comparisons. The split and metrics are standard, and the paper is honest about its main limitation: evaluation uses ground-truth boundary values and true ERA5 global state, so the numbers are an upper bound on what you would get with an operational global forecast driving the regional model. The stress-test note is right that this qualifies every headline figure, but it is disclosed in both Section 4.1 and Section 6, and it does not invalidate the benchmark for controlled model comparison.\n\nSoft spots are minor but real. The parameter count is stated as 25-30M in Section 5 and 30-35M in Appendix 9.1; that needs to be reconciled. Each baseline is a single training run, so ranking noise is unquantified; the paper should at least include seed variance for one or two models. The 'first benchmark' claim sits awkwardly next to BharatBench, though the 12-km resolution and full baseline suite genuinely set this effort apart. The biggest substantive gap is that the paper does not test what happens when the boundary/global input drifts, even by adding noise; a short experiment of that kind would make the benchmark much stronger.\n\nWho is this for? ML researchers and regional meteorology people who want a standard testbed for India. It is not a scientific-advance paper; it is an infrastructure paper, and it does its job. I would send it to peer review and ask for a revision that fixes the parameter-count inconsistency, adds seed variability, and either adds the noisy-boundary experiment or discusses its absence more directly. The raw material is solid.","headline":"A genuinely useful benchmark for regional ML forecasting over India, with an honest but load-bearing limitation: all headline numbers assume perfect boundary/global forcing.","tokens_in":18877,"tokens_out":3039,"would_cite":true,"duration_ms":36942,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"IndiaWeatherBench gives the ML community a standardized 12-km benchmark for forecasting weather over India.","keywords":["regional weather forecasting","benchmark","IMDAA","India","boundary conditioning","limited area modeling","machine learning","probabilistic forecasting"],"falsifier":"Run the same baselines with auxiliary inputs taken from a real operational global forecast instead of ground-truth reanalysis; if rankings reorder or Stormer's coarse-conditioning gap disappears, the evaluation protocol does not reflect operational skill.","tokens_in":17978,"feed_emoji":"🌦️","tokens_out":5268,"duration_ms":62091,"temperature":0.7,"pith_summary":"IndiaWeatherBench is a new benchmark, built on the IMDAA regional reanalysis, that gives the machine-learning community a common testbed for forecasting weather over India at 12-km resolution. It curates 20 years of 6-hourly atmospheric fields into a 256×256 grid, fixes train/validation/test splits, and supplies deterministic and probabilistic metrics. The paper also trains four baselines — UNet, Stormer, GraphCast, and a hierarchical graph model — under two ways of feeding in outside information: high-resolution boundary forcing and coarse global (ERA5) conditioning. The central finding is that model rankings depend on which conditioning strategy is used, so benchmarks must control for it. If the benchmark is adopted, regional forecasting studies over India become reproducible and comparable for the first time.","feed_headline":"IndiaWeatherBench gives forecasters one 12-km leaderboard for India","feed_subtitle":"Curated IMDAA data, fixed splits, and four baselines make regional AI weather models directly comparable.","key_machinery":"The load-bearing object is the curated IMDAA-derived dataset itself: 43 channels at 0.12° resolution over 6°N–36.72°N, 66.6°E–97.25°E, subsampled to 6-hourly steps from 2000–2019 with fixed splits, shipped in Zarr and HDF5. Around it, the benchmark standardizes the experimental protocol: prediction of state increments, latitude-weighted losses, two boundary conditioning strategies (a 10-pixel ground-truth boundary ring versus bilinearly upsampled 124×124 ERA5), matched model sizes, and four metrics. The combination is what makes results from different architectures and conditioning choices directly comparable.","core_discovery":"The paper establishes that regional machine-learning weather forecasting over India can be reduced to a standardized, reproducible task: given a 6-hourly history of 39 channels on a 12-km, 256×256 grid over the subcontinent, predict the next state, optionally conditioned on boundary or coarse-global context. Its benchmark results show that with boundary forcing, the transformer Stormer and graph model GraphCast lead across most variables and lead times, while the hierarchical graph model Hi underperforms GraphCast and UNet lags slightly; with coarse ERA5 conditioning, Stormer collapses to the worst model because its patch tokenization mixes incompatible spatial scales. Under a 2019 heatwave","pith_inferences":["If the benchmark is used with real-time operational global forecasts instead of ground-truth ERA5, the reported rankings may shift; the paper does not yet test this, but it is the natural next experiment.","Stormer's failure under coarse conditioning could be tested directly: separating the upsampled global input into its own token stream or using a larger patch size should either recover its skill or confirm the scale-mixing hypothesis.","The omission of precipitation metrics means the benchmark may miss the variable most critical to Indian agriculture; adding categorical or threshold-weighted precipitation scores could change which architecture is favored.","Fine resolution appears to matter: prior India benchmark data exist at about 1°, and models that look weak globally (e.g., UNet) stay competitive here, suggesting regional-scale data may alter architectural preferences."],"forward_implications":["Future regional forecasting papers on India can report numbers on the same splits and metrics, making leaderboard-style comparison possible.","The strong interaction between architecture and conditioning implies that a model's ranking without specifying boundary strategy is meaningless; future claims should state both.","Extreme-event scrutiny is built in: the heatwave case shows that aggregate RMSE rankings do not predict which model behaves best under rare extremes.","The probabilistic baseline gives a concrete starting point for uncertainty-aware regional forecasting; its underdispersion identifies a target for improvement.","Because the pipeline is modular, the same dataset construction and evaluation code can be transplanted to other regional reanalyses."],"supporting_citations":[{"why":"Supplies the IMDAA regional reanalysis that IndiaWeatherBench is built from.","marker":"[Ash+20]"},{"why":"Sets the benchmark conventions: 6-hourly subsampling, standardized splits, and metric suite.","marker":"[Ras+23]"},{"why":"Introduces Hi-LAM, the hierarchical graph architecture and boundary-forcing strategy used here.","marker":"[OLL23]"},{"why":"Provides Stormer, the transformer baseline whose tokenization is central to the conditioning results.","marker":"[Ngu+23c]"},{"why":"Provides GraphCast, a baseline architecture and the diffusion formulation adopted for probabilistic forecasts.","marker":"[Lam+23]"},{"why":"Provides ERA5, the coarse global reanalysis used for coarse-resolution conditioning.","marker":"[Her+20]"},{"why":"Prior India-focused IMDAA dataset; the paper distinguishes itself from this coarse-resolution effort.","marker":"[CPM24]"},{"why":"UNet architecture used as the convolutional baseline.","marker":"[RFB15]"},{"why":"EDM diffusion framework for the probabilistic training objective.","marker":"[Kar+22]"}],"fun_headline_variants":["A 12-km benchmark to settle India's AI weather forecast race","IndiaWeatherBench: standardizing regional AI weather models","New open benchmark ranks AI models for India's weather","Regional weather AI gets a fair playground: IndiaWeatherBench","Benchmarking 12-km AI forecasts over India, openly"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The rankings are computed with perfect auxiliary inputs—true boundary pixels or the true ERA5 global state—so they may not carry over to operational forecasts where those inputs contain real errors.","fun_headline_variants_meta":{"raw":{"variants":["A 12-km benchmark to settle India's AI weather forecast race","IndiaWeatherBench: standardizing regional AI weather models","New open benchmark ranks AI models for India's weather","Regional weather AI gets a fair playground: IndiaWeatherBench","Benchmarking 12-km AI forecasts over India, openly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000377,"raw_usage":{"total_tokens":1836,"prompt_tokens":727,"completion_tokens":1109,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":1025}},"tokens_in":471,"tokens_out":1109,"duration_ms":10717,"temperature":1.0,"reasoning_tokens":1025,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:20:17.517736+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same baselines with auxiliary inputs taken from a real operational global forecast instead of ground-truth reanalysis; if rankings reorder or Stormer's coarse-conditioning gap disappears, the evaluation protocol does not reflect operational skill.","supporting_citations":[],"review_version":1}