{"id":"fa4f9bb9-cf15-417e-80ce-798fc17af776","arxiv_id":"2605.24945","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"RealBench is a benchmark for data-driven weather forecasting that enforces operational conditions via a 2025 OOD test set, operational analysis, in-situ observations, and event-specific extreme metrics to expose gaps versus reanalysis-based evaluation.","lead":"RealBench introduces a benchmark for AI weather models that uses real-time operational analysis, 2025 out-of-distribution data, and direct in-situ observations from over 10,000 stations instead of reanalysis products. A smart generalist should read it to understand why current benchmark scores may not predict real-world forecasting reliability, especially for extreme events.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"Reader's weakest_assumption matches the only plausible load-bearing point visible from the abstract. Because the full text is referenced but yields no additional internal contradiction or missing justification that can be verified here, the UNVERDICTED stance is retained.","tokens_in":1763,"tokens_out":255,"duration_ms":17042,"concrete_test":"Re-run the headline extreme-event metrics from the paper using only the subset of stations with documented calibration metadata and >90% temporal coverage during the target events; if the reported discrepancy magnitude changes by more than 15% relative to the full station set, the completeness assumption materially affects the result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that RealBench's ground truth (low-latency operational analysis + >10k in-situ stations) is sufficiently complete and unbiased relative to reanalysis, and that the 2025 test window is strictly OOD. The abstract states these design choices explicitly and reports resulting discrepancies for extremes, but without the full manuscript's data-alignment protocols, station-error characterization, or model-training cutoffs, no internal inconsistency or unsupported assumption can be isolated from the provided text alone.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces RealBench, a benchmark for data-driven numerical weather forecasting that prioritizes operational realism over reanalysis-based evaluation. It defines a strictly out-of-distribution 2025 test window, combines low-latency operational analysis with >10k in-situ station observations as ground truth, and supplies event-specific metrics for extremes (heatwaves, cold surges, tropical cyclones). The central claim is that reanalysis-based benchmarks produce systematic mismatches with real-world performance, with RealBench revealing substantial discrepancies especially on extremes; the implementation is released on GitHub.","tokens_in":1822,"tokens_out":496,"duration_ms":18661,"significance":"If the reported discrepancies are reproducible and the ground-truth protocol is shown to be unbiased, the work would meaningfully shift evaluation standards in AI weather forecasting away from ERA5-style reanalysis toward operationally relevant targets. This could reduce over-optimism in published scores and encourage models that generalize under real-time data constraints. The public benchmark release is a concrete strength that enables community follow-up.","major_comments":[{"comment":"The abstract states that evaluation results reveal substantial discrepancies, yet the manuscript provides no quantitative tables, error bars, or statistical tests comparing reanalysis vs. RealBench metrics (e.g., no RMSE or event-specific scores for the 2025 period). This absence prevents verification of the central claim.","section":"Abstract / Evaluation section"},{"comment":"No protocol is described for aligning >10k in-situ station observations with model grid points or for characterizing station measurement error and representativeness (e.g., § on data sources). Without these details the assumption that the combined ground truth is complete and unbiased cannot be assessed.","section":"Data sources and evaluation framework"},{"comment":"The claim that the 2025 test set is strictly OOD requires explicit documentation of all model training data cutoffs and any reanalysis overlap; this information is not supplied, leaving the no-leakage guarantee unverified.","section":"Test-set construction"}],"minor_comments":[{"comment":"The GitHub link is provided but the repository contents (data loaders, exact metric implementations) are not referenced in the text; adding a pointer to specific scripts would improve reproducibility.","section":"Availability statement"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive comments, which highlight areas where additional detail will strengthen the manuscript's verifiability. We address each major comment below and commit to revisions that directly respond to the concerns raised.","responses":[{"response":"We agree that the absence of explicit quantitative comparisons limits immediate verification of the central claim. In the revised manuscript we will expand the evaluation section with tables that directly compare reanalysis-based and RealBench metrics on the 2025 period, including RMSE, event-specific scores, error bars, and statistical significance tests. These additions will be cross-referenced from the abstract.","revision_made":"yes","referee_comment":"[Abstract / Evaluation section] The abstract states that evaluation results reveal substantial discrepancies, yet the manuscript provides no quantitative tables, error bars, or statistical tests comparing reanalysis vs. RealBench metrics (e.g., no RMSE or event-specific scores for the 2025 period). This absence prevents verification of the central claim."},{"response":"We acknowledge that the current description of the ground-truth protocol is insufficient. The revised data-sources section will specify the alignment procedure (including interpolation method and grid-matching criteria) and will characterize station representativeness and measurement uncertainty by citing established protocols and, where feasible, providing quantitative bounds on error sources.","revision_made":"yes","referee_comment":"[Data sources and evaluation framework] No protocol is described for aligning >10k in-situ station observations with model grid points or for characterizing station measurement error and representativeness (e.g., § on data sources). Without these details the assumption that the combined ground truth is complete and unbiased cannot be assessed."},{"response":"We will add a dedicated subsection under test-set construction that tabulates the training-data cutoffs for every model included in the benchmark and explicitly states the absence of overlap with the 2025 window or with any reanalysis products used during training. This documentation will be supported by references to the original model papers and the released benchmark code.","revision_made":"yes","referee_comment":"[Test-set construction] The claim that the 2025 test set is strictly OOD requires explicit documentation of all model training data cutoffs and any reanalysis overlap; this information is not supplied, leaving the no-leakage guarantee unverified."}],"tokens_in":1460,"tokens_out":505,"duration_ms":22126,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"RealBench is the main point: it builds a benchmark that swaps reanalysis for low-latency operational analysis plus over 10,000 in-situ stations and uses a 2025 test window to stay out of distribution. The idea is to expose how standard benchmarks can overstate performance, especially on extremes.\n\nWhat is actually new is the specific package—2025 split, operational ground truth, large station set, and event-specific metrics for heatwaves, cold surges, and cyclones. The paper does a clean job spelling out why reanalysis creates a systematic mismatch with real forecasting constraints and why that matters for operational use. Code release on github helps with checking the implementation.\n\nThe soft spots sit in the results and methods. The abstract states substantial discrepancies without numbers, error analysis, or details on grid alignment and leakage controls, so the practical size of the effect is hard to gauge. The central assumption that the chosen ground truth is complete and unbiased needs explicit checks in the full text; if those are missing or weak, the discrepancies become harder to trust. Minor issues like station representativeness or latency effects could also matter but are secondary.\n\nThis is for people who select or develop AI weather models for real deployment and care about extreme-event skill. A reader working on operational systems or extremes would get the most from it.\n\nSend it for peer review. The motivation is practical and the benchmark design is concrete enough to be worth referee time, even if the quantitative claims will need close scrutiny.","headline":"RealBench pushes evaluation toward operational analysis and station data on a 2025 OOD window, but the abstract leaves the size of the claimed discrepancies and the alignment protocols unclear.","tokens_in":2340,"tokens_out":383,"would_cite":false,"duration_ms":34219,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"RealBench shows that reanalysis-based benchmarks for AI weather models produce systematically different results from real operational data, especially on extremes.","keywords":["AI weather forecasting","benchmark evaluation","operational conditions","extreme events","reanalysis mismatch","in-situ observations","data-driven models","tropical cyclones"],"falsifier":"If model rankings and absolute scores computed on reanalysis data for the same 2025 period turn out to be nearly identical to those computed on the RealBench operational and station data, the claimed systematic mismatch would be falsified.","tokens_in":2652,"feed_emoji":"🌦","tokens_out":735,"duration_ms":21402,"temperature":0.7,"pith_summary":"The paper introduces RealBench as a new evaluation framework for data-driven weather forecasting models. It argues that prior benchmarks rely on reanalysis products like ERA5, which use delayed data assimilation and therefore fail to match the constraints of actual real-time forecasting. RealBench instead uses a 2025 test period designed to be strictly out-of-distribution, combines low-latency operational analysis with direct in-situ measurements from more than 10,000 stations, and adds event-specific metrics for heatwaves, cold surges, and tropical cyclones. Evaluation with this setup produces substantially different performance numbers than reanalysis-based scoring, with the largest differences appearing during extreme events. A sympathetic reader would care because deployment decisions for AI forecasting systems depend on benchmarks that actually reflect the conditions under which those systems will be used.","feed_headline":"Benchmark shows reanalysis overestimates AI weather model skill on extremes","feed_subtitle":"RealBench evaluates on 2025 out-of-distribution data and 10,000+ station observations, revealing gaps that standard reanalysis metrics miss.","key_machinery":"RealBench benchmark, which supplies an out-of-distribution 2025 test set, multi-source operational ground truth, and extreme-event metrics to measure model performance directly against observations instead of reanalysis.","core_discovery":"RealBench is a benchmark that evaluates AI weather models against low-latency operational analysis and a global network of in-situ station observations on a 2025 test set chosen to avoid data leakage. It demonstrates that metrics computed from reanalysis products diverge from metrics computed from these real measurements, with the divergence especially pronounced when models are scored on extreme events using event-specific criteria rather than standard global averages.","pith_inferences":["Adoption of this style of benchmark could shift model development toward objectives that explicitly account for latency and observational sparsity.","The framework might be extended to include direct head-to-head comparisons between AI models and traditional physics-based numerical weather prediction systems under identical operational conditions.","If the discrepancies remain large, training pipelines may need to incorporate more recent or lower-latency data sources to close the gap."],"forward_implications":["Models that rank highly under reanalysis evaluation can rank differently when scored against operational data and in-situ observations.","Standard global metrics alone are insufficient for assessing value on high-impact extremes; event-specific scores are required.","Training and selection of data-driven forecasting systems should incorporate operational constraints to reduce the observed performance gap.","A benchmark built on real-time data sources provides a more relevant testbed for next-generation model development."],"fun_headline_variants":["RealBench shows reanalysis overestimates AI skill on extremes","Reanalysis overstates AI performance on real weather extremes","Benchmark reveals AI extremes gap between reanalysis and reality","RealBench highlights divergence in AI extreme event scoring"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The 2025 test period contains no data leakage from training sets and the combination of low-latency analysis plus station observations forms a sufficiently complete and unbiased ground truth.","fun_headline_variants_meta":{"raw":{"variants":["RealBench shows reanalysis overestimates AI skill on extremes","Reanalysis overstates AI performance on real weather extremes","Benchmark reveals AI extremes gap between reanalysis and reality","RealBench highlights divergence in AI extreme event scoring"]},"model":"grok-4.3","cost_usd":0.010845,"raw_usage":{"total_tokens":4718,"prompt_tokens":706,"num_sources_used":0,"completion_tokens":61,"cost_in_usd_ticks":108453000,"prompt_tokens_details":{"text_tokens":706,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3951,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":706,"tokens_out":61,"duration_ms":33626,"temperature":1.0,"reasoning_tokens":3951,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-30T11:51:19.815884+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"If model rankings and absolute scores computed on reanalysis data for the same 2025 period turn out to be nearly identical to those computed on the RealBench operational and station data, the claimed systematic mismatch would be falsified.","supporting_citations":[],"review_version":1}