{"id":"282dae19-1756-44b2-b1ea-54f065d975ba","arxiv_id":"2412.18239","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"An observation-only AI model trained on the RTMA analysis reports lower 0-12h surface forecast errors than operational HRRR and IFS-HRES over the contiguous US.","lead":"OMG-HD, a Microsoft-built AI model, forecasts surface weather over the United States directly from station, satellite, and radar observations, reporting lower errors than operational models like HRRR and ECMWF's IFS for the first 12 hours. The paper argues that end-to-end machine learning forecasting can bypass traditional data assimilation while remaining competitive or better for short-range predictions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation is circular: OMG-HD is trained on RTMA and verified against RTMA, and the station-based 'hold-out' evaluation does not break circularity because RTMA assimilates the same station observations used for verification.","rationale":"The reader correctly identified the RTMA training–verification circularity as the weakest assumption. My analysis extends that concern: the station-based evaluation, which the reader regarded as providing 'some independent support,' is not truly independent because RTMA assimilates the same surface stations used for verification. Thus, the hold-out station test does not break the circularity as cleanly as implied. This does not overturn the paper's potential value—the architecture and end-to-end concept are interesting, and the masked-input ablation shows genuine robustness—but it raises the correctness risk. The appropriate verdict remains CONDITIONAL: the claim can be accepted only after verification against a truly independent observational dataset. I therefore recommend no change to the reader's verdict, hence UNCHANGED. My agreement is partial because I go further than the reader by showing that the secondary evaluation is also affected by the training-label coupling, not merely secondary in importance.","tokens_in":14205,"tokens_out":5357,"duration_ms":52994,"concrete_test":"Evaluate OMG-HD and HRRR against an observational network that is independent of RTMA's assimilation, such as the U.S. Climate Reference Network (USCRN) stations, over the same test period (the 1st–3rd of each month in 2023). Compute RMSE for 2-m temperature, 10-m wind speed, and specific humidity at USCRN locations, using only stations that are not part of the METAR/ISD network feeding RTMA. If OMG-HD's advantage over HRRR persists on these fully withheld observations, the headline claim is supported. If the advantage shrinks or disappears, the reported gains are an artifact of training and verifying on the same analysis product.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that OMG-HD outperforms IFS-HRES and HRRR at lead times up to 12 hours over CONUS—rests on two evaluations, both of which are coupled to the training labels. Section A.1 states that the labels used to train OMG-HD are derived from RTMA, and the primary evaluation in Fig. 2a verifies forecasts against RTMA. Since RTMA is the direct training target, any model optimized to match RTMA will show artificially low RMSE against RTMA compared with models that were not trained on it. This is the reader's concern, but it goes deeper. The supposedly independent station evaluation (Fig. 2b,c) selects about 1,600 'high-quality stations' from the same network of over 18,000 stations that provide input observations. RTMA is a real-time mesoscale analysis that assimilates available surface observations, necessarily including these very stations. Therefore, the station observations used for verification were already assimilated into the RTMA labels during training. The hold-out mechanism (Section 3.3.1) excludes those stations from direct training as station labels, but their values still influenced the RTMA labels through assimilation; it is not a clean out-of-sample test. The Q variable is further derived from RH, T, and SP, and for the hold-out evaluation SP is replaced with RTMA SP, re-introducing the training analysis. Thus, both reported comparisons are tainted by label–verification coupling. The abstract's 'up to 13%' wording is also inconsistent with the text's statement that these are averages over lead times, but that is a minor overstatement compared with the circularity. The claim may still be true, but the evidence as presented does not demonstrate it. The only way to settle this is to verify against observations that were not ingested into RTMA at all.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OMG-HD, a high-resolution AI weather forecasting model that takes raw observations (surface stations, GOES-16 satellite channels, MRMS radar, topography, time encodings) over the CONUS region and produces 12-hour forecasts of 2-meter temperature, 10-meter wind speed, 2-meter specific humidity, and surface pressure. The architecture consists of an Assimilating Block, which converts sparse multi-source observations into a gridded latent state, and a Forecasting Block, which generates autoregressive 6-hour forecasts. The model is trained end-to-end with losses against RTMA labels. The evaluation compares OMG-HD against HRRR, IFS-HRES, and GFS using both RTMA-gridded verification and station observations, with additional experiments on hold-out stations, masked input stations, and two case studies of extreme events. The central claim is that OMG-HD outperforms operational NWP models at lead times up to 12 hours across CONUS.","tokens_in":14481,"tokens_out":5445,"duration_ms":53510,"significance":"If the reported results are sound, the paper would demonstrate a noteworthy proof of concept: an AI model trained and run directly from observations can compete with operational high-resolution NWP systems for short-range surface forecasting over a large domain. The architectural choices are described in detail, the data sources and training procedure are specified, and the masking ablation addresses an operationally relevant failure mode. The case studies are illustrative and the authors are candid in the discussion about limitations such as the CONUS-only scope, the use of RTMA labels, and the lack of boundary conditions. However, the main significance is conditional: the headline comparison is measured against RTMA, which is also the training target, so the reported margins over HRRR and IFS-HRES are not an independent measure of forecast skill. The hold-out station evaluation does not fully resolve this problem, as the station observations used for verification are part of the same observational network that feeds the RTMA labels.","major_comments":[{"comment":"The primary evaluation in Fig. 2a verifies forecasts against RTMA, and Section A.1 states that RTMA is used to derive the training labels. Because the optimization losses L1, L2, and L3 are computed directly against RTMA, the RMSE of OMG-HD against RTMA is not an unbiased comparison with HRRR, IFS-HRES, or GFS, none of which were trained on RTMA. The reported 13%, 17%, 48%, and 32% improvements over HRRR are therefore partly a measure of how well OMG-HD fits the RTMA analysis, not an independent measure of forecast quality. The paper should either provide an evaluation against a truth dataset that was not used as a training label, or explicitly reframe the claim as skill relative to the RTMA analysis rather than a general statement of forecast superiority.","section":"Section 3.1 and Fig. 2a"},{"comment":"The hold-out station test does not break the label-verification coupling. The hold-out stations are selected from the same ISD/Synoptic station network that supplies the input observations, and RTMA assimilates those station reports into the gridded labels used for training. Excluding a station from the direct station-label loss does not remove its influence from the RTMA target at that location, because the RTMA analysis was generated with those very observations. Furthermore, for the hold-out evaluation the variable Q is derived from RH, T, and SP, and Section A.1 states that RTMA SP is used as a substitute for station SP, which re-introduces the training analysis into the verification. Thus Fig. 2c does not provide the independent out-of-sample evidence needed to support the generalization claim.","section":"Section 3.3.1 and Fig. 2c, with Section A.1"},{"comment":"The comparison of the embedded assimilation state in Fig. 5 is also evaluated against RTMA labels, so it inherits the same circularity as Fig. 2a. The claim that OMG-HD produces a more accurate initial state than the baselines is essentially a statement that the model's gridded output, trained to match RTMA, is closer to RTMA than the NWP model backgrounds. This may be true, but it does not demonstrate that the latent state is meteorologically superior in an independent sense; the figure should be interpreted with that caveat or replaced with a comparison using independent observations.","section":"Section 3.2 and Fig. 5"}],"minor_comments":[{"comment":"The abstract says 'up to a 13% improvement' for 2-meter temperature, while Section 3.1 says the 13% is the performance gain 'averaged over lead times.' These are different quantities, and since the RMSE varies with lead time in Fig. 2a, the abstract should state whether the percentage is a lead-time-averaged value or a maximum, and should identify the lead times at which the improvement is not achieved.","section":"Abstract and Section 3.1"},{"comment":"The subsection header contains a typo: 'F orecasting block' should read 'Forecasting block.'","section":"Section A.2.1"},{"comment":"The text refers to 'Digital Evaluation Models (DEMs)' in Section A.1; this should be 'Digital Elevation Models.'","section":"Section A.2.1 and Table 2"},{"comment":"The main RMSE plots show no confidence intervals or significance tests. Given that the headline claim is about outperforming operational NWP models, reporting uncertainty intervals or at least the number of verification times would help assess whether the differences are robust.","section":"Figure 2"},{"comment":"The paper says 'The labels used to train OMG-HD are derived from the Real-Time Mesoscale Analysis (RTMA)' and later notes that RTMA has 'small discrepancy from raw observations.' This is a useful caveat, but the evaluation sections should reference it directly when interpreting the RTMA-based results, since the discrepancy is exactly what makes the training-verification coupling consequential.","section":"Section A.1"}],"recommendation":"major_revision","confidential_remarks":"The central evaluation is circular in a way that is likely to be recognized by reviewers in the field: the model is trained on RTMA and the headline skill scores are computed against RTMA. The station evaluation, including the hold-out variant, does not cleanly separate the model from its training target because the station observations are assimilated into RTMA. I would not reject the manuscript on this basis, because the architecture, the direct-observations approach, and the robustness experiments are potentially valuable. The authors should be asked to provide a genuinely independent verification, for example using station variables that are not assimilated into the training labels, or using a held-out temporal period and a truth source independent of RTMA. If such an evaluation is not feasible, the claims need to be substantially narrowed and the paper reframed as a demonstration of fitting an observation-to-RTMA mapping rather than as a forecast-skill benchmark. The paper also compares only briefly with related end-to-end models; a more careful positioning against Aardvark, FengWu-4DVar, and similar systems would help the reader assess novelty."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"OMG-HD is a serious attempt at a hard problem: an end-to-end AI forecast system that takes in station, satellite, and radar data and outputs surface fields at 0.05° over CONUS. The model design is sensible — Swin transformer for assimilation, AFNO for forecasting, trained end-to-end — and the ablation with masked station input is a good robustness check. If the headline result were solid, beating HRRR and IFS-HRES at 12h lead times would be a meaningful advance. But the evaluation doesn't yet support the headline.\n\nThe primary comparison in Fig. 2a verifies against RTMA, and RTMA is also the training label. That is a textbook circular benchmark: the model has been fitted to that exact field, the baselines have not. The station evaluation in Fig. 2b,c is better but not clean. RTMA assimilates the very stations used for verification, so those values have already influenced the labels the model was trained on. For Q, the station value is derived from RH, T, and SP, and in the hold-out set SP is taken from RTMA, so the training analysis leaks in again. I agree with the stress-test note: there is no truly out-of-sample verification in this paper.\n\nThere are also smaller issues. The abstract says 'up to 13%' while the text says the average improvement is 13%; those are inconsistent. No significance tests or confidence intervals are shown, and no code or weights are released, so the reader has to take the RMSE curves on faith. The authors are honest about the model's limits — CONUS only, surface variables only, no boundary conditions — and that helps.\n\nThe central claim may well be true. An observation-only model might genuinely beat operational NWP for the first 12 hours. But the evidence as presented does not demonstrate it. A referee should ask the authors to verify against a dataset that was not ingested into RTMA — an independent station network, or an analysis that withholds those stations — and to release at least the inference code or weights. This deserves peer review, but it needs major revision before the claim can be taken seriously.","headline":"A promising E2E AI forecast model whose headline claim is undercut by a circular RTMA evaluation; the idea is worth a serious referee but the evidence needs independent verification.","tokens_in":15161,"tokens_out":3300,"would_cite":true,"duration_ms":30755,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":["92.60.Wc"],"model":"deepseek-v4-flash","headline":"A model trained only on raw observations outperforms two operational weather systems for 12-hour surface forecasts across the US.","keywords":["AI weather prediction","end-to-end forecasting","observational data","OMG-HD","data assimilation","regional forecasting","RTMA","surface variables"],"falsifier":"Evaluate forecasts from OMG-HD and HRRR against raw station observations for all four variables at stations withheld from both training and the RTMA analysis; if the reported RMSE margin shrinks below statistical significance, the claimed advantage would be, at least in part, an artifact of the verification target.","tokens_in":13965,"feed_emoji":"🌦️","tokens_out":4344,"duration_ms":39491,"temperature":0.7,"pith_summary":"This paper claims that a kilometer-scale weather forecast can be made directly from raw observations—surface stations, radar, and satellite channels—without any numerical weather model or data assimilation step. It introduces OMG-HD, a two-block neural network that first converts six hours of heterogeneous observations into a gridded atmospheric state and then steps that state forward in time. Evaluated against RTMA and station data, OMG-HD reports lower RMSE than HRRR and IFS-HRES for 2-m temperature, wind speed, specific humidity, and surface pressure, with gains up to 48% for humidity. If correct, this would mean an operational forecast system could be updated in real time from incoming observations, bypassing the latency and information loss of traditional data assimilation.","feed_headline":"Observation-only AI model beats top weather forecasters","feed_subtitle":"OMG-HD cuts out data assimilation and beats ECMWF and HRRR on 12-hour surface forecasts over the US.","key_machinery":"The central mechanism is the two-block architecture: an Assimilating Block built on Shifted-Window Transformer (Swin) layers fuses the past six hours of station, radar, satellite, time-encoding, and topography inputs into a gridded 26-channel initial state, with 6 channels supervised against RTMA surface variables and 20 free latent channels; a Forecasting Block using Adaptive Fourier Neural Operator (AFNO) layers then advances this state in 6-hour autoregressive steps, with a combined loss L1+L2+L3 supervising the analyzed state and the 6- and 12-hour forecasts against RTMA. This learned initial state plays the role of a data-assimilation analysis, constructed not by solving observation operators but by end-to-end training with the forecast objective.","core_discovery":"OMG-HD establishes, on the paper's own terms, that an end-to-end AI weather prediction model can learn to make kilometer-scale, twelve-hour forecasts directly from raw observations—station measurements, radar reflectivity, and satellite channels—without using any NWP output as input or as a training target. The model learns its own assimilation step in a dedicated Assimilating Block, mapping sparse heterogenous observations into a dense 26-channel gridded state whose hidden channels encode information such as upper-air conditions, and then makes autoregressive forecasts with a Fourier-based Forecasting Block. Compared against the operational HRRR and IFS-HRES baselines over CONUS, the paper reports average RMSE reductions of 13% for 2-m temperature, 17% for 10-m wind speed, 48% for 2-m specific humidity, and 32% for surface pressure, and shows the model remains competitive even when up to 50% of input stations are masked and on hold-out stations not seen during training.","pith_inferences":["A natural next test is to run the same architecture over a global domain with full satellite coverage, to see whether the learned analysis generalizes beyond CONUS and beyond 12-hour lead times.","Because OMG-HD was trained on RTMA and verified against RTMA, an independent evaluation against raw station observations—especially for surface pressure, which was partly replaced by RTMA values—would separate true forecast skill from fit to the analysis.","The strong humidity improvement suggests the satellite and radar channels carry moisture information that station-only inputs miss; an ablation isolating each input source could identify which channel drives the gain and guide future observation targeting."],"forward_implications":["Short-range operational forecasts could be produced directly from new observations within minutes, without waiting for an NWP cycle with data assimilation.","AI weather prediction need not depend on reanalysis products for input or initialization; model updates could track instrument networks in real time.","The 20 latent channels in the analyzed state suggest a learned analysis can carry information about the atmosphere beyond the supervised surface variables.","If the result holds under independent verification, it sets a new baseline that operational short-range regional systems would be measured against."],"supporting_citations":[{"why":"Defines RTMA, the analysis product used as training labels and as the primary verification truth.","marker":"[24]"},{"why":"Describes the HRRR operational model that serves as the main baseline OMG-HD must outperform.","marker":"[27, 28]"},{"why":"Documents the IFS-HRES model cycle used as the second operational baseline.","marker":"[26]"},{"why":"Provides the WeatherReal-Synoptic station dataset used for training and evaluation.","marker":"[41]"},{"why":"Supplies the Integrated Surface Database as an additional station observation source.","marker":"[42]"},{"why":"Introduces AFNO, the Fourier-based spatial mixing used in the Forecasting Block.","marker":"[23]"},{"why":"Introduces Swin Transformer V2, the backbone of the Assimilating Block.","marker":"[48]"},{"why":"Describes the hold-out station methodology used to test generalization to unseen locations.","marker":"[21]"}],"fun_headline_variants":["OMG-HD: AI forecasts straight from raw weather data, beats HRRR","AI weather model ditches data assimilation, tops ECMWF and HRRR","Forecasts from raw observations: OMG-HD outperforms operational models","No reanalysis needed: OMG-HD predicts weather directly from sensors","AI model uses stations, radar, satellite—beats HRRR by up to 48%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results are measured against RTMA, the same analysis product OMG-HD was trained on, so the comparison assumes RTMA is an unbiased ground truth that does not systematically favor a model fitted to it.","fun_headline_variants_meta":{"raw":{"variants":["OMG-HD: AI forecasts straight from raw weather data, beats HRRR","AI weather model ditches data assimilation, tops ECMWF and HRRR","Forecasts from raw observations: OMG-HD outperforms operational models","No reanalysis needed: OMG-HD predicts weather directly from sensors","AI model uses stations, radar, satellite—beats HRRR by up to 48%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000942,"raw_usage":{"total_tokens":4060,"prompt_tokens":1012,"completion_tokens":3048,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":2945}},"tokens_in":628,"tokens_out":3048,"duration_ms":19770,"temperature":1.0,"reasoning_tokens":2945,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:53:24.200365+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate forecasts from OMG-HD and HRRR against raw station observations for all four variables at stations withheld from both training and the RTMA analysis; if the reported RMSE margin shrinks below statistical significance, the claimed advantage would be, at least in part, an artifact of the verification target.","supporting_citations":[{"cited_title":"Weather and Forecasting26(5), 593–612 (2011)","cited_arxiv_id":null,"evidence_quote":"Defines RTMA, the analysis product used as training labels and as the primary verification truth."},{"cited_title":"ECMWF Newsletter 147, 14–19 (2016)","cited_arxiv_id":null,"evidence_quote":"Documents the IFS-HRES model cycle used as the second operational baseline."},{"cited_title":"Bulletin of the American Meteorological Society 92(6), 704–708 (2011)","cited_arxiv_id":null,"evidence_quote":"Supplies the Integrated Surface Database as an additional station observation source."},{"cited_title":"In: International Conference on Learning Representations (2021)","cited_arxiv_id":null,"evidence_quote":"Introduces AFNO, the Fourier-based spatial mixing used in the Forecasting Block."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp","cited_arxiv_id":null,"evidence_quote":"Introduces Swin Transformer V2, the backbone of the Assimilating Block."}],"review_version":1}