Pith. sign in

REVIEW 3 major objections 4 minor 14 references

OCELOT shows that global weather forecasts can be learned directly from raw, heterogeneous observations — no reanalysis targets needed — and still produce coherent 12-hour fields with skill beyond persistence.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review

2026-08-02 03:00 UTC pith:3L222D7H

load-bearing objection A credible observation-space forecasting system with a genuinely held-out 2025 evaluation; the main claim needs one clarification about bias-corrected radiances before it is fully clean. the 3 major comments →

arxiv 2607.14196 v1 pith:3L222D7H submitted 2026-07-15 physics.ao-ph

OCELOT: Direct Atmospheric Forecasting from Heterogeneous Earth Observations Using a Graph-Transformer Hybrid Model

classification physics.ao-ph PACS 92.60.Wc
keywords observation-space forecastingreanalysis-independent weather predictiongraph attention networktransformerspherical icosahedral meshsatellite radiancesshort-range weather forecastingheterogeneous observations
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper sets out to establish that a global weather forecast model can be trained directly on raw, heterogeneous Earth observations—satellite radiances, radiosonde and aircraft profiles, scatterometer backscatter, surface reports—without ever using a gridded reanalysis or analysis state as a training target. If true, the payoff is that forecasting no longer depends on expensive reanalysis products that inherit the assumptions of numerical models and data-assimilation systems, and new observing systems can be added without re-projecting the world onto a fixed grid. The authors report that their model, OCELOT, produces spatially coherent +12 h forecasts across independent observing systems on 2025 data it never saw during training: microwave temperature-sounding channels show RMSE of 1.24–1.87 K, radiosonde and aircraft temperature profiles vary smoothly with pressure, and surface 2-m air temperature RMSE grows only from about 3.2 K at +3 h to 3.6 K at +12 h. In paired observation-space comparisons, OCELOT clearly beats same-location persistence at longer lead times for 2-m temperature and 10-m winds, while remaining less accurate than an operational, cycled data-assimilation forecast system.

Core claim

On the paper's own terms, OCELOT's central discovery is that observation-space forecasting is feasible and stable at global scale: a shared spherical latent mesh, fed by per-instrument graph-attention encoders and advanced by a hybrid temporal-transformer/spatial-graph processor, learns a common atmospheric evolution operator from raw measurements alone. The evidence is the held-out 2025 evaluation: independent microwave sounders agree with each other at +12 h (pooled RMSE 1.24–1.87 K for temperature-sounding channels), vertical temperature profiles from radiosondes and aircraft are coherent in pressure with no abrupt layer-wise jumps, and surface errors grow gradually rather than explode. T

What carries the argument

The load-bearing object is a fixed spherical icosahedral latent mesh—a sphere tessellated into triangles with 40,962 nodes and merged multiscale edges—that acts as the model's internal atmospheric state. Each instrument family has its own bipartite graph-attention encoder/decoder pair; encoders aggregate native observations onto nearby mesh nodes using edge features of great-circle distance and relative position, and decoders map the evolved mesh back to native observation locations. The processor interleaves sliding-window temporal self-attention at each node (window length 4, matching the four 3-hour rollout steps) with distance-weighted spatial message passing along mesh edges. Critical t

Load-bearing premise

The load-bearing premise is that a horizontal spherical latent mesh, with vertical position carried only by per-observation metadata, preserves enough vertical coupling to reconstruct physically coherent profiles and cross-instrument agreement; if that vertical compression scrambles the relationships, the claimed profile skill and shared-atmosphere interpretation collapse.

What would settle it

Run trained OCELOT v1 after (a) removing pressure-level conditioning from radiosonde/aircraft decoder queries or (b) adding noise to scan-angle metadata. If vertical profile RMSE becomes discontinuous in pressure and independent microwave sounders stop agreeing, the metadata-conditioning mechanism is load-bearing. A second decisive test: retrain with radiosonde observations removed from the input stream; if aircraft and microwave profile skill degrades significantly, the shared mesh is genuinely coupling instruments rather than each instrument being fit independently.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the central claim is right, reanalysis-free training is viable: raw archived observations alone can support a globally coherent short-range forecast model, removing a costly, assumption-laden dependency.
  • Cross-instrument skill agreement implies the latent mesh stores shared atmospheric-layer information, so on-boarding a new observing system should reduce to adding one encoder/decoder pair and retraining on the shared mesh.
  • Stable rollout with gradual, spatially coherent error growth indicates the learned latent dynamics are well-conditioned at 12 h, making longer horizons a scaling problem rather than a stability crisis.
  • The paper's attribution of the surface gap to missing static terrain/land-sea features and surface-sensitive channels predicts that adding those inputs should shrink near-surface RMSE—an easily checked near-term consequence.
  • Because each lead time is produced by an independently supervised head from one shared rollout, the +12 h numbers reflect direct prediction rather than feedback-amplified autoregression.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the 2D-latent-plus-metadata vertical compression is the most fragile design choice; perturbing or removing pressure-level/scan-angle conditioning in a controlled run would show whether vertical coherence is genuinely learned or merely memorized by the decoder.
  • Editorial inference: training on raw, QC'd observations without background checks means the model learns the observing network's actual climatology; major network changes (a satellite gap, a new sounder) may change skill in ways that reanalysis-trained models hide, making network-shift stress tests a natural next experiment.
  • Editorial inference: the instrument-rebalanced, channel-weighted MSE loss is an implicit policy for how much each sensor counts; retuning these weights toward specific variables (e.g., surface fields for severe weather) could reshape skill without any architectural change.
  • Editorial inference: the observation-space formulation may enable direct forecast-sensitivity observation impact estimates, since the model can be queried at native observation points without mapping to a gridded analysis—this is a concrete downstream use the paper mentions as future work, but the architecture already supports it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents OCELOT, a graph-transformer hybrid model that forecasts future Earth observations directly from raw, heterogeneous satellite and in-situ measurements, without training on gridded reanalysis. The model uses per-instrument GATv2 encoders/decoders, a shared 40,962-node icosahedral latent mesh, and a hybrid sliding-window Transformer/spatial-GNN processor. The model is trained on 2015–2023 observations, validated on 2024, and evaluated out-of-sample on 2025 with 730 initializations. Reported results include radiance RMSEs of 1.24–3.95 K, smooth vertical profile structure for radiosondes/aircraft, and surface 2-m temperature RMSEs growing from ~3.2 K at +3 h to ~3.6 K at +12 h. Comparisons against GFS show OCELOT trailing by 0.7–0.8 K for surface temperature, while outperforming a same-location persistence baseline at +12 h. The central claim is that observation-space forecasting can recover large-scale atmospheric structure and provide meaningful short-range skill without reanalysis supervision.

Significance. If the central claim holds, this is a significant proof-of-concept: a global forecasting system trained and verified entirely in observation space, with no reanalysis targets, that produces physically coherent fields and exceeds simple persistence. The paper provides several strengths: a genuinely out-of-sample 2025 evaluation period, public source code at a fixed release tag, explicit time splits (train 2015–2023, validation 2024, test 2025), and comparisons against GFS and persistence on a common observation subset. The architecture is modular and described in enough detail to be reproduced. The main uncertainty is not the architecture but the provenance of the satellite radiance targets; if those targets have been bias-corrected using model background information, the 'no reanalysis supervision' claim is weakened. The absence of reported bootstrap confidence intervals and the lack of a radiance persistence baseline also limit the strength of the quantitative claims as currently written.

major comments (3)
  1. [§3 and §9] The paper's central claim—'without reanalysis supervision'—depends on the satellite radiance targets being raw observations. The data are described as coming from 'operational NOAA GDAS observation dumps' and being 'quality-controlled,' but it is not stated whether the radiance values are raw or have undergone variational bias correction (VarBC). In GDAS, bias-corrected radiances are routinely written to observation dumps. If the Zarr archives contain bias-corrected radiances, the training targets already contain model/analysis information, directly undermining the reanalysis-independence claim. Please specify the exact preprocessing applied to each radiance stream and, if bias correction was applied, either remove the claim or re-state it with the qualification.
  2. [§6, 'Unless otherwise stated'] The text states that 'confidence intervals are estimated by bootstrap resampling over forecast initializations with replacement,' yet no confidence intervals are reported for any of the headline numbers (e.g., radiance RMSEs 1.24–1.87 K and 3.95 K, surface RMSE 3.2–3.6 K, wind RMSEs 2.53 and 2.61 m s⁻¹). Without these intervals, the reader cannot assess whether the differences between lead times or between OCELOT and persistence are statistically meaningful. Either provide the bootstrap intervals for at least the key pooled results, or remove the claim that bootstrap uncertainty estimates are being used.
  3. [§6.2 and Abstract] The abstract and §6.2 highlight radiance RMSEs as evidence of forecast skill, but no baseline is provided for these channels. RMSE values alone do not establish 'meaningful skill' unless compared to persistence, climatology, or some reference forecast. For the surface variables, persistence is used as a baseline; a similar persistence or persistence-of-observation baseline for the radiance channels should be added for the central claim to be supported across the full observing system. This is directly load-bearing for the statement that OCELOT provides 'short-range skill' across diverse observing systems.
minor comments (4)
  1. [§4.2] The 2D latent mesh with vertical information carried only through metadata is a notable design choice. The smooth vertical profiles in §6.3 are encouraging, but a sensitivity test or an explicit discussion of the risk of vertical decoupling (e.g., whether cross-instrument coherence could arise from shared horizontal structure alone) would strengthen the interpretation.
  2. [§6.1 vs §6.4] The text reports a single-initialization RMSE increase from 2.74 K to 3.23 K for 2025-04-25 in §6.4, while §6.1 reports pooled 3.2 K to 3.6 K. These are different quantities (one case vs pooled), and the distinction should be stated in the text to avoid confusion.
  3. [Table 1 and §6] Table 1 lists MSLP as a prognostic surface variable, but no MSLP results are reported in the evaluation. Please clarify whether MSLP was included in the final model run and, if so, report its skill or note that it was masked/inactive.
  4. [Minor typographical issues] Equation (3) has 'Where' capitalized, and the denominator formatting for √dk is slightly off. Also, the sentence in §3.2 'These filters retain observations...' could be clearer about the distinction between QC filtering and bias correction.

Circularity Check

0 steps flagged

No significant circularity: OCELOT's out-of-sample evaluation is independent of its training inputs.

full rationale

OCELOT's central claim is an empirical, out-of-sample result: a model trained on 2015-2023 observations and validated on 2024 is evaluated on fully held-out 2025 observations (§3.1, §6). The forecast equation (2) maps an input observation set to target observations; the loss (4)-(7) is a masked, instrument-rebalanced MSE in observation space. No parameter is fitted to the 2025 evaluation data, and no reported metric is constructed from a training target by definition. The GFS and persistence comparisons (§6.5) are external baselines on matched observations, and the persistence baseline is intentionally simple; outperforming it at +12 h is not a tautology. The consistency across independent microwave instruments is an emergent empirical property, not an architectural constraint, because decoders are instrument-specific and are only tied through the shared latent mesh. Citations to GraphDOP and other prior work provide context and architectural inspiration but are not used as proof of OCELOT's results. The paper's limitations (§8.1) explicitly disclaim untested physical conservation laws and controlled ablations, which are completeness caveats rather than circular steps. A possible data-provenance question—whether the GDAS archive radiances are bias-corrected before storage—could affect the 'no reanalysis supervision' claim, but the text does not state such processing, and this is a data-lineage concern, not an in-paper circular derivation.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 0 invented entities

The model's trainable weights are learned from data and not listed as free parameters, but the hyperparameters, normalization constants, loss weights, and mesh choice are hand-chosen or fitted and materially affect the reported skill. No new physical entities are introduced; the latent mesh is an internal representation, not a claimed physical object.

free parameters (5)
  • Latent mesh resolution (icosahedral k=6, 40,962 nodes) = k=6
    Chosen as an engineering compromise between expressiveness and cost; no controlled mesh-resolution ablation is reported (§4.2).
  • Core hyperparameters (hidden width 192, depth 4, W=4, 4 heads, dropout 0.1) = 192/4/4/4/0.1
    Hand-chosen; no ablation isolating their effect (§4, Table 2).
  • Channel weights wic and instrument rebalancing in the loss = not reported (all wic=1 reduces to plain masked MSE, §5.3)
    The loss is the training target; if nonuniform weights were used, they were fitted to improve validation and are not reported.
  • Per-instrument/per-variable normalization statistics (mean/std) = computed from 2015-2023
    Fitted to training data and held fixed; all forecasts depend on these constants (§3.2).
  • Configured pressure levels for profiles = 16 levels
    Radiosonde/aircraft targets are conditioned on nearest of 16 standard levels; the choice affects vertical profile error structure (§3.2, §4.4).
axioms (3)
  • domain assumption A 2D horizontal latent mesh plus metadata conditioning can carry enough vertical atmospheric information for coherent profile and radiance forecasts.
    Section 4.2 explicitly chooses not to use a 3D mesh and instead encodes vertical position/channel identity through features/metadata; the success of the central claim depends on this compression being sufficient.
  • domain assumption GDAS observation dumps and their quality-control flags provide unbiased, reanalysis-free training and verification data.
    Section 3.1/3.2 treats GDAS only as a QC'd observation source, but operational observation preprocessing can embed model-dependent QC decisions; the paper does not quantify this contamination.
  • domain assumption Learned encoder/decoder mappings can serve as observation operators without explicit radiative-transfer or instrument-physics constraints.
    The model replaces physical forward models with GATv2 projections; nothing in §4 enforces instrument physics, so cross-channel consistency is purely learned.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of OCELOT: Direct Atmospheric Forecasting from Heterogeneous Earth Observations Using a Graph-Transformer Hybrid Model." pith.science (2026). https://pith.science/paper/3L222D7H

@misc{pith2026260714196,
  author       = {Pith},
  title        = {Pith review of: OCELOT: Direct Atmospheric Forecasting from Heterogeneous Earth Observations Using a Graph-Transformer Hybrid Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3L222D7H}},
  note         = {Machine review of arXiv:2607.14196}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

This study presents OCELOT (Observation-Centric Estimation and Learning for Outlook Trajectories), a global machine-learning forecasting system that predicts future Earth observations directly from heterogeneous satellite and in-situ measurements. Unlike data-driven weather models trained on gridded reanalysis states, OCELOT operates natively in observation space, preserving instrument-specific sampling, viewing geometry, and measurement characteristics. The system combines per-instrument graph-attention encoders, a shared spherical icosahedral latent mesh, a hybrid sliding-window Transformer/spatial graph neural network processor, and metadata-conditioned decoders to produce forecasts up to 12 h ahead. OCELOT is trained on observations for the years 2015 through 2023, validated on the year 2024, and evaluated out of sample on 2025 observations across satellite radiances, radiosondes, aircraft, and surface networks. In the 2025 evaluation, OCELOT produces spatially coherent +12 h forecasts across independent observing systems: microwave temperature-sounding channels show RMSE values of 1.24-1.87 K, while the more surface- and cloud-sensitive AVHRR infrared window channel shows a higher RMSE of 3.95 K. Vertical profile diagnostics show physically consistent radiosonde and aircraft temperature structure. Surface forecasts remain stable through 12 h, with 2-m air-temperature RMSE increasing from about 3.2 K at +3 h to about 3.6 K at +12 h. In paired observation-space comparisons, OCELOT remains less accurate than operational GFS but substantially outperforms persistence at longer lead times for 2-m temperature and 10-m wind components. These results demonstrate that observation-space forecasting can recover large-scale atmospheric structure and provide meaningful short-range skill without reanalysis supervision.

Figures

Figures reproduced from arXiv: 2607.14196 by Andrew Collard, Azadeh Gholoubi, Cory Martin, Daniel Holdaway, Daryl Kleist, Mu-Chieh Ko, Nicholas Esposito, Ronald McLaren, Russ Treadon, Xin Jin.

Figure 1
Figure 1. Figure 1: Schematic of the OCELOT v1 architecture for direct, multi-instrument observation forecasting. (a) Heterogeneous observations from microwave sounders, passive microwave imagers, VIS/IR imagers, ASCAT, radiosondes, aircraft reports, and surface networks enter the model in their native sensor geometry. (b) [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Vertical profile diagnostics for OCELOT observation-space forecasts, pooled across the full held￾out 2025 evaluation set, consisting of 730 forecast initializations from the 00 UTC and 12 UTC cycles, and aggregated over the 0–12 h forecast window. Top row: mean profile of matched observations (solid blue) versus OCELOT predictions (dashed red). Bottom row: pooled RMSE (solid green) with ±1σ across-initiali… view at source ↗
Figure 4
Figure 4. Figure 4: Temporal rollout of surface 2-m air-temperature forecast errors. Panels (a)-(d) show OCELOT minus observation differences at +3, +6, +9, and +12 h for initialization 2025-04-25 00 UTC, using a common symmetric +/-10 K color scale. Panel (e) shows the lead-time evolution of error across the 2025 00 UTC and 12 UTC initialization cycles: thin grey lines denote per-initialization RMSE, the solid blue curve sho… view at source ↗
Figure 5
Figure 5. Figure 5: Observation-space comparison of surface 2-m air temperature at +12 h for the 2025-04-23 00 UTC and 12 UTC forecast initializations. Rows show the 00 UTC initialization verified at 2025-04-23 12 UTC and the 12 UTC initialization verified at 2025-04-24 00 UTC. Columns show OCELOT prediction, matched surface-observation truth, OCELOT − truth residual, and GFS − truth residual, with GFS interpolated to the mat… view at source ↗
Figure 6
Figure 6. Figure 6: Surface-observation RMSE for OCELOT, GFS, and a same-location persistence baseline, pooled over 730 2025 GFS-comparison initializations. Panels show 2-m air temperature, 10-m zonal wind, and 10-m meridional wind at +3, +6, +9, and +12 h lead times. All methods are evaluated on the same subset of observations for which the persistence baseline is available. Persistence uses the nearest prior same-location o… view at source ↗
Figure 7
Figure 7. Figure 7: Example same-date 00 UTC and 12 UTC mesh-space diagnostics for OCELOT 2-m air￾temperature forecasts verified against the GFS analysis. OCELOT is decoded on the fixed icosahedral mesh at +12 h, and the GFS analysis valid at the forecast verification time is interpolated to the same mesh nodes. Columns show OCELOT, GFS analysis at valid time, and OCELOT − analysis. Rows show the 2025-10-22 00 UTC and 12 UTC … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 2 canonical work pages

  1. [1]

    Nature, 641, 1172– 1179, https://doi.org/10.1038/s41586-025-08897-0

    Allen, A., and Coauthors, 2025: End-to-end data-driven weather prediction. Nature, 641, 1172– 1179, https://doi.org/10.1038/s41586-025-08897-0

  2. [2]

    arXiv, 2412.15687, https://doi.org/10.48550/arXiv.2412.15687

    Alexe, M., and Coauthors, 2024: GraphDOP: Towards skilful data-driven medium-range weather forecasts learnt and initialized directly from observations. arXiv, 2412.15687, https://doi.org/10.48550/arXiv.2412.15687

  3. [3]

    N., 2017: A review of operational methods of variational and ensemble-variational data assimilation

    Bannister, R. N., 2017: A review of operational methods of variational and ensemble-variational data assimilation. Quart. J. Roy. Meteor. Soc., 143, 607–633, https://doi.org/10.1002/qj.2982

  4. [4]

    Bi, K., L. Xie, H. Zhang, X. Chen, X. Gu, and Q. Tian, 2023: Accurate medium-range global weather forecasting with 3D neural networks. Nature, 619, 533–538, https://doi.org/10.1038/s41586-023-06185-3. 34

  5. [5]

    arXiv, 2510.20416, https://doi.org/10.48550/arXiv.2510.20416

    Boucher, E., and Coauthors, 2025: Learning coupled Earth System dynamics with GraphDOP. arXiv, 2510.20416, https://doi.org/10.48550/arXiv.2510.20416

  6. [6]

    Alon, and E

    Brody, S., U. Alon, and E. Yahav, 2022: How attentive are graph attention networks? Proc. Int. Conf. Learning Representations, https://doi.org/10.48550/arXiv.2105.14491

  7. [7]

    arXiv, 2304.02948, https://doi.org/10.48550/arXiv.2304.02948

    Chen, K., and Coauthors, 2023: FengWu: Pushing the skillful global medium-range weather forecast beyond 10 days lead. arXiv, 2304.02948, https://doi.org/10.48550/arXiv.2304.02948

  8. [8]

    Gelaro, R., R. H. Langland, S. Pellerin, and R. Todling, 2010: The THORPEX observation impact intercomparison experiment. Mon. Wea. Rev., 138, 4009–4025, https://doi.org/10.1175/2010MWR3393.1

  9. [9]

    Cambridge University Press, 341 pp

    Kalnay, E., 2003: Atmospheric Modeling, Data Assimilation and Predictability. Cambridge University Press, 341 pp

  10. [10]

    arXiv, 2202.07575, https://doi.org/10.48550/arXiv.2202.07575

    Keisler, R., 2022: Forecasting global weather with graph neural networks. arXiv, 2202.07575, https://doi.org/10.48550/arXiv.2202.07575

  11. [11]

    Science, 382, 1416–1421, https://doi.org/10.1126/science.adi2336

    Lam, R., and Coauthors, 2023: Learning skillful medium-range global weather forecasting. Science, 382, 1416–1421, https://doi.org/10.1126/science.adi2336

  12. [12]

    H., and N

    Langland, R. H., and N. L. Baker, 2004: Estimation of observation impact using the NRL atmospheric variational data-assimilation adjoint system. Tellus A, 56, 189–201, https://doi.org/10.3402/tellusa.v56i3.14413

  13. [13]

    Alexe, E

    Lean, P., M. Alexe, E. Boucher, E. Pinnington, S. Lang, P. Laloyaux, N. Bormann, and A. McNally, 2025: Learning from nature: Insights into GraphDOP’s representations of the Earth System. arXiv, 2508.18018, https://doi.org/10.48550/arXiv.2508.18018

  14. [14]

    arXiv, 2202.11214, https://doi.org/10.48550/arXiv.2202.11214

    Pathak, J., and Coauthors, 2022: FourCastNet: A global data-driven high-resolution weather model using adaptive Fourier neural operators. arXiv, 2202.11214, https://doi.org/10.48550/arXiv.2202.11214

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.