REVIEW 3 major objections 4 minor 14 references
OCELOT shows that global weather forecasts can be learned directly from raw, heterogeneous observations — no reanalysis targets needed — and still produce coherent 12-hour fields with skill beyond persistence.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
2026-08-02 03:00 UTC pith:3L222D7H
load-bearing objection A credible observation-space forecasting system with a genuinely held-out 2025 evaluation; the main claim needs one clarification about bias-corrected radiances before it is fully clean. the 3 major comments →
OCELOT: Direct Atmospheric Forecasting from Heterogeneous Earth Observations Using a Graph-Transformer Hybrid Model
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On the paper's own terms, OCELOT's central discovery is that observation-space forecasting is feasible and stable at global scale: a shared spherical latent mesh, fed by per-instrument graph-attention encoders and advanced by a hybrid temporal-transformer/spatial-graph processor, learns a common atmospheric evolution operator from raw measurements alone. The evidence is the held-out 2025 evaluation: independent microwave sounders agree with each other at +12 h (pooled RMSE 1.24–1.87 K for temperature-sounding channels), vertical temperature profiles from radiosondes and aircraft are coherent in pressure with no abrupt layer-wise jumps, and surface errors grow gradually rather than explode. T
What carries the argument
The load-bearing object is a fixed spherical icosahedral latent mesh—a sphere tessellated into triangles with 40,962 nodes and merged multiscale edges—that acts as the model's internal atmospheric state. Each instrument family has its own bipartite graph-attention encoder/decoder pair; encoders aggregate native observations onto nearby mesh nodes using edge features of great-circle distance and relative position, and decoders map the evolved mesh back to native observation locations. The processor interleaves sliding-window temporal self-attention at each node (window length 4, matching the four 3-hour rollout steps) with distance-weighted spatial message passing along mesh edges. Critical t
Load-bearing premise
The load-bearing premise is that a horizontal spherical latent mesh, with vertical position carried only by per-observation metadata, preserves enough vertical coupling to reconstruct physically coherent profiles and cross-instrument agreement; if that vertical compression scrambles the relationships, the claimed profile skill and shared-atmosphere interpretation collapse.
What would settle it
Run trained OCELOT v1 after (a) removing pressure-level conditioning from radiosonde/aircraft decoder queries or (b) adding noise to scan-angle metadata. If vertical profile RMSE becomes discontinuous in pressure and independent microwave sounders stop agreeing, the metadata-conditioning mechanism is load-bearing. A second decisive test: retrain with radiosonde observations removed from the input stream; if aircraft and microwave profile skill degrades significantly, the shared mesh is genuinely coupling instruments rather than each instrument being fit independently.
If this is right
- If the central claim is right, reanalysis-free training is viable: raw archived observations alone can support a globally coherent short-range forecast model, removing a costly, assumption-laden dependency.
- Cross-instrument skill agreement implies the latent mesh stores shared atmospheric-layer information, so on-boarding a new observing system should reduce to adding one encoder/decoder pair and retraining on the shared mesh.
- Stable rollout with gradual, spatially coherent error growth indicates the learned latent dynamics are well-conditioned at 12 h, making longer horizons a scaling problem rather than a stability crisis.
- The paper's attribution of the surface gap to missing static terrain/land-sea features and surface-sensitive channels predicts that adding those inputs should shrink near-surface RMSE—an easily checked near-term consequence.
- Because each lead time is produced by an independently supervised head from one shared rollout, the +12 h numbers reflect direct prediction rather than feedback-amplified autoregression.
Where Pith is reading between the lines
- Editorial inference: the 2D-latent-plus-metadata vertical compression is the most fragile design choice; perturbing or removing pressure-level/scan-angle conditioning in a controlled run would show whether vertical coherence is genuinely learned or merely memorized by the decoder.
- Editorial inference: training on raw, QC'd observations without background checks means the model learns the observing network's actual climatology; major network changes (a satellite gap, a new sounder) may change skill in ways that reanalysis-trained models hide, making network-shift stress tests a natural next experiment.
- Editorial inference: the instrument-rebalanced, channel-weighted MSE loss is an implicit policy for how much each sensor counts; retuning these weights toward specific variables (e.g., surface fields for severe weather) could reshape skill without any architectural change.
- Editorial inference: the observation-space formulation may enable direct forecast-sensitivity observation impact estimates, since the model can be queried at native observation points without mapping to a gridded analysis—this is a concrete downstream use the paper mentions as future work, but the architecture already supports it.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents OCELOT, a graph-transformer hybrid model that forecasts future Earth observations directly from raw, heterogeneous satellite and in-situ measurements, without training on gridded reanalysis. The model uses per-instrument GATv2 encoders/decoders, a shared 40,962-node icosahedral latent mesh, and a hybrid sliding-window Transformer/spatial-GNN processor. The model is trained on 2015–2023 observations, validated on 2024, and evaluated out-of-sample on 2025 with 730 initializations. Reported results include radiance RMSEs of 1.24–3.95 K, smooth vertical profile structure for radiosondes/aircraft, and surface 2-m temperature RMSEs growing from ~3.2 K at +3 h to ~3.6 K at +12 h. Comparisons against GFS show OCELOT trailing by 0.7–0.8 K for surface temperature, while outperforming a same-location persistence baseline at +12 h. The central claim is that observation-space forecasting can recover large-scale atmospheric structure and provide meaningful short-range skill without reanalysis supervision.
Significance. If the central claim holds, this is a significant proof-of-concept: a global forecasting system trained and verified entirely in observation space, with no reanalysis targets, that produces physically coherent fields and exceeds simple persistence. The paper provides several strengths: a genuinely out-of-sample 2025 evaluation period, public source code at a fixed release tag, explicit time splits (train 2015–2023, validation 2024, test 2025), and comparisons against GFS and persistence on a common observation subset. The architecture is modular and described in enough detail to be reproduced. The main uncertainty is not the architecture but the provenance of the satellite radiance targets; if those targets have been bias-corrected using model background information, the 'no reanalysis supervision' claim is weakened. The absence of reported bootstrap confidence intervals and the lack of a radiance persistence baseline also limit the strength of the quantitative claims as currently written.
major comments (3)
- [§3 and §9] The paper's central claim—'without reanalysis supervision'—depends on the satellite radiance targets being raw observations. The data are described as coming from 'operational NOAA GDAS observation dumps' and being 'quality-controlled,' but it is not stated whether the radiance values are raw or have undergone variational bias correction (VarBC). In GDAS, bias-corrected radiances are routinely written to observation dumps. If the Zarr archives contain bias-corrected radiances, the training targets already contain model/analysis information, directly undermining the reanalysis-independence claim. Please specify the exact preprocessing applied to each radiance stream and, if bias correction was applied, either remove the claim or re-state it with the qualification.
- [§6, 'Unless otherwise stated'] The text states that 'confidence intervals are estimated by bootstrap resampling over forecast initializations with replacement,' yet no confidence intervals are reported for any of the headline numbers (e.g., radiance RMSEs 1.24–1.87 K and 3.95 K, surface RMSE 3.2–3.6 K, wind RMSEs 2.53 and 2.61 m s⁻¹). Without these intervals, the reader cannot assess whether the differences between lead times or between OCELOT and persistence are statistically meaningful. Either provide the bootstrap intervals for at least the key pooled results, or remove the claim that bootstrap uncertainty estimates are being used.
- [§6.2 and Abstract] The abstract and §6.2 highlight radiance RMSEs as evidence of forecast skill, but no baseline is provided for these channels. RMSE values alone do not establish 'meaningful skill' unless compared to persistence, climatology, or some reference forecast. For the surface variables, persistence is used as a baseline; a similar persistence or persistence-of-observation baseline for the radiance channels should be added for the central claim to be supported across the full observing system. This is directly load-bearing for the statement that OCELOT provides 'short-range skill' across diverse observing systems.
minor comments (4)
- [§4.2] The 2D latent mesh with vertical information carried only through metadata is a notable design choice. The smooth vertical profiles in §6.3 are encouraging, but a sensitivity test or an explicit discussion of the risk of vertical decoupling (e.g., whether cross-instrument coherence could arise from shared horizontal structure alone) would strengthen the interpretation.
- [§6.1 vs §6.4] The text reports a single-initialization RMSE increase from 2.74 K to 3.23 K for 2025-04-25 in §6.4, while §6.1 reports pooled 3.2 K to 3.6 K. These are different quantities (one case vs pooled), and the distinction should be stated in the text to avoid confusion.
- [Table 1 and §6] Table 1 lists MSLP as a prognostic surface variable, but no MSLP results are reported in the evaluation. Please clarify whether MSLP was included in the final model run and, if so, report its skill or note that it was masked/inactive.
- [Minor typographical issues] Equation (3) has 'Where' capitalized, and the denominator formatting for √dk is slightly off. Also, the sentence in §3.2 'These filters retain observations...' could be clearer about the distinction between QC filtering and bias correction.
Circularity Check
No significant circularity: OCELOT's out-of-sample evaluation is independent of its training inputs.
full rationale
OCELOT's central claim is an empirical, out-of-sample result: a model trained on 2015-2023 observations and validated on 2024 is evaluated on fully held-out 2025 observations (§3.1, §6). The forecast equation (2) maps an input observation set to target observations; the loss (4)-(7) is a masked, instrument-rebalanced MSE in observation space. No parameter is fitted to the 2025 evaluation data, and no reported metric is constructed from a training target by definition. The GFS and persistence comparisons (§6.5) are external baselines on matched observations, and the persistence baseline is intentionally simple; outperforming it at +12 h is not a tautology. The consistency across independent microwave instruments is an emergent empirical property, not an architectural constraint, because decoders are instrument-specific and are only tied through the shared latent mesh. Citations to GraphDOP and other prior work provide context and architectural inspiration but are not used as proof of OCELOT's results. The paper's limitations (§8.1) explicitly disclaim untested physical conservation laws and controlled ablations, which are completeness caveats rather than circular steps. A possible data-provenance question—whether the GDAS archive radiances are bias-corrected before storage—could affect the 'no reanalysis supervision' claim, but the text does not state such processing, and this is a data-lineage concern, not an in-paper circular derivation.
Axiom & Free-Parameter Ledger
free parameters (5)
- Latent mesh resolution (icosahedral k=6, 40,962 nodes) =
k=6
- Core hyperparameters (hidden width 192, depth 4, W=4, 4 heads, dropout 0.1) =
192/4/4/4/0.1
- Channel weights wic and instrument rebalancing in the loss =
not reported (all wic=1 reduces to plain masked MSE, §5.3)
- Per-instrument/per-variable normalization statistics (mean/std) =
computed from 2015-2023
- Configured pressure levels for profiles =
16 levels
axioms (3)
- domain assumption A 2D horizontal latent mesh plus metadata conditioning can carry enough vertical atmospheric information for coherent profile and radiance forecasts.
- domain assumption GDAS observation dumps and their quality-control flags provide unbiased, reanalysis-free training and verification data.
- domain assumption Learned encoder/decoder mappings can serve as observation operators without explicit radiative-transfer or instrument-physics constraints.
Cite this review
Pith. "Pith review of OCELOT: Direct Atmospheric Forecasting from Heterogeneous Earth Observations Using a Graph-Transformer Hybrid Model." pith.science (2026). https://pith.science/paper/3L222D7H
@misc{pith2026260714196,
author = {Pith},
title = {Pith review of: OCELOT: Direct Atmospheric Forecasting from Heterogeneous Earth Observations Using a Graph-Transformer Hybrid Model},
year = {2026},
howpublished = {\url{https://pith.science/paper/3L222D7H}},
note = {Machine review of arXiv:2607.14196}
}
read the original abstract
This study presents OCELOT (Observation-Centric Estimation and Learning for Outlook Trajectories), a global machine-learning forecasting system that predicts future Earth observations directly from heterogeneous satellite and in-situ measurements. Unlike data-driven weather models trained on gridded reanalysis states, OCELOT operates natively in observation space, preserving instrument-specific sampling, viewing geometry, and measurement characteristics. The system combines per-instrument graph-attention encoders, a shared spherical icosahedral latent mesh, a hybrid sliding-window Transformer/spatial graph neural network processor, and metadata-conditioned decoders to produce forecasts up to 12 h ahead. OCELOT is trained on observations for the years 2015 through 2023, validated on the year 2024, and evaluated out of sample on 2025 observations across satellite radiances, radiosondes, aircraft, and surface networks. In the 2025 evaluation, OCELOT produces spatially coherent +12 h forecasts across independent observing systems: microwave temperature-sounding channels show RMSE values of 1.24-1.87 K, while the more surface- and cloud-sensitive AVHRR infrared window channel shows a higher RMSE of 3.95 K. Vertical profile diagnostics show physically consistent radiosonde and aircraft temperature structure. Surface forecasts remain stable through 12 h, with 2-m air-temperature RMSE increasing from about 3.2 K at +3 h to about 3.6 K at +12 h. In paired observation-space comparisons, OCELOT remains less accurate than operational GFS but substantially outperforms persistence at longer lead times for 2-m temperature and 10-m wind components. These results demonstrate that observation-space forecasting can recover large-scale atmospheric structure and provide meaningful short-range skill without reanalysis supervision.
Figures
Reference graph
Works this paper leans on
-
[1]
Nature, 641, 1172– 1179, https://doi.org/10.1038/s41586-025-08897-0
Allen, A., and Coauthors, 2025: End-to-end data-driven weather prediction. Nature, 641, 1172– 1179, https://doi.org/10.1038/s41586-025-08897-0
-
[2]
arXiv, 2412.15687, https://doi.org/10.48550/arXiv.2412.15687
Alexe, M., and Coauthors, 2024: GraphDOP: Towards skilful data-driven medium-range weather forecasts learnt and initialized directly from observations. arXiv, 2412.15687, https://doi.org/10.48550/arXiv.2412.15687
-
[3]
N., 2017: A review of operational methods of variational and ensemble-variational data assimilation
Bannister, R. N., 2017: A review of operational methods of variational and ensemble-variational data assimilation. Quart. J. Roy. Meteor. Soc., 143, 607–633, https://doi.org/10.1002/qj.2982
doi:10.1002/qj.2982 2017
-
[4]
Bi, K., L. Xie, H. Zhang, X. Chen, X. Gu, and Q. Tian, 2023: Accurate medium-range global weather forecasting with 3D neural networks. Nature, 619, 533–538, https://doi.org/10.1038/s41586-023-06185-3. 34
-
[5]
arXiv, 2510.20416, https://doi.org/10.48550/arXiv.2510.20416
Boucher, E., and Coauthors, 2025: Learning coupled Earth System dynamics with GraphDOP. arXiv, 2510.20416, https://doi.org/10.48550/arXiv.2510.20416
-
[6]
Brody, S., U. Alon, and E. Yahav, 2022: How attentive are graph attention networks? Proc. Int. Conf. Learning Representations, https://doi.org/10.48550/arXiv.2105.14491
-
[7]
arXiv, 2304.02948, https://doi.org/10.48550/arXiv.2304.02948
Chen, K., and Coauthors, 2023: FengWu: Pushing the skillful global medium-range weather forecast beyond 10 days lead. arXiv, 2304.02948, https://doi.org/10.48550/arXiv.2304.02948
-
[8]
Gelaro, R., R. H. Langland, S. Pellerin, and R. Todling, 2010: The THORPEX observation impact intercomparison experiment. Mon. Wea. Rev., 138, 4009–4025, https://doi.org/10.1175/2010MWR3393.1
-
[9]
Cambridge University Press, 341 pp
Kalnay, E., 2003: Atmospheric Modeling, Data Assimilation and Predictability. Cambridge University Press, 341 pp
2003
-
[10]
arXiv, 2202.07575, https://doi.org/10.48550/arXiv.2202.07575
Keisler, R., 2022: Forecasting global weather with graph neural networks. arXiv, 2202.07575, https://doi.org/10.48550/arXiv.2202.07575
-
[11]
Science, 382, 1416–1421, https://doi.org/10.1126/science.adi2336
Lam, R., and Coauthors, 2023: Learning skillful medium-range global weather forecasting. Science, 382, 1416–1421, https://doi.org/10.1126/science.adi2336
-
[12]
Langland, R. H., and N. L. Baker, 2004: Estimation of observation impact using the NRL atmospheric variational data-assimilation adjoint system. Tellus A, 56, 189–201, https://doi.org/10.3402/tellusa.v56i3.14413
-
[13]
Lean, P., M. Alexe, E. Boucher, E. Pinnington, S. Lang, P. Laloyaux, N. Bormann, and A. McNally, 2025: Learning from nature: Insights into GraphDOP’s representations of the Earth System. arXiv, 2508.18018, https://doi.org/10.48550/arXiv.2508.18018
-
[14]
arXiv, 2202.11214, https://doi.org/10.48550/arXiv.2202.11214
Pathak, J., and Coauthors, 2022: FourCastNet: A global data-driven high-resolution weather model using adaptive Fourier neural operators. arXiv, 2202.11214, https://doi.org/10.48550/arXiv.2202.11214
This paper was first reviewed by deepseek-v4-flash on August 2, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.