REVIEW 4 major objections 5 minor 1 cited by
Uncertainty Quantification for Surface Ozone Emulators using Deep Learning
T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Uncertainty-aware deep learning estimates surface ozone model bias and maps where correcting that bias is hardest.
desk verdict Solid, modest UQ comparison for ozone emulators, but the central claim about uncertainty identifying correctable bias is unsupported by quantitative validation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is predictive uncertainty used as a proxy for bias-correction difficulty. MC Dropout estimates epistemic uncertainty through stochastic forward passes with dropout at test time; CQR produces conformalized prediction intervals from three quantile heads, and interval length is the uncertainty score. The U-Net is the shared architecture that maps multi-channel MOMO-Chem fields (28 or 51 channels) to per-pixel bias estimates. The argument works by showing that high-uncertainty pixels coincide with high bias and high RMSE in two independent UQ frameworks.
What would settle it
Take all TOAR stations in a region the paper labels high-uncertainty beyond dense station coverage—for example the interior U.S. East Coast or southeastern Europe—withhold them during training, and compare predictive uncertainty against the absolute bias residual at those withheld stations. If the rank correlation is near zero or negative, the extrapolated UQ maps are not a reliable basis for station selection.
Extended reading notes
Core claim
The central claim is that an uncertainty-aware U-Net, trained on MOMO-Chem atmospheric fields, produces surface-ozone bias predictions whose predictive uncertainty is informative about the difficulty of bias correction. The model predicts the residual between MOMO-Chem eight-hour ozone and TOAR ground-station observations, for regional grids over North America and Europe. Two UQ mechanisms are compared: MC Dropout, which treats dropout as a Bayesian approximation and yields epistemic uncertainty, and CQR, which uses three quantile heads (0.05, 0.5, 0.95) to form a 90% prediction interval whose length is the uncertainty score. On the June 2019 test set, RMSE is roughly 10 ppb for MC Dropout a
Load-bearing premise
The load-bearing premise is that predictive uncertainty computed at grid cells with no TOAR ground station still measures how hard local ozone bias is to correct, so extrapolated maps and station rankings remain trustworthy away from any training target.
Editorial extensions
If this is right
- If UQ scores mark difficulty, agencies can rank TOAR stations by uncertainty and spend bias-correction effort only on stations the emulator says are learnable.
- Spatial agreement between MC Dropout and CQR high-uncertainty regions gives a criterion for trusting the station ranking where the two methods coincide.
- The land-use feature experiments imply that adding more satellite inputs alone does not reliably reduce uncertainty; model improvement needs to address the noisy, high-bias regimes themselves.
- Extrapolated UQ maps, if trustworthy, provide a first-pass guide to where new ground-based ozone monitoring would be most informative.
- Because high uncertainty tracks high bias, the emulator can be used to flag regions where MOMO-Chem's underlying chemistry, emissions, or boundary-layer processes are least constrained.
Reading between the lines
- A direct extension is an active-learning loop: apply bias correction at high-UQ stations first and test whether RMSE drops faster than with random or low-UQ selection; this would turn the qualitative spatial agreement into a quantitative operational rule.
- The disagreement the paper observes between MC Dropout and CQR in extrapolated areas suggests a conservative principle: rely on UQ-based station rankings mainly where the two methods agree, and treat disagreement zones as places needing new observations rather than model fixes.
- The same uncertainty-as-difficulty criterion should transfer to other chemical species or other data-assimilation systems, since nothing in the argument is specific to ozone photochemistry.
- A quantitative version of the paper's station-selection claim would be a rank correlation between predictive interval length and station-level RMSE across all TOAR stations; the paper reports representative stations but does not compute this global rank statistic.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper trains U-Net emulators to predict MOMO-Chem surface ozone bias (model-minus-TOAR residuals) over North America and Europe for June 2019, with and without land-use features. It compares two UQ methods: Monte-Carlo Dropout (epistemic uncertainty) and Conformalized Quantile Regression (prediction interval length). The authors report RMSE and UQ metrics, show spatial UQ/RMSE maps, inspect two example stations, and extrapolate UQ beyond station coverage. They conclude that both UQ methods identify spatially consistent high- and low-difficulty regions, that land-use input reduces epistemic uncertainty in North America, and that UQ can discern optimal and sub-optimal stations for MOMO-Chem bias correction.
Significance. If supported, the result would provide a practical way to select surface stations for bias correction of a chemical transport model, and a useful comparison of two UQ approaches in an Earth-science emulator. The paper uses external TOAR targets, evaluates on a held-out June 2019 test set, and averages over five random seeds, which are strengths. However, the central claim currently rests on visual map agreement, with no quantitative UQ-error concordance, no empirical coverage check for CQR, and no statistical test for the land-use comparison. These gaps are fixable and the contribution could become solid after a major revision.
major comments (4)
- [Section 3 (Evaluation) and Section 4/Fig. 2] The central claim that UQ scores identify easy/difficult stations and regions is supported only by visual comparison of maps. There is no station-level quantitative test. I would require: (i) per-station Spearman correlation between mean absolute error and UQ score (CQR interval length, MCD epistemic variance) on the held-out June 2019 set; (ii) binned reliability of UQ versus error; (iii) empirical coverage of the nominal 90% CQR intervals. Without these, the Conclusion's statement that both techniques show overlap 'that match with regions of high bias' and the station-selection use case are unsupported.
- [Table 1] CQR RMSE is much larger than MCD RMSE (NA 28-feats: 18.40 vs 10.76; EU 28-feats: 16.50 vs 10.52), and this discrepancy is not discussed. If CQR point predictions are far worse, its interval lengths may be dominated by poor fit rather than by intrinsic difficulty, which confounds the UQ maps in Figs. 2-4. Please explain the gap (e.g., quantile loss optimization, calibration split size), report CQR empirical coverage, and verify that interval length adds information beyond the point-error magnitude.
- [Section 4, land-use results] The statement that 'inclusion of GEE data improves epistemic uncertainty in North America, but does not otherwise show significant improvement' is made without a significance test. The Table 1 seed intervals overlap (e.g., NA MCD 10.76±0.05 vs 10.88±0.29; EU CQR 16.50±0.06 vs 15.93±1.23). Please report paired tests across seeds or effect sizes with confidence intervals; 'significant' is otherwise unsupported.
- [Section 4, Spatial Extrapolation / Fig. 4] UQ maps are extrapolated beyond TOAR station coverage, and consistency between methods is used as evidence. This is not validation: away from training targets, both methods could share spatial smoothness artifacts. Please evaluate extrapolation by holding out spatial regions/stations, or by comparing extrapolated UQ against withheld station errors in those regions. The current analysis does not support the practical claim that UQ remains meaningful off-station.
minor comments (5)
- [Section 5] Typos: 'approximaton' and 'surfaze' should be corrected.
- [Figure 2] Colourbar scaling varies per metric, so visual 'overlap' between maps may be affected by arbitrary scaling. Consider using a common normalization or rank-based maps for comparability.
- [Table 1] Please clarify how Max/Min/Avg interval and epistemic values are computed over time and across seeds, and report the units.
- [Section 4, station selection] The selected max-UQ station had the highest UQ in only 2/5 seeds for CQR, and the min-UQ station in 1/5 seeds. Please report all seeds or a stability metric, since this bears directly on the station-selection use case.
- [Section 3, CQR setup] Please specify the calibration split (size, temporal ordering) and the conformal score used, for reproducibility.
Circularity Check
No significant circularity: predictions target external TOAR-derived bias and are evaluated on held-out 2019 data; the UQ-to-bias agreement is an empirical claim, albeit only qualitatively supported.
full rationale
The paper's target variable is surface ozone bias, defined as the residual between MOMO-Chem output and external TOAR station measurements (Section 2). Models are trained on 2005-2018 June data and evaluated on a held-out June 2019 test set with RMSE against that same external target. The UQ scores (MC-Dropout epistemic uncertainty and CQR interval length) are computed from predictive distributions and are not constructed from test-set error or from the bias maps used for comparison. The central claim that high-UQ regions overlap with high-bias regions is therefore an empirical observation, not an identity. The only self-references are citations to MOMO-Chem [2] (the external model being emulated) and the authors' own airPy package; neither substitutes for the derivation. The lack of a quantitative per-station UQ-error correlation or empirical coverage check is a real validation gap, but it concerns the strength of evidence, not circularity. No equation or fitted parameter is defined in terms of the claimed prediction, so the derivation chain is not circular.
Assumptions & free parameters
free parameters (3)
- MC Dropout rate =
0.1
- CQR target coverage =
90%
- Training hyperparameters =
learning rate 1e-3, 200 epochs, Adam
assumptions (4)
- domain assumption TOAR 8-hour daytime surface ozone measurements are treated as unbiased ground truth and representative of 11.1 km grid cells.
- domain assumption MOMO-Chem chemical and physical fields contain sufficient information to predict MOMO-Chem's own bias at unobserved locations.
- domain assumption MC Dropout with dropout rate 0.1 provides a meaningful epistemic uncertainty estimate.
- domain assumption The U-Net trained with station-derived targets generalizes to grid cells without TOAR coverage.
Cite this review
Pith. "Pith review of Uncertainty Quantification for Surface Ozone Emulators using Deep Learning." pith.science (2026). https://pith.science/paper/MDMADTYU
@misc{pith2026250804885,
author = {Pith},
title = {Pith review of: Uncertainty Quantification for Surface Ozone Emulators using Deep Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/MDMADTYU}},
note = {Machine review of arXiv:2508.04885}
}
read the original abstract
Air pollution is a global hazard, and as of 2023, 94\% of the world's population is exposed to unsafe pollution levels. Surface Ozone (O3), an important pollutant, and the drivers of its trends are difficult to model, and traditional physics-based models fall short in their practical use for scales relevant to human-health impacts. Deep Learning-based emulators have shown promise in capturing complex climate patterns, but overall lack the interpretability necessary to support critical decision making for policy changes and public health measures. We implement an uncertainty-aware U-Net architecture to predict the Multi-mOdel Multi-cOnstituent Chemical data assimilation (MOMO-Chem) model's surface ozone residuals (bias) using Bayesian and quantile regression methods. We demonstrate the capability of our techniques in regional estimation of bias in North America and Europe for June 2019. We highlight the uncertainty quantification (UQ) scores between our two UQ methodologies and discern which ground stations are optimal and sub-optimal candidates for MOMO-Chem bias correction, and evaluate the impact of land-use information in surface ozone residual modeling.
Figures
Forward citations
Cited by 1 Pith paper
-
Leveraging Deep Learning for Physical Model Bias of Global Air Quality Estimates
A CNN learns to correct surface ozone model bias, outperforming traditional ML and adding satellite land use as a useful input.
Reference graph
Works this paper leans on
-
[1]
Toward cleaner air and better health: Current state, challenges, and priorities
Wei Huang, Hongbing Xu, Jing Wu, Minghui Ren, Yang Ke, and Jie Qiao. Toward cleaner air and better health: Current state, challenges, and priorities. Science, 385(6707):386–390, 2024. doi: 10.1126/science.adp7832
-
[2]
K. Miyazaki, K. W. Bowman, K. Yumimoto, T. Walker, and K. Sudo. Evaluation of a multi- model, multi-constituent assimilation framework for tropospheric chemical reanalysis. Atmospheric Chemistry and Physics, 20(2):931–967, 2020. doi: 10.5194/acp-20-931-2020
-
[3]
Isabelle Bey, Daniel J. Jacob, Robert M. Yantosca, Jennifer A. Logan, Brendan D. Field, Ar- lene M. Fiore, Qinbin Li, Honguy Y . Liu, Loretta J. Mickley, and Mar- tin G. Schultz. Global modeling of tropospheric chemistry with assimilated meteo- rology: Model description and eval- uation. Journal of Geophysical Research: Atmo- spheres, 106(D19):23073–23095...
-
[4]
Chaser: A global chemical model of the troposphere 1
Kengo Sudo, Masaaki Takahashi, Jun-ichi Kurokawa, and Hajime Akimoto. Chaser: A global chemical model of the troposphere 1. model description. Journal of Geophysical Research: Atmo- spheres, 107(D17):ACH 7–1–ACH 7–20, 2002. doi: https://doi.org/10.1029/2001JD001113
-
[5]
S. Watanabe, T. Hajima, K. Sudo, T. Nagashima, T. Takemura, H. Okajima, T. Nozawa, H. Kawase, M. Abe, T. Yokohata, T. Ise, H. Sato, E. Kato, K. Takata, S. Emori, and M. Kawamiya. Miroc-esm 2010: model description and basic results of cmip5-20c3m experiments. Geoscientific Model Development, 4(4):845–872, 2011. doi: 10.5194/gmd-4-845-2011
-
[6]
A. Doury, S. Somot, and S. et al Gadat. Regional climate model emulator based on deep learning: concept and first evaluation of a novel hybrid downscaling approach. Climate Dynamics, 60: 1751–1779, 2023. doi: 10.1007/s00382-022-06343-9
-
[7]
D. Watson-Parris, Y . Rao, D. Olivi ´e, Ø. Seland, P. Nowack, and et al. Camps-Valls, G. Climatebench v1.0: A benchmark for data-driven climate projections. Journal of Advances in Modeling Earth Systems, (14), 2022. doi: 10.1029/2021MS002954
-
[8]
Uncertainty quantification and inter- pretability for clinical trial approval prediction
Lu Y , Chen T, Hao N, Van Rechem C, Chen J, and Fu T. Uncertainty quantification and inter- pretability for clinical trial approval prediction. Health Data Sci, 2024. doi: 10.34133/hds.0126
Show all 16 references
-
[9]
Un- certainty aware tropical cyclone wind speed estimation from satellite data, 2024
Nils Lehmann, Nina Maria Gottschling, Stefan Depeweg, and Eric Nalisnick. Un- certainty aware tropical cyclone wind speed estimation from satellite data, 2024. URL: https://arxiv.org/abs/2404.08325
2024 arXiv
-
[10]
Sulla-Menashe and M.A
D. Sulla-Menashe and M.A. Friedl. Mcd12q1 modis/terra+aqua land cover type yearly l3 global 500m sin grid v006
-
[11]
Gridded population of the world, version 4 (gpwv4): Population density, 2018
CIESIN. Gridded population of the world, version 4 (gpwv4): Population density, 2018
2018
-
[12]
Schultz, Sabine Schr¨oder, Olga Lyapina, Owen R
M.G. Schultz, Sabine Schr¨oder, Olga Lyapina, Owen R. Cooper, and Ian Galbally. Tropospheric ozone assessment report: Database and metrics data of global surface ozone observations. Elementa: Science of the Anthropocene, (5:58), 2017. doi: 10.1525/elementa.244
2017 doi
-
[13]
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of M...
2016
-
[14]
Cand ‘es
Yaniv Romano, Evan Patterson, and Emmanuel J. Cand ‘es. Conformalized quantile regression,
-
[15]
Aleatoric and epistemic uncertainty in machine learn- ing: A tutorial introduction
Eyke H¨ullermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learn- ing: A tutorial introduction. CoRR, abs/1910.09457, 2019. URL http://arxiv.org/abs/1910.09457. 6
1910 arXiv
-
[2019]
URL https://arxiv.org/abs/1905.03222
1905 arXiv
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.