Pith. sign in

REVIEW 2 major objections 6 minor 33 references

Probabilistic Emulation of the Community Radiative Transfer Model Using Machine Learning

T0 review · 2 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A neural network emulates the GOES ABI radiative transfer model with 0.3 K RMSE while predicting its own error, which the authors argue is enough to begin using it to assimilate far more satellite observations.

desk verdict A solid, incremental probabilistic CRTM emulator for GOES ABI, but the random temporal split and tiny test set leave the headline accuracy claims under-supported for true generalization. read the letter →

arxiv 2504.16192 v1 pith:Z4FRCENF submitted 2025-04-22 physics.ao-ph cs.LGstat.ML

Lucas Howard , Aneesh C. Subramanian , Gregory Thompson , Benjamin Johnson , Thomas Auligne This is my paper · ORCID
classification physics.ao-phcs.LGstat.ML
keywords CRTMGOESABIneuralnetworkemulatorprobabilisticpredictionradiativetransferdataassimilationSHAPbrightnesstemperature
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a fully connected neural network can emulate the Community Radiative Transfer Model for the GOES ABI instrument with accuracy comparable to existing deterministic emulators, while additionally predicting its own error. The emulator reaches 0.3 K RMSE averaged over channels, below 0.1 K for 9 of 10 infrared channels in clear sky, and produces error estimates that are reliable over the range where most samples fall. The authors position this as a step toward assimilating far more ABI observations, since only 0.02% are currently used, and they use SHAP analysis to argue the network has learned physically meaningful relationships.

What carries the argument

The central object is a fully connected neural network with three hidden layers of 512 nodes, trained with the continuous ranked probability score (CRPS) as the loss function. The network has a dual output head: a sigmoid-scaled brightness temperature and a standard deviation produced by $\delta + \alpha \cdot \text{softmax}(x)$, where $\delta$ sets a floor of 0.001 K. Training on 121 six-hourly GFS/CRTM scans and tuning on 15 validation scans yields the probabilistic emulator; SHAP (Shapley additive explanations) is then applied to 1000 clear-sky and 1000 cloudy pixels to attribute output changes to input variables, serving as a physical consistency check.

What would settle it

Run the trained emulator on CRTM/GFS data from a different month or year, such as July 2022, and compute the per-channel RMSE against CRTM; if the average RMSE exceeds 0.3 K by a large margin, the generalization claim is refuted.

Watch

Extended reading notes

Core claim

The central claim is that a probabilistic neural network can serve as a fast, accurate surrogate for CRTM for the GOES ABI, outputting both a predicted brightness temperature and a predicted error standard deviation for channels 7–16. On a held-out test set the network matches CRTM to within 0.3 K RMSE overall, and under clear skies the RMSE is below 0.1 K for nine of the ten infrared channels. The predicted error standard deviations are well calibrated for the bulk of the error distribution, with a systematic underprediction only for the largest, rarest errors. The network runs about five times faster than CRTM on a single CPU, and SHAP attributions show water vapor, ozone, and solar zenith angle affecting the expected channels, which the authors take as evidence the network learned physics rather than memorizing the training set.

Load-bearing premise

The random split of 151 consecutive six-hourly scans into training, validation, and test sets treats weather states separated by six hours as independent, so the reported accuracy may overstate performance on truly independent atmospheric states.

Editorial extensions

If this is right

  • If the emulator's accuracy holds in operational settings, assimilating ABI observations could increase from 0.02% to a much larger fraction, since the bottleneck is observation-operator cost.
  • The probabilistic output allows a thresholding strategy: use the neural operator only where its predicted error is small and fall back to CRTM elsewhere.
  • The roughly fivefold speed-up scales with cloudiness, since CRTM is slower for cloudy scenes, so the benefit grows exactly where current all-sky assimilation is most expensive.
  • The reliable error predictions could feed observation-error covariance specifications in data assimilation, and the tropospheric Jacobian agreement suggests adjoint-based DA methods could use the emulator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Pith inference: If the network's error calibration holds across seasons, the same architecture could be retrained for other instruments or hyperspectral sensors, where the channel count makes CRTM cost prohibitive.
  • Pith inference: The monotonic calibration even at large errors suggests a post-hoc recalibration could extend reliable error estimates to the tail, making the threshold approach safer than the paper claims.
  • Pith inference: Because the training data span only one month, the SHAP attribution of solar zenith angle on channel 7 should be tested on data from a different time of year, where the sun-angle range differs.
  • Pith inference: The divergence of Jacobians above 200 hPa could be remedied by adding stratospheric water vapor variability to the training distribution, a testable extension.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper proposes a fully connected neural network emulator of CRTM for GOES ABI channels 7-16. The network takes 1,166 atmospheric/surface/metadata inputs and outputs predicted brightness temperatures plus per-channel predictive standard deviations. Training data are 151 simulated GFS/CRTM scans from February 15 to March 15, 2022, split randomly into 121/15/15 train/validation/test scans. The model is trained with a CRPS loss, and the authors report a 0.3 K all-channel RMSE, clear-sky RMSE below 0.1 K for 9 of 10 IR channels, a speedup of about 5x over CRTM on a single CPU, and SHAP analyses showing physically plausible feature attributions. A Jacobian comparison for three water vapor channels is also presented.

Significance. The study addresses a relevant operational bottleneck and offers a practical probabilistic emulator with a public code/data repository (Howard 2024) and evaluation against an external benchmark (CRTM), which are strengths. The calibration analysis (Fig. 3) and the distinction between RMSE and normalized RMSE are thoughtful. If the generalization evidence were strengthened, the work would make a solid contribution to fast radiative transfer for data assimilation.

major comments (2)
  1. [§2d, §3a, Figs. 2-3] The central generalization claim rests on a random split of only 151 scans from one 30-day period of 6-hourly GFS states. Synoptic atmospheric states are autocorrelated on timescales longer than 6 h, so random scan-level splitting can place nearly identical states in training and test; the reported 0.3 K all-channel RMSE and <0.1 K clear-sky RMSE therefore likely estimate interpolation within the sampled February-March regime, not performance on genuinely new weather. The 15-scan test set is too small to give stable estimates, and no confidence intervals are reported for any RMSE value. A temporally blocked split (e.g., contiguous time blocks) or an evaluation on a separate season is needed before the abstract's claim that the emulator 'will perform well when presented with new data' is justified.
  2. [§3a, Fig. 6] The Jacobian comparison is computed for a single scan and shows a 'significant divergence' from CRTM above 200 hPa for the water vapor channels. Since the paper itself states that stable, accurate Jacobians are critical for DA, this divergence is a substantive limitation rather than a cosmetic one; the statement that the divergence is unimportant because water vapor is scarce at those levels is plausible but not demonstrated (e.g., no analysis of the Jacobians' impact on radiance assimilation increments). A quantitative, multi-scan Jacobian error metric, or an explicit narrowing of the claimed validity range, is required for the conclusion that the emulator 'reproduces the relevant physics.'
minor comments (6)
  1. [§3b] The figure callouts are wrong: the water vapor SHAP profiles appear in Figure 7, not Figure 5; the stacked bars for atmospheric variables are Figure 8, not Figure 6; and the meta-variable plot is Figure 9, not Figure 8.
  2. [Conclusion vs §3a] 'Roughly an order of magnitude faster' is inconsistent with the factor of about 5 shown in Figure 5; please align these numbers.
  3. [Table 1] The surface variables list 'Soil Temperature' twice; the duplicate should be removed or renamed.
  4. [Discussion] The sentence 'The probabilistic prediction is of little use. However, if the predictions are not sufficiently accurate...' is incomplete and needs rewriting.
  5. [§2e and Figs. 8-9] The method name 'Shapely' should be 'Shapley,' and the axis label 'Man(|SHAP|)' should be 'Mean(|SHAP|).'
  6. [Table 2] The description 'brigthness temperature normalization' contains a typo; it should be 'brightness temperature normalization.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central accuracy claim is benchmarked against CRTM on a held-out test set, and the probabilistic and XAI analyses are self-evaluative but not circular reductions.

full rationale

The paper's central claim is that a neural network emulates CRTM for ABI with an RMSE of 0.3 K and reliable error estimates. This is assessed against CRTM-generated brightness temperatures on a held-out test split, which is an external benchmark for the emulator rather than an input constructed from the emulator's own outputs. The NN is trained to minimize CRPS against CRTM targets, and the predicted error standard deviations are evaluated by comparing them with actual NN-CRTM residuals on the test set; this is a standard predictive assessment and does not reduce to a self-definition or a fitted parameter renamed as a prediction. The SHAP/XAI analysis is an interpretation of the trained model itself, so it cannot independently verify the physics, but the paper uses it only as additional plausibility and not as the basis for the accuracy numbers. The self-citations, such as Johnson et al. (2023) and Howard et al. (2024), describe CRTM capabilities and probabilistic NN methods but are not load-bearing in the sense of forcing the paper's conclusions by citation alone. The random temporal split of 151 scans is a generalization and data-independence concern, not circularity, because the test targets are still CRTM outputs not seen during training. No enumerated circularity pattern is therefore present.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests entirely on fitted neural-network parameters and on domain assumptions about CRTM, the representativeness of the GFS training data, and the validity of random splitting and SHAP attribution. No new physical entities or forces are introduced.

free parameters (4)
  • Neural network weights (3x512 dense layers) = Not reported
    The emulator's predictions are entirely determined by these trained weights; they are fitted to CRTM output via Adam/CRPS.
  • Output normalization bounds β_min, β_max = 180 K, 355 K
    Chosen by hand to bound the sigmoid output; affects the scale and gradient of the output layer.
  • Minimum predicted standard deviation δ = 0.001 K
    Chosen by hand to stabilize training; prevents zero-variance predictions.
  • Hyperparameters (depth, width, L2, learning rate, batch size) = Table 2
    Manually tuned on validation performance; not fitted to data but selected by the authors.
assumptions (5)
  • domain assumption CRTM output is a valid reference for ABI brightness temperatures.
    The emulator is trained and evaluated against CRTM, so the utility of the emulator inherits CRTM's accuracy and error characteristics.
  • domain assumption GFS FV3 forecast fields are representative of the atmospheric states that ABI observes.
    Training data are 30 days of simulated scans from one season; representativeness is assumed for operational use.
  • domain assumption Random scan-level splitting yields independent training and test samples.
    Six-hourly scans are likely temporally correlated, so test error may be underestimated.
  • domain assumption The CRPS-trained network gives calibrated predictive distributions.
    CRPS is a proper scoring rule, but the finite data and model capacity assumptions are implicit.
  • standard math SHAP attributions faithfully represent the network's behavior.
    SHAP is an additive feature attribution method; its validity depends on model linearity assumptions that are only approximate for a dense NN.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Probabilistic Emulation of the Community Radiative Transfer Model Using Machine Learning." pith.science (2026). https://pith.science/paper/Z4FRCENF

@misc{pith2026250416192,
  author       = {Pith},
  title        = {Pith review of: Probabilistic Emulation of the Community Radiative Transfer Model Using Machine Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z4FRCENF}},
  note         = {Machine review of arXiv:2504.16192}
}
read the original abstract

The continuous improvement in weather forecast skill over the past several decades is largely due to the increasing quantity of available satellite observations and their assimilation into operational forecast systems. Assimilating these observations requires observation operators in the form of radiative transfer models. Significant efforts have been dedicated to enhancing the computational efficiency of these models. Computational cost remains a bottleneck, and a large fraction of available data goes unused for assimilation. To address this, we used machine learning to build an efficient neural network based probabilistic emulator of the Community Radiative Transfer Model (CRTM), applied to the GOES Advanced Baseline Imager. The trained NN emulator predicts brightness temperatures output by CRTM and the corresponding error with respect to CRTM. RMSE of the predicted brightness temperature is 0.3 K averaged across all channels. For clear sky conditions, the RMSE is less than 0.1 K for 9 out of 10 infrared channels. The error predictions are generally reliable across a wide range of conditions. Explainable AI methods demonstrate that the trained emulator reproduces the relevant physics, increasing confidence that the model will perform well when presented with new data.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 10 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archive author booktitle chapter doi edition editor eid eprint howpublished department institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 '...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION if.digit duplicate "0" = swap duplicate "1" = swap duplicate "2" = swap duplicate "3" = swap duplicate "4" = swap duplicate "5" = swap duplicate "6" = swap duplicate "7" = swap duplicate "8" = swap "9" = or or or or or or or or or FUNCTION n.separate 't := "" #0 'numnames := t empty not t #-1 #1 subs...

  3. [3]

    Abarbanel, H. D. I., P. J. Rozdeba, and S. Shirman, 2018: Machine Learning : Deepest Learning as Statistical Data Assimilation Problems . Neural Computation, 30 (8), 2025--2055, doi:10.1162/neco_a_01094

    doi:10.1162/neco_a_01094 2018

  4. [4]

    N., 2017: A review of operational methods of variational and ensemble-variational data assimilation

    Bannister, R. N., 2017: A review of operational methods of variational and ensemble-variational data assimilation. Quarterly Journal of the Royal Meteorological Society, 143 (703), 607--633, doi:10.1002/qj.2982

    doi:10.1002/qj.2982 2017

  5. [5]

    Barnes, E. A., R. J. Barnes, and M. DeMaria, 2023: Sinh-arcsinh-normal distributions to add uncertainty to neural network regression tasks: Applications to tropical cyclone intensity forecasts. Environmental Data Science, 2, e15, doi:10.1017/eds.2023.7

    work page doi:10.1017/eds.2023.7 2023

  6. [6]

    Thorpe, and G

    Bauer, P., A. Thorpe, and G. Brunet, 2015: The quiet revolution of numerical weather prediction. Nature, 525 (7567), 47--55, doi:10.1038/nature14956

    doi:10.1038/nature14956 2015

  7. [7]

    Bulletin of the American Meteorological Society, 102 (4), E710--E716, doi:10.1175/BAMS-D-20-0307.1

    Bonavita, M., and Coauthors, 2021: Machine Learning for Earth System Observation and Prediction . Bulletin of the American Meteorological Society, 102 (4), E710--E716, doi:10.1175/BAMS-D-20-0307.1

    doi:10.1175/bams-d-20-0307.1 2021

  8. [8]

    Du, and J

    Buehner, M., P. Du, and J. Bédard, 2018: A New Approach for Estimating the Observation Impact in Ensemble – Variational Data Assimilation . Monthly Weather Review, 146 (2), 447--465, doi:10.1175/MWR-D-17-0252.1

    work page doi:10.1175/mwr-d-17-0252.1 2018

Show all 33 references
  1. [9]

    Chapman, W. E., L. D. Monache, S. Alessandrini, A. C. Subramanian, F. M. Ralph, S.-P. Xie, S. Lerch, and N. Hayatbini, 2022: Probabilistic Predictions from Deterministic Atmospheric River Forecasts with Deep Learning . Monthly Weather Review, 150 (1), 215--234, doi:10.1175/MWR...

    2022 doi

  2. [10]

    ://keras.io

    Chollet, F., and others , 2015: Keras. ://keras.io

    2015

  3. [11]

    Edwards, C. A., A. M. Moore, I. Hoteit, and B. D. Cornuelle, 2015: Regional Ocean Data Assimilation . Annual Review of Marine Science, 7 (1), 21--42, doi:10.1146/annurev-marine-010814-015821

    2015 doi

  4. [12]

    Eyre, J. R., W. Bell, J. Cotton, S. J. English, M. Forsythe, S. B. Healy, and E. G. Pavelin, 2022: Assimilation of satellite data in numerical weather prediction. Part II : Recent years. Quarterly Journal of the Royal Meteorological Society, 148 (743), 521--556, doi:10.1002/qj.4228

    2022 doi

  5. [13]

    Geer, A. J., 2021: Learning earth system models from observations: machine learning or data assimilation? Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 379 (2194), 20200\,089, doi:10.1098/rsta.2020.0089

    2021

  6. [14]

    J., and Coauthors, 2018: All-sky satellite data assimilation at operational weather forecasting centres

    Geer, A. J., and Coauthors, 2018: All-sky satellite data assimilation at operational weather forecasting centres. Quarterly Journal of the Royal Meteorological Society, 144 (713), 1191--1217, doi:10.1002/qj.3202

    2018 doi

  7. [15]

    Science Advances, 8 (14), eabn3488, doi:10.1126/sciadv.abn3488

    Gettelman, A., and Coauthors, 2022: The future of Earth system prediction: Advances in model-data fusion. Science Advances, 8 (14), eabn3488, doi:10.1126/sciadv.abn3488

    2022 doi

  8. [16]

    M., 2022: Explainable AI for earth observation: A review including societal and regulatory perspectives

    Gevaert, C. M., 2022: Explainable AI for earth observation: A review including societal and regulatory perspectives. International Journal of Applied Earth Observation and Geoinformation, 112, 102\,869, doi:10.1016/j.jag.2022.102869

    2022

  9. [17]

    Gneiting, T., and A. E. Raftery, 2007: Strictly Proper Scoring Rules , Prediction , and Estimation . Journal of the American Statistical Association, 102 (477), 359--378, doi:10.1198/016214506000001437

    2007 doi

  10. [18]

    Zenodo, ://zenodo.org/records/13963758, doi:10.5281/zenodo.13963758

    Howard, L., 2024: Probabilistic Emulation of the Community Radiative Transfer Model Using Machine Learning . Zenodo, ://zenodo.org/records/13963758, doi:10.5281/zenodo.13963758

    2024

  11. [19]

    Howard, L. J., A. Subramanian, and I. Hoteit, 2024: A Machine Learning Augmented Data Assimilation Method for High - Resolution Observations . Journal of Advances in Modeling Earth Systems, 16 (1), e2023MS003\,774, doi:10.1029/2023MS003774

    2024 doi

  12. [20]

    Johnson, B. T., C. Dang, P. Stegmann, Q. Liu, I. Moradi, and T. Auligne, 2023: The Community Radiative Transfer Model ( CRTM ): Community - Focused Collaborative Model Development Accelerating Research to Operations . Bulletin of the American Meteorological Society, 104 (10), ...

    2023 doi

  13. [21]

    P., and J

    Kingma, D. P., and J. Ba, 2015: Adam: A Method for Stochastic Optimization . 3rd International Conference on Learning Representations , ICLR 2015, San Diego , CA , USA , May 7-9, 2015, Conference Track Proceedings , Y. Bengio, and Y. LeCun, Eds., ://arxiv.org/abs/1412.6980

    2015 arXiv

  14. [22]

    Terasaki, and T

    Liang, J., K. Terasaki, and T. Miyoshi, 2023: A Machine Learning Approach to the Observation Operator for Satellite Radiance Data Assimilation . Journal of the Meteorological Society of Japan. Ser. II, 101 (1), 79--95, doi:10.2151/jmsj.2023-005

    2023 doi

  15. [23]

    Garrett, Q

    Liang, X., K. Garrett, Q. Liu, E. S. Maddy, K. Ide, and S. Boukabara, 2022: A Deep - Learning - Based Microwave Radiative Transfer Emulator for Data Assimilation and Remote Sensing . IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 15, 8819--88...

    2022

  16. [24]

    Liang, X., and Q. M. Liu, 2020: Applying Deep Learning to Clear - Sky Radiance Simulation for VIIRS with Community Radiative Transfer Model — Part 2: Model Architecture and Assessment . Remote Sensing, 12 (22), 3825, doi:10.3390/rs12223825

    2020 doi

  17. [25]

    Papastefanopoulos, and S

    Linardatos, P., V. Papastefanopoulos, and S. Kotsiantis, 2021: Explainable AI : A Review of Machine Learning Interpretability Methods . Entropy, 23 (1), 18, doi:10.3390/e23010018

    2021 doi

  18. [26]

    M., and S.-I

    Lundberg, S. M., and S.-I. Lee, 2017: A unified approach to interpreting model predictions. Proceedings of the 31st International Conference on Neural Information Processing Systems , Curran Associates Inc., Red Hook, NY, USA, 4768--4777, NIPS '17

    2017

  19. [27]

    Penny, S. G., T. A. Smith, T.-C. Chen, J. A. Platt, H.-Y. Lin, M. Goodliff, and H. D. I. Abarbanel, 2022: Integrating Recurrent Neural Networks With Data Assimilation for Scalable Data - Driven State Estimation . Journal of Advances in Modeling Earth Systems, 14 (3), e2021MS00...

    2022 doi

  20. [28]

    Schmit, T. J., P. Griffith, M. M. Gunshor, J. M. Daniels, S. J. Goodman, and W. J. Lebair, 2017: A Closer Look at the ABI on the GOES - R Series . Bulletin of the American Meteorological Society, 98 (4), 681--698, doi:10.1175/BAMS-D-15-00230.1

    2017 doi

  21. [29]

    Lguensat, D

    Sonnewald, M., R. Lguensat, D. C. Jones, P. D. Dueben, J. Brajard, and V. Balaji, 2021: Bridging observations, theory and numerical simulation of the ocean using machine learning. Environmental Research Letters, 16 (7), 073\,008, doi:10.1088/1748-9326/ac0eb0

    2021 doi

  22. [30]

    Stegmann, P. G., B. Johnson, I. Moradi, B. Karpowicz, and W. McCarty, 2022: A deep learning approach to fast radiative transfer. Journal of Quantitative Spectroscopy and Radiative Transfer, 280, 108\,088, doi:10.1016/j.jqsrt.2022.108088

    2022

  23. [31]

    Proceedings of ECMWF Seminar on Recent Developments in Data Assimilation for Atmosphere and Ocean , ECMWF , Reading , UK , 8--12

    Thépaut, J.-N., 2003: Satellite data assimilation in numerical weather prediction: An overview. Proceedings of ECMWF Seminar on Recent Developments in Data Assimilation for Atmosphere and Ocean , ECMWF , Reading , UK , 8--12

    2003

  24. [32]

    W., and J

    Uccellini, L. W., and J. E. T. Hoeve, 2019: Evolving the National Weather Service to Build a Weather - Ready Nation : Connecting Observations , Forecasts , and Warnings to Decision - Makers through Impact - Based Decision Support Services . Bulletin of the American Meteorologi...

    2019 doi

  25. [33]

    Journal of Advances in Modeling Earth Systems, 14 (4), e2021MS002\,875, doi:10.1029/2021MS002875

    Ukkonen, P., 2022: Exploring Pathways to More Accurate Machine Learning Emulation of Atmospheric Radiative Transfer . Journal of Advances in Modeling Earth Systems, 14 (4), e2021MS002\,875, doi:10.1029/2021MS002875

    2022 doi

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.