REVIEW 3 major objections 4 minor 38 references
Distribution-Free Uncertainty-Aware Virtual Sensing via Conformalized Neural Operators
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that a single trained neural operator, paired with Monte Carlo dropout and split conformal calibration, produces distribution-free 95% spatial prediction intervals in virtual sensing.
desk verdict A practical MC-dropout + conformal wrapper for DeepONets, but the z=1.96 multiplier in Eq. (6) breaks the conformal guarantee and makes the reported 98–99% coverage overcoverage, not near-nominal calibration. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the calibrated interval $$I(u_t) = [\mu(u_t) - z \cdot q \cdot \$\sigma$(u_t), \; \mu(u_t) + z \cdot q \cdot \$\sigma$(u_t)],$$ with $z = 1.96$. The mean $\mu$ and spread $\sigma$ come from $n_c$ stochastic forward passes of the trained DeepONet with dropout active at inference, and the vector $q$ is a per-location conformal quantile: for each spatial index $j$, $q_j$ is the $\lceil (1-\alpha)(n+1)\rceil / n$ quantile of the normalized calibration residuals $|y_{i,j} - \mu_j(u_i)|/\sigma_j(u_i)$. That normalization is what makes the bandwidth distribution-free: it rescales the MC-dropout variance by the observed error at each location, so the interval width adapts spatially through data alone, with no new training.
What would settle it
Take the calibrated dose model, hold out a solar-storm day not represented in the calibration period, and compute the fraction of spatial grid points covered. A coverage far below 95% — the paper already reports 1.14% on May 10, 2023 — would falsify the claim that near-nominal coverage holds on any new input, and would show the guarantee is conditional on exchangeability rather than universal.
Extended reading notes
Core claim
The central claim is that conformalized MC dropout gives operator learning a lightweight, distribution-free UQ layer: for a test input, the interval is $I(u_t) = [\mu(u_t) - z \cdot q \cdot \sigma(u_t), \; \mu(u_t) + z \cdot q \cdot \sigma(u_t)]$ with $z = 1.96$, where $\mu$ and $\sigma$ are the mean and standard deviation of $n_c$ stochastic forward passes of the same trained network, and $q$ is a vector of per-location conformal quantiles computed from calibration residuals. The paper reports that this construction achieves average empirical coverage of 99.68% on lid-driven cavity turbulence, 98.96% on elastoplastic deformation, and 98.63% on cosmic-radiation dose estimation, all near or above the nominal 95% level. It also reports that under-coverage concentrates in hard samples: the per-sample minimum coverage across cases is 92.59%, 53.33%, and 1.14%, with the lowest value occurring on an atypical radiation day, so the guarantee is marginal and breaks under distribution shift.
Load-bearing premise
The guarantee rests on calibration and test input-output pairs being exchangeable, meaning the residual pattern seen during calibration must still be representative when the model is deployed.
Editorial extensions
If this is right
- Existing trained Sequential DeepONets can be deployed with uncertainty by enabling dropout and storing $n_c$ forward passes, with no architectural change or retraining.
- One calibration pass produces intervals whose marginal coverage is guaranteed by split conformal theory without assuming Gaussian errors or any parametric family.
- The intervals are spatially resolved, so a monitoring system can flag dangerous regions such as cavity corners, necking zones, or the South Atlantic Anomaly rather than reporting a single global uncertainty.
- Failure-rate and relative-error scatter plots can serve as a diagnostic to identify samples where the model is simultaneously inaccurate and overconfident, which is the dangerous regime for safety-critical sensing.
Reading between the lines
- Because the conformal quantile $q_j$ is estimated independently at each location, the method leaves spatial structure in $q$ unexploited; smoothing $q$ or conditioning it on local density and gradient strength could recover the under-covered tail samples without inflating widths elsewhere.
- The 1.14% minimum coverage on an atypical radiation day is a concrete measure of how fast a fixed calibration set goes stale, so a natural sequel is an adaptive or online conformal update that monitors exchangeability and refreshes $q$ when it breaks.
- The same normalized-residual calibration can wrap any base uncertainty proxy, so MC dropout is replaceable by heteroscedastic head uncertainties or randomized priors without changing the conformal layer.
- In digital-twin deployments, calibration data and operational data will rarely be exchangeable, and the paper's own Table 3 numbers quantify the size of the safety margin operators should assume rather than treating 95% coverage as a hard guarantee.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CMCO, a framework that combines Monte Carlo dropout with split conformal prediction on a sequential DeepONet to produce distribution-free prediction intervals for virtual sensing. It is evaluated on three tasks: turbulent kinetic energy reconstruction in a lid-driven cavity, von Mises stress field prediction in elastoplastic deformation, and global cosmic radiation dose estimation from sparse neutron monitor inputs. The central claim is that CMCO achieves near-nominal 95% marginal coverage with minimal computational overhead.
Significance. If the coverage guarantee were correctly implemented, CMCO would be a practically attractive UQ wrapper for neural operators: it requires only a single trained model, has low inference overhead, and is evaluated on three physically meaningful problems. The paper also provides extensive empirical coverage statistics and visualizations. However, the central theoretical claim is undermined by a nonstandard interval construction that inflates the intervals, and the reported coverage is systematically above nominal, so the results as presented do not substantiate the headline claim.
major comments (3)
- [Section 2.3, Eq. (6)] The prediction interval is defined as mu(ut) +/- z * q * sigma(ut) with z = 1.96, where q is the conformal quantile of normalized residuals from Eq. (5). Standard split conformal prediction with normalized nonconformity scores would construct the interval as mu(ut) +/- q * sigma(ut), which has marginal coverage at least 1 - alpha under exchangeability. The extra multiplier z changes the effective coverage to P(e <= 1.96 q), which is not controlled at the nominal level and systematically inflates the intervals. This invalidates the theoretical guarantee claimed in Section 5 and explains the 98-99% average coverage in Table 3.
- [Section 4, Table 3] The reported average empirical coverages are 99.68% (Case I), 98.96% (Case II), and 98.63% (Case III), all far above the nominal 95% target. The abstract and Section 5 describe these as 'near-nominal', which is inaccurate. Moreover, the minimum coverages of 92.59%, 53.33%, and 1.14% show that averaging hides severe undercoverage for individual samples. The paper should report the full coverage distribution or bootstrap confidence intervals, and the overcoverage should be acknowledged as a direct consequence of the z multiplier in Eq. (6).
- [Section 3.3 and Figure 9] The split conformal guarantee relies on exchangeability of calibration and test data. In the cosmic radiation case, the test inputs are 7-day sequences of neutron monitor readings ordered in time; atypical days such as May 10, 2023 (Figure 9b) are not exchangeable with the calibration set, which likely explains the catastrophic 1.14% coverage for that sample. The manuscript does not address this violation or propose a remedy (e.g., sliding-window or weighted conformal prediction), so the distribution-free claim is not established for this application.
minor comments (4)
- [Figure 9 caption] The caption states that 'Prediction intervals were obtained using conformalized RP-DeepONet', but the paper proposes CMCO (conformalized MC-dropout DeepONet). This is inconsistent with the rest of the manuscript and should be corrected.
- [Section 2.3, Eq. (4)] The normalized residual divides by sigma_j(u_i), which can be zero or near zero in regions where the MC-dropout ensemble is degenerate. The authors should specify a numerical stability correction (a small epsilon) for this division.
- [Section 3.1 vs. Appendix] The main text states that the lid-driven cavity branch network is a four-layer GRU with Tanh activations, while the appendix specifies 'Type: LSTM' and ReLU activations for the same case. These inconsistencies should be reconciled.
- [Section 2.3, Eq. (6)] The notation 'z · q · sigma' is ambiguous because q and sigma are vectors and z is a scalar. The paper should explicitly state that the multiplication is elementwise, or use a clearer componentwise notation.
Circularity Check
No material circularity: CMCO's empirical coverage claim is grounded in held-out conformal calibration, not in a self-citation or a fitted target; the z=1.96 factor is a validity bug, not a circular reduction.
full rationale
The derivation chain is not circular. The conformal quantile q_j in Eq. (5) is estimated from a calibration split that is disjoint from the test samples used for coverage evaluation (Table 1). The coverage metric in Eq. (7) is evaluated on held-out test inputs, so the headline coverage numbers are not forced by the calibration fit. The calibration rule is attributed to the authors' own prior work [31], but split conformal quantile calibration is a standard external result whose validity does not depend on [31]; the self-citation is therefore not load-bearing. The reuse of the prior cosmic-dose model [2] is as a component, again with independent test evaluation. The main methodological weakness is that Eq. (6) inserts a fixed Gaussian multiplier z=1.96 after computing the conformal quantile, so the implemented interval is wider than the standard split-conformal interval and the reported 98-99% coverages exceed nominal; the 1.14% worst-case coverage also shows that exchangeability can fail. These are correctness and validity concerns, not circular reductions: no prediction is equivalent by construction to a fitted input or to a self-cited claim.
Assumptions & free parameters
free parameters (5)
- z (Gaussian multiplier) =
1.96
- q_j (per-location conformal quantile) =
computed via Eq. (5) from calibration residuals
- MC dropout rate p =
0.1
- Number of MC forward passes n_c =
10
- Nominal level alpha =
0.05
assumptions (4)
- domain assumption Calibration and test samples are exchangeable (split conformal premise)
- domain assumption MC dropout provides a meaningful dispersion scale sigma
- domain assumption Simulation codes (ANSYS Fluent RANS, Abaqus J2 plasticity, PARMA/PHITS) provide ground truth
- standard math Standard neural network training converges and generalizes
Cite this review
Pith. "Pith review of Distribution-Free Uncertainty-Aware Virtual Sensing via Conformalized Neural Operators." pith.science (2026). https://pith.science/paper/27RIC2II
@misc{pith2026250711574,
author = {Pith},
title = {Pith review of: Distribution-Free Uncertainty-Aware Virtual Sensing via Conformalized Neural Operators},
year = {2026},
howpublished = {\url{https://pith.science/paper/27RIC2II}},
note = {Machine review of arXiv:2507.11574}
}
read the original abstract
Robust uncertainty quantification (UQ) remains a critical barrier to the safe deployment of deep learning in real-time virtual sensing, particularly in high-stakes domains where sparse, noisy, or non-collocated sensor data are the norm. We introduce the Conformalized Monte Carlo Operator (CMCO), a framework that transforms neural operator-based virtual sensing with calibrated, distribution-free prediction intervals. By unifying Monte Carlo dropout with split conformal prediction in a single DeepONet architecture, CMCO achieves spatially resolved uncertainty estimates without retraining, ensembling, or custom loss design. Our method addresses a longstanding challenge: how to endow operator learning with efficient and reliable UQ across heterogeneous domains. Through rigorous evaluation on three distinct applications: turbulent flow, elastoplastic deformation, and global cosmic radiation dose estimation-CMCO consistently attains near-nominal empirical coverage, even in settings with strong spatial gradients and proxy-based sensing. This breakthrough offers a general-purpose, plug-and-play UQ solution for neural operators, unlocking real-time, trustworthy inference in digital twins, sensor fusion, and safety-critical monitoring. By bridging theory and deployment with minimal computational overhead, CMCO establishes a new foundation for scalable, generalizable, and uncertainty-aware scientific machine learning.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Kazuma Kobayashi and Syed Bahauddin Alam. Explainable, interpretable, and trustworthy ai for an intelligent digital twin: A case study on remaining useful life.Engineering Applications of Artificial Intelligence, 129:107620, 2024
work page 2024
-
[2]
Kazuma Kobayashi, Samrendra Roy, Seid Koric, Diab Abueidda, and Syed Bahauddin Alam. From proxies to fields: Spatiotemporal reconstruction of global radiation from sparse sensor sequences. arXiv preprint arXiv:2506.12045, 2025
arXiv 2025
-
[3]
Virtual sensing to enable real-time monitoring of inaccessible locations & unmeasurable parameters
Kazuma Kobayashi, Farid Ahmed, and Syed Bahauddin Alam. Virtual sensing to enable real-time monitoring of inaccessible locations & unmeasurable parameters. arXiv preprint arXiv:2412.00107, 2024
arXiv 2024
-
[4]
Raisa Hossain, Farid Ahmed, Kazuma Kobayashi, Seid Koric, Diab Abueidda, and Syed Bahauddin Alam. Virtual sensing-enabled digital twin framework for real-time monitoring of nuclear systems leveraging deep neural operators. npj Materials Degradation, 9(1):21, 2025
work page 2025
-
[5]
Kazuma Kobayashi and Syed Bahauddin Alam. Deep neural operator-driven real-time inference to enable digital twin solutions for nuclear energy systems. Scientific reports, 14(1):2101, 2024
work page 2024
-
[6]
Fourier neural operator for parametric partial differential equations
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020
arXiv 2010
-
[7]
Spherical fourier neural operators: Learning stable dynamics on the sphere
Boris Bonev, Thorsten Kurth, Christian Hundt, Jaideep Pathak, Maximilian Baust, Karthik Kashinath, and Anima Anandkumar. Spherical fourier neural operators: Learning stable dynamics on the sphere. In International conference on machine learning, pages 2806–2823. PMLR, 2023
work page 2023
-
[8]
Neural operator: Graph kernel network for partial differential equations
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003.03485, 2020
arXiv 2003
Show all 38 references
-
[9]
Multipole graph neural operator for parametric partial differential equations
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. Advances in Neural Information Processing Systems, 33:6755–6766, 2020
2020
-
[10]
Wavelet neural operator for solving parametric partial differential equations in computational mechanics problems
Tapas Tripura and Souvik Chakraborty. Wavelet neural operator for solving parametric partial differential equations in computational mechanics problems. Computer Methods in Applied Mechanics and Engineering, 404:115783, 2023
2023
-
[11]
A wavelet neural operator based elastography for localization and quantification of tumors
Tapas Tripura, Abhilash Awasthi, Sitikantha Roy, and Souvik Chakraborty. A wavelet neural operator based elastography for localization and quantification of tumors. Computer Methods and Programs in Biomedicine, 232:107436, 2023
2023
-
[12]
Learning nonlinear operators via deeponet based on the universal approximation theorem of operators
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence, 3(3):218–229, 2021
2021
-
[13]
Tianping Chen and Hong Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems.IEEE transactions on neural networks, 6(4):911–917, 1995
1995
-
[14]
Mionet: Learning multiple-input operators via tensor product
Pengzhan Jin, Shuai Meng, and Lu Lu. Mionet: Learning multiple-input operators via tensor product. SIAM Journal on Scientific Computing, 44(6):A3490–A3514, 2022
2022
-
[15]
Sequential deep operator networks (s-deeponet) for predicting full-field solutions under time-dependent loads
Junyan He, Shashank Kushwaha, Jaewan Park, Seid Koric, Diab Abueidda, and Iwona Jasiuk. Sequential deep operator networks (s-deeponet) for predicting full-field solutions under time-dependent loads. Engineering Applications of Artificial Intelligence, 127:107258, 2024
2024
-
[16]
Predictions of transient vector solution fields with sequential deep operator network
Junyan He, Shashank Kushwaha, Jaewan Park, Seid Koric, Diab Abueidda, and Iwona Jasiuk. Predictions of transient vector solution fields with sequential deep operator network. Acta Mechanica, 235(8):5257–5272, 2024
2024
-
[17]
Bridging sequential deep operator network and video diffusion: Residual refinement of spatio-temporal pde solutions
Jaewan Park, Farid Ahmed, Kazuma Kobayashi, Seid Koric, Syed Bahauddin Alam, Iwona Jasiuk, and Diab Abueidda. Bridging sequential deep operator network and video diffusion: Residual refinement of spatio-temporal pde solutions. arXiv preprint arXiv:2507.06133, 2025
2025 arXiv
-
[18]
Fully convolutional network enhanced deeponet-based surrogate of predicting the travel-time fields
Yifan Mei, Yijie Zhang, Xueyu Zhu, Rongxi Gou, and Jinghuai Gao. Fully convolutional network enhanced deeponet-based surrogate of predicting the travel-time fields. IEEE Transactions on Geoscience and Remote Sensing, 2024. 15 Distribution-Free UQ Virtual Sensing
2024
-
[19]
Porous-deeponet: Learning the solution operators of parametric reactive transport equations in porous media
Pan Huang, Yifei Leng, Cheng Lian, and Honglai Liu. Porous-deeponet: Learning the solution operators of parametric reactive transport equations in porous media. Engineering, 39:94–103, 2024
2024
-
[20]
Ai-driven uncertainty quantification & multi-physics approach to evaluate cladding materials in a microreactor
Alexander Foutch, Kazuma Kobayashi, Ayodeji Alajo, Dinesh Kumar, and Syed Bahauddin Alam. Ai-driven uncertainty quantification & multi-physics approach to evaluate cladding materials in a microreactor. Progress in Nuclear Energy, 186:105793, 2025
2025
-
[21]
Degradation-aware and machine learning- driven uncertainty quantification in crystal plasticity finite element: Texture-driven plasticity in 316l stainless steel
Dinesh Kumar, Eralp Demir, Julio Spadotto, Kazuma Kobayashi, Syed Bahauddin Alam, Brian Connolly, Ed Pickering, Paul Wilcox, David Knowles, and Mahmoud Mostafavi. Degradation-aware and machine learning- driven uncertainty quantification in crystal plasticity finite element: Te...
2025 arXiv
-
[22]
Ai-driven non-intrusive uncertainty quantification of advanced nuclear fuels for digital twin-enabling technology
Kazuma Kobayashi, Dinesh Kumar, and Syed Bahauddin Alam. Ai-driven non-intrusive uncertainty quantification of advanced nuclear fuels for digital twin-enabling technology. Progress in Nuclear Energy, 172:105177, 2024
2024
-
[23]
Practical applications of gaussian process with uncertainty quantification and sensitivity analysis for digital twin for accident-tolerant fuel
Kazuma Kobayashi, Dinesh Kumar, Matthew Bonney, and Syed Alam. Practical applications of gaussian process with uncertainty quantification and sensitivity analysis for digital twin for accident-tolerant fuel. In Handbook of Smart Energy Systems, pages 503–514. Springer, 2023
2023
-
[24]
Uncertainty quantification and sensitivity analysis for digital twin enabling technology: Application for bison fuel performance code
Kazuma Kobayashi, Dinesh Kumar, Matthew Bonney, Souvik Chakraborty, Kyle Paaren, Shoaib Usman, and Syed Alam. Uncertainty quantification and sensitivity analysis for digital twin enabling technology: Application for bison fuel performance code. In Handbook of Smart Energy Syst...
2023
-
[25]
Quantitative risk assessment of a high power density small modular reactor (SMR) core using uncertainty and sensitivity analyses
Dinesh Kumar et al. Quantitative risk assessment of a high power density small modular reactor (SMR) core using uncertainty and sensitivity analyses. Energy, 227:120400, 2021
2021
-
[26]
Multi-criteria decision making under uncertainties in composite materials selection and design
Dinesh Kumar et al. Multi-criteria decision making under uncertainties in composite materials selection and design. Composite Structures, 279:114680, 2022
2022
-
[27]
Bayesian neural networks: An introduction and survey
Ethan Goan and Clinton Fookes. Bayesian neural networks: An introduction and survey. Case Studies in Applied Bayesian Data Science: CIRM Jean-Morlet Chair, Fall 2018, pages 45–87, 2020
2018
-
[28]
Hands- on bayesian neural networks—a tutorial for deep learning users
Laurent Valentin Jospin, Hamid Laga, Farid Boussaid, Wray Buntine, and Mohammed Bennamoun. Hands- on bayesian neural networks—a tutorial for deep learning users. IEEE Computational Intelligence Magazine, 17(2):29–48, 2022
2022
-
[29]
Simple and scalable predictive uncertainty estimation using deep ensembles
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017
2017
-
[30]
Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processi...
2019
-
[31]
Distribution free uncertainty quantification for neuroscience-inspired deep neural operators
Shailesh Garg and Souvik Chakraborty. Distribution free uncertainty quantification for neuroscience-inspired deep neural operators. Journal of Computational Physics, 534:114012, 2025
2025
-
[32]
Gaussian processes in machine learning
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning, pages 63–71. Springer, 2003
2003
-
[33]
Randomized prior functions for deep reinforcement learning
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. Advances in neural information processing systems, 31, 2018
2018
-
[34]
Analytical model for estimating terrestrial cosmic ray fluxes nearly anytime and anywhere in the world: Extension of parma/expacs
Tatsuhiko Sato. Analytical model for estimating terrestrial cosmic ray fluxes nearly anytime and anywhere in the world: Extension of parma/expacs. PloS one, 10(12):e0144679, 2015
2015
-
[35]
Analytical model for estimating the zenith angle dependence of terrestrial cosmic ray fluxes
Tatsuhiko Sato. Analytical model for estimating the zenith angle dependence of terrestrial cosmic ray fluxes. PloS one, 11(8):e0160390, 2016
2016
-
[36]
EXPACS: EXcel-based program for calculating atmospheric cosmic-ray spectrum
Japan Atomic Energy Agency. EXPACS: EXcel-based program for calculating atmospheric cosmic-ray spectrum. http://phits.jaea.go.jp/expacs/. Accessed: November 4, 2024
2024
-
[37]
Benchmark study of particle and heavy-ion transport code system using shielding integral benchmark archive and database for accelerator-shielding experiments
Yosuke Iwamoto, Shintaro Hashimoto, Tatsuhiko Sato, Norihiro Matsuda, Satoshi Kunieda, Yurdunaz Çelik, Naoya Furutachi, and Koji Niita. Benchmark study of particle and heavy-ion transport code system using shielding integral benchmark archive and database for accelerator-shiel...
2022
-
[38]
Recent improvements of the particle and heavy ion transport code system–phits version 3.33
Tatsuhiko Sato, Yosuke Iwamoto, Shintaro Hashimoto, Tatsuhiko Ogawa, Takuya Furuta, Shin-Ichiro Abe, Takeshi Kai, Yusuke Matsuya, Norihiro Matsuda, Yuho Hirata, et al. Recent improvements of the particle and heavy ion transport code system–phits version 3.33. Journal of Nuclea...
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.