A numerical study into neural network surrogate model performance for uncertainty propagation
Pith reviewed 2026-05-19 18:53 UTC · model grok-4.3
The pith
Neural network surrogates for stochastic heat conduction exhibit order-of-magnitude larger errors at distribution tails due to extrapolation.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Serving as a test case is the heat conduction equation driven by a highly stochastic source term that causes large variations in the temperature solution. Neural network surrogates, including fully connected networks and Deep Operator Networks, are trained using both data-driven and physics-informed losses. The results indicate that the largest errors arise on inputs that require extrapolation beyond the training data, and that these errors are substantially bigger than those for average inputs. The fully connected network trained with a weak form residual loss yields the best accuracy on the numerically generated test datasets, particularly for the extreme samples.
What carries the argument
The weak form residual loss for training fully connected neural networks on the stochastic heat conduction problem, which integrates the governing equation over the domain to enforce physical consistency and aid generalization to out-of-distribution inputs.
If this is right
- Errors in surrogate predictions for uncertainty propagation are primarily driven by the need to extrapolate to extreme stochastic inputs.
- Using a weak-form residual loss during training improves a model's ability to handle samples outside the training distribution.
- Deep Operator Networks do not outperform simpler fully connected networks in this setting for tail accuracy.
- Explicit identification of outlier samples is necessary to properly account for their contribution to overall uncertainty.
Where Pith is reading between the lines
- Adaptive sampling strategies during dataset generation could focus on filling in the tails to reduce extrapolation demands.
- The findings may extend to other stochastic boundary value problems where solution variability is high.
- Integrating uncertainty quantification techniques with these surrogates could provide error bounds for extreme predictions.
Load-bearing premise
The sampled training datasets cover the probability space of the stochastic source term sufficiently well that the observed large errors on extreme samples stem mainly from extrapolation rather than from inadequate training or model limitations.
What would settle it
Generating an augmented training set that includes additional samples from the tails of the source term distribution and then re-evaluating the maximum prediction errors on a held-out extreme test set; if the errors remain similarly large, the extrapolation explanation would be weakened.
Figures
read the original abstract
Neural network surrogate models have emerged as a promising approach to model solution fields for a wide variety of boundary value problems encountered in physical modeling. Stochastic problems represent an area of particularly high interest because of the potential to significantly reduce the repeated evaluation of expensive forward models via traditional numerical solvers when conducting parametric analysis. However, many studies found in the literature primarily focus on the ability of neural network surrogate models to represent deterministic samples or mean field solutions and largely overlook surrogate model performance at the tails of the distribution. The present study examines in detail the ability of neural network surrogate models to capture the full distribution of solution fields over the entire probability space, while emphasis is placed at the tails of the distribution. Serving as a canonical problem is the heat conduction equation with a highly stochastic source term, inducing extremely large variation in the thermal solution field. Comparisons are made between a classic feed-forward fully connected network and a Deep Operator Network architecture, using both data-driven and physics-informed loss functions. Results show that the worst-case prediction errors are an order of magnitude larger than the mean field error, highlighting the importance of the outlier samples. The large errors associated with extreme samples result from the networks having to extrapolate beyond the bounds of the training data. A method for identifying these samples is presented along with a discussion of potential approaches to account of their errors. Among the models considered, the fully connected neural network trained using a weak form residual loss performs best in handling these extrapolated inputs, achieving the highest prediction accuracy for the numerically produced datasets.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents a numerical study comparing neural network surrogate models (fully connected networks and DeepONets) with data-driven and physics-informed loss functions (including weak-form residual) for uncertainty propagation in the heat conduction equation subject to a highly stochastic source term. Emphasis is placed on performance across the full probability space, particularly at the distribution tails. Key results include an order-of-magnitude gap between mean and worst-case prediction errors, attribution of large errors on extreme samples to extrapolation beyond training bounds, presentation of a method to identify such samples, and the conclusion that the fully connected network trained with weak-form residual loss performs best on extrapolated inputs for the generated datasets.
Significance. If the experimental details hold, the work usefully draws attention to the practical challenges of achieving reliable surrogate coverage over entire distributions in stochastic PDE problems rather than just mean fields. The reported order-of-magnitude disparity between average and outlier errors, together with the explicit comparison of architectures and loss formulations, supplies concrete numerical evidence that could guide model selection in uncertainty quantification applications. The identification method for extreme samples is a potentially reusable contribution.
major comments (1)
- [Dataset generation and numerical experiments] The description of training dataset generation (referenced in the abstract and results discussion) provides no quantitative information on Monte Carlo sample count, tail quantiles retained, or coverage metrics for the stochastic source term distribution. This detail is load-bearing for the central claim that large errors on extreme samples arise primarily from extrapolation rather than under-sampling of tails, optimization failure, or capacity limits; without it, the ranking among models could be confounded by unequal handling of rare but in-distribution events.
minor comments (1)
- The abstract states that 'a method for identifying these samples is presented' but does not reference the specific section, figure, or algorithm number where this method appears, making it difficult for readers to locate and evaluate the procedure.
Simulated Author's Rebuttal
We thank the referee for their careful and constructive review of our manuscript. The major comment on dataset generation details is well taken, and we address it point by point below. We will revise the manuscript to incorporate the requested quantitative information.
read point-by-point responses
-
Referee: [Dataset generation and numerical experiments] The description of training dataset generation (referenced in the abstract and results discussion) provides no quantitative information on Monte Carlo sample count, tail quantiles retained, or coverage metrics for the stochastic source term distribution. This detail is load-bearing for the central claim that large errors on extreme samples arise primarily from extrapolation rather than under-sampling of tails, optimization failure, or capacity limits; without it, the ranking among models could be confounded by unequal handling of rare but in-distribution events.
Authors: We agree that the current manuscript provides only a qualitative description of the training dataset generation and lacks the specific quantitative details requested. We will add these to a new or expanded subsection on dataset generation, reporting the Monte Carlo sample count, the quantile thresholds used to identify tail samples, and coverage metrics (such as the spanned range of the stochastic source term and any binning or density checks performed). This revision will directly support the claim that the large errors on extreme samples result from extrapolation beyond training bounds. We will also add a short discussion clarifying that all compared models were trained and tested on identical datasets, so relative rankings are not confounded by differential sampling of rare events; we will further note the diagnostic checks (e.g., loss convergence and residual norms) that indicate optimization and capacity were not the dominant factors for the observed outliers. revision: yes
Circularity Check
No circularity: results from direct numerical experiments on standard problem
full rationale
The paper is an empirical numerical study comparing neural network architectures and loss functions (data-driven vs. physics-informed, fully connected vs. DeepONet) on a canonical stochastic heat conduction problem. All performance claims, including the ranking of the fully connected network with weak-form residual loss on extrapolated samples, are obtained from training on generated datasets and evaluating error metrics against reference solutions. No derivation chain exists that reduces a claimed prediction or first-principles result to its own inputs by construction; there are no self-definitional equations, fitted parameters renamed as predictions, load-bearing self-citations, or uniqueness theorems imported from prior author work. The methodology is self-contained and externally falsifiable via direct numerical solver comparisons.
Axiom & Free-Parameter Ledger
free parameters (1)
- Neural network architecture and training hyperparameters
axioms (1)
- domain assumption The stochastic source term produces extremely large variation in the thermal solution field
Reference graph
Works this paper leans on
-
[1]
I. E. Lagaris, A. Likas, D. I. Fotiadis, Artificial neural networks for solving ordinary and partial differential equations, IEEE Transactions on Neural Networks 9 (1998) 987–1000
work page 1998
-
[2]
M. Raissi, P. Perdikaris, G. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computa- tional Physics 378 (2019) 686–707. doi:10.1016/j.jcp.2018.10.045
-
[3]
S. Cai, Z. Wang, S. Wang, P. Perdikaris, G. E. Karniadakis, Physics-informed neural networks for heat transfer problems, Journal of Heat Transfer 143 (2021)
work page 2021
-
[4]
N. Sukumar, A. Srivastava, Exact imposition of boundary conditions with distance functions in physics- informed deep neural networks, Computer Methods in Applied Mechanics and Engineering 389 (2022) 114333
work page 2022
-
[5]
R. Xu, D. Zhang, M. Rong, N. Wang, Weak form theory-guided neural network (TgNN-wf) for deep learning of subsurface single- and two-phase flow, Journal of Computational Physics 436 (2021) 110318. doi:10.1016/j.jcp.2021.110318
-
[6]
S. Cai, Z. Mao, Z. Wang, M. Yin, G. E. Karniadakis, Physics-informed neural networks (PINNs) for fluid mechanics: A review, Acta Mechanica Sinica 37 (2021) 1727–1738
work page 2021
-
[7]
S. Goswami, M. Yin, Y. Yu, G. E. Karniadakis, A physics-informed variational DeepONet for predicting crack path in quasi-brittle materials, Computer Methods in Applied Mechanics and Engineering 391 (2022) 114587. doi:10.1016/j.cma.2022.114587
-
[8]
A. A. Howard, M. Perego, G. E. Karniadakis, P. Stinis, Multifidelity deep operator networks for data-driven and physics-informed problems, Journal of Computational Physics 493 (2023) 112462. 25
work page 2023
-
[9]
Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, A. Anandkumar, Fourier neural operator for parametric partial differential equations, arXiv preprint arXiv:2010.08895 (2020)
work page internal anchor Pith review Pith/arXiv arXiv 2010
- [10]
-
[11]
S. Markidis, The old and the new: Can physics-informed deep-learning replace traditional linear solvers?, Frontiers in big Data 4 (2021) 669097
work page 2021
-
[12]
F. F. de la Mata, A. Gij´ on, M. Molina-Solana, J. G´ omez-Romero, Physics-informed neural networks for data-driven simulation: Advantages, limitations, and opportunities, Physica A: Statistical Mechanics and its Applications 610 (2023) 128415
work page 2023
-
[13]
T. G. Grossmann, U. J. Komorowska, J. Latz, C.-B. Sch¨ onlieb, Can physics-informed neural networks beat the finite element method?, IMA Journal of Applied Mathematics 89 (2024) 143–174
work page 2024
-
[14]
A. Sacchetti, B. Bachmann, K. L¨ offel, U.-M. K¨ unzi, B. Paoli, Neural networks to solve partial differential equations: A comparison with finite elements, IEEE Access 10 (2022) 32271–32279
work page 2022
-
[15]
J. Hou, Y. Li, S. Ying, Enhancing PINNs for solving PDEs via adaptive collocation point movement and adaptive loss weighting, Nonlinear Dynamics 111 (2023) 15233–15261
work page 2023
-
[16]
M. A. Nabian, R. J. Gladstone, H. Meidani, Efficient training of physics-informed neural networks via impor- tance sampling, Computer-Aided Civil and Infrastructure Engineering 36 (2021) 962–977
work page 2021
- [17]
- [18]
- [19]
-
[20]
Y. Yang, P. Perdikaris, Adversarial uncertainty quantification in physics-informed neural networks, Journal of Computational Physics 394 (2019) 136–152
work page 2019
-
[21]
B. L¨ utjens, C. H. Crawford, M. Veillette, D. Newman, Pce-pinns: Physics-informed neural networks for uncertainty propagation in ocean modeling, arXiv preprint arXiv:2105.02939 (2021)
-
[22]
Z. Hu, K. Shukla, G. E. Karniadakis, K. Kawaguchi, Tackling the curse of dimensionality with physics-informed neural networks, Neural Networks 176 (2024) 106369. 26
work page 2024
- [23]
- [24]
-
[25]
M. A. Nabian, H. Meidani, A deep learning solution approach for high-dimensional random differential equa- tions, Probabilistic Engineering Mechanics 57 (2019) 14–25
work page 2019
-
[26]
L. Lu, P. Jin, G. Pang, Z. Zhang, G. E. Karniadakis, Learning nonlinear operators via deeponet based on the universal approximation theorem of operators, Nature machine intelligence 3 (2021) 218–229
work page 2021
-
[27]
Z. Li, H. Zheng, N. Kovachki, D. Jin, H. Chen, B. Liu, K. Azizzadenesheli, A. Anandkumar, Physics-informed neural operator for learning partial differential equations, ACM/JMS Journal of Data Science 1 (2024) 1–27
work page 2024
-
[28]
K. Kontolati, S. Goswami, G. Em Karniadakis, M. D. Shields, Learning nonlinear operators in latent spaces for real-time predictions of complex dynamics in physical systems, Nature Communications 15 (2024) 5101
work page 2024
-
[29]
J. Exenberger, S. Ranftl, R. Peharz, Deep polynomial chaos expansion, arXiv preprint arXiv:2507.21273 (2025)
- [30]
-
[31]
S. Karumuri, L. Graham-Brady, S. Goswami, Physics-informed latent neural operator for real-time predictions of complex physical systems, arXiv preprint arXiv:2501.08428 (2025)
- [32]
-
[33]
E. Pickering, S. Guth, G. E. Karniadakis, T. P. Sapsis, Discovering and forecasting extreme events via active learning in neural operators, Nature Computational Science 2 (2022) 823–833
work page 2022
-
[34]
M. Wabartha, A. Durand, V. Francois-Lavet, J. Pineau, Handling black swan events in deep learning with diversely extrapolated neural networks, in: Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, 2021, pp. 2140–2147
work page 2021
-
[35]
C. Anderson, The Long Tail: Why the Future of Business is Selling Less of More, Hyperion, New York, 2006
work page 2006
-
[36]
Y. Cui, M. Jia, T.-Y. Lin, Y. Song, S. Belongie, Class-balanced loss based on effective number of samples, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 9268–9277
work page 2019
-
[37]
L. Yang, H. Jiang, Q. Song, J. Guo, A survey on long-tailed visual recognition, International Journal of Computer Vision 130 (2022) 1837–1872
work page 2022
-
[38]
S. H. Rudy, T. P. Sapsis, Output-weighted and relative entropy loss functions for deep learning precursors of extreme events, Physica D: Nonlinear Phenomena 443 (2023) 133570. 27
work page 2023
-
[39]
C. Song, R. Xiao, C. Zhang, X. Zhao, B. Sun, Simulation-free reliability analysis with importance sampling- based adaptive training physics-informed neural networks: Method and application to chloride penetration, Reliability Engineering & System Safety 246 (2024) 110083
work page 2024
-
[40]
T. Chen, H. Chen, Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems, IEEE transactions on neural networks 6 (1995) 911–917
work page 1995
-
[41]
N. Kovachki, Z. Li, B. Liu, K. Azizzadenesheli, K. Bhattacharya, A. Stuart, A. Anandkumar, Neural operator: Learning maps between function spaces with applications to pdes, Journal of Machine Learning Research 24 (2023) 1–97
work page 2023
-
[42]
M. Steininger, K. Kobs, P. Davidson, A. Krause, A. Hotho, Density-based weighting for imbalanced regression, Machine Learning 110 (2021) 2187–2211
work page 2021
-
[43]
S. Wang, H. Wang, P. Perdikaris, Learning the solution operator of parametric partial differential equations with physics-informed DeepONets, Science advances 7 (2021) eabi8605
work page 2021
-
[44]
D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014)
work page internal anchor Pith review Pith/arXiv arXiv 2014
-
[45]
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al., Tensorflow: Large-scale machine learning on heterogeneous distributed systems, arXiv preprint arXiv:1603.04467 (2016)
work page internal anchor Pith review Pith/arXiv arXiv 2016
-
[46]
F. P. Preparata, M. I. Shamos, Computational Geometry: An Introduction, Springer Science & Business Media, 2012
work page 2012
-
[47]
N. Wade, K. Teferra, https://github.com/USNavalResearchLaboratory/UQPINN, 2026. 28
work page 2026
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.