Quantifying Out-of-Training Uncertainty of Neural-Network based Turbulence Closures

Cody Grogan; Izabela Gutowska; Mauricio Tano; Som Dhulipala; Som Dutta

REVIEW 4 major objections 4 minor 1 references

Quantifying Out-of-Training Uncertainty of Neural-Network based Turbulence Closures

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Deep ensembles rival exact Gaussian processes for turbulence-closure uncertainty.

desk verdict Readable abstract, unreadable body; the abstract suggests a useful benchmark result, but the NLL claim needs a variance-protocol check before it supports the DE recommendation. read the letter →

arxiv 2508.16891 v1 pith:WJPLB4CW submitted 2025-08-23 cs.LG physics.flu-dyn

Cody Grogan , Som Dhulipala , Mauricio Tano , Izabela Gutowska , Som Dutta This is my paper · ORCID

classification cs.LGphysics.flu-dyn

keywords epistemicuncertaintyturbulenceclosuredeepensemblesGaussianprocessMonte-Carlodropoutstochasticvariationalinferenceout-of-distributiongeneralizationCFDsurrogatemodeling

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks which uncertainty-quantification method a practitioner should attach to a neural-network surrogate for a turbulence closure when the surrogate is queried outside its training regime. Using a published algebraic turbulence closure as the reference output, the authors compare exact Gaussian processes (GP), deep ensembles (DE), Monte-Carlo dropout (MCD), and stochastic variational inference (SVI). They find the exact GP is the most accurate, but deep ensembles come close, deliver the best out-of-training negative log-likelihood of all four methods, and avoid the GP's O(n^3) per-step cost. The paper concludes that simple retraining with different seeds, rather than bespoke Bayesian machinery, is a practical source of epistemic uncertainty for these surrogates.

What carries the argument

The benchmark setup: a published algebraic turbulence closure acts as the reference output; each method is trained on the same in-distribution input region and evaluated on two out-of-training input regions. The central objects are the four epistemic-UQ mechanisms — exact GP (posterior variance from kernel regression), deep ensembles (variance across independently initialized and trained networks), Monte-Carlo dropout (variance across stochastic forward passes), and stochastic variational inference (approximate posterior over weights) — compared by RMSE, negative log-likelihood, and miscalibration error. The mechanism carrying the result is the ensemble's spread: it is the only NN-based unce

What would settle it

Run the same four methods on the same training region but replace the reference output with direct numerical simulation or experimental measurements on a flow regime where the algebraic closure is known to be inaccurate; if deep ensembles no longer match or beat the exact GP's out-of-training negative log-likelihood, the paper's practical equivalence claim is specific to surrogate mimicry, not to genuine extrapolation.

Watch

Extended reading notes

Core claim

The paper's central empirical claim is a ranking: for a neural-network surrogate of an algebraic turbulence closure, predictive accuracy follows GP > DE > SVI > MCD, but the accuracy gap between the exact GP and deep ensembles is small enough that the two are effectively tied in out-of-training regions. More importantly for UQ, deep ensembles produce the lowest negative log-likelihood on both out-of-training test cases, indicating the most accurate probabilistic predictions on those regions, while SVI and DE share the best miscalibration error on one of the two cases. Since the GP costs O(n^3) per training step and the neural-network methods scale far more cheaply, the paper argues deep ense

Load-bearing premise

The benchmark treats the cited published algebraic turbulence closure as the ground truth for both accuracy and calibration; if that closure is inaccurate in the out-of-training regimes, the reported rankings show which method best mimics the surrogate, not which method gives trustworthy physical uncertainty.

Editorial extensions

If this is right

Practitioners building ML surrogate closures can adopt deep ensembles as the default epistemic-UQ method, getting near-GP accuracy and the best out-of-training NLL at a fraction of the training cost.
The exact GP's O(n^3) per-step complexity is prohibitive for CFD-scale training sets, so its accuracy edge does not translate to practical deployment; deep ensembles inherit the role of calibration reference.
Monte-Carlo dropout and stochastic variational inference underperform naive ensembling on this benchmark, cautioning against assuming that fancier approximate inference is automatically better.
The DE result suggests that out-of-training uncertainty in turbulence closure surrogates can be obtained by retraining with different seeds, a simple post-hoc procedure, instead of bespoke Bayesian machinery.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's ground truth is itself a surrogate, so the ranking measures fidelity to that reference closure, not to physical turbulence; if the reference closure fails in the out-of-training regime, all four methods fail together, and DE's NLL lead may not survive contact with measured data.
The same comparison on a known PDE solver with test parameters just outside the training box would show whether DE's extrapolation behavior generalizes to other smooth scalar regressors; a positive result would strengthen the case that ensemble variance is a universal out-of-distribution signal.
Recomputing the metrics after applying conformal prediction or a quantile recalibration to DE's variance would reveal how much of GP's remaining edge is a calibration artifact; if recalibrated DE still leads, the practical argument becomes stronger.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague — the abstract of this one is in good shape; the body is not. The supplied full text is mangled and even carries the arXiv ID of a different paper, so I could only review the abstract. That caps everything below.

What the abstract actually reports: a four-way epistemic-UQ comparison (exact GP, deep ensembles, MC dropout, SVI) on a published algebraic turbulence closure, with an explicit out-of-training focus. The claimed ranking — GP beats DE beats SVI beats MCD on accuracy, and DE has the best out-of-training NLL among the NN methods — is concretely useful if it holds. It's not a novel method paper, but it answers a practical question: can a cheap deep ensemble replace the expensive exact GP for this closure? That's worth knowing for CFD practitioners.

What the abstract does well is keep the scope honest: it frames the comparison as fidelity to the reference closure, reports the GP's O(n^3) cost, and doesn't overclaim miscalibration results (SVI and DE are only best for one of the two cases). No obvious internal contradiction between the stated numbers and the rankings.

The soft spots are real but mostly unverifiable from the abstract. No error bars, no dataset size, no train/test split description, no definition of 'out-of-training regions.' That alone would make me cautious about the ranking's robustness. The stress-test concern about NLL is also fair: each method constructs predictive variance differently, and without a shared observation-noise protocol, DE's best NLL could just reflect a narrow, overconfident predictive distribution. The abstract doesn't include calibration curves or any variance-scaling sensitivity check. Also, the whole benchmark is against the reference closure, not against measured turbulence data, so the 'trustworthy uncertainty' claim only extends to how well each method mimics that surrogate.

I can't judge the methods' implementation, the hyperparameters, or the actual tables, because the body is unreadable. If the authors supply a clean version, I'd want a referee to confirm the NLL computation, ask for calibration plots, and check the out-of-training region construction. The paper's a legitimate benchmark, not a breakthrough, and the abstract alone is not enough to cite.

Recommendation: tell the authors to fix the file and resubmit; with a readable body, this deserves a serious referee. Without one, I'd desk-reject the corrupted submission.

Referee Report

4 major / 4 minor

Summary. The manuscript benchmarks four uncertainty quantification (UQ) methods—exact Gaussian Process (GP), Deep Ensembles (DE), Monte-Carlo Dropout (MCD), and Stochastic Variational Inference (SVI)—as epistemic-UQ wrappers for a neural-network surrogate of a published algebraic turbulence closure. It reports in-training accuracy (GP RMSE 2.14e-5; DE RMSE 4.59e-4), out-of-training accuracy, miscalibration error, negative log-likelihood (NLL), and training-cost complexity. The main claimed conclusions are: GP is the most accurate but computationally expensive; DE is the best NN-based method on out-of-training NLL in both tested cases; SVI and DE have the best miscalibration error in one of the cases; overall accuracy ranking is GP > DE > SVI > MCD.

Significance. The problem is timely and practically important: ML-based turbulence closures are proposed as cheap surrogates, but their reliability outside the training distribution is a known bottleneck. If the reported results are reproducible, the paper would provide a useful practical message: deep ensembles can approximate GP-level accuracy and give reasonable out-of-training uncertainty at much lower cost than an exact GP. The benchmark design has a genuine strength: it compares against an external deterministic reference (the algebraic closure), so accuracy and calibration are not circularly defined. However, the contribution is currently conditional: the supplied full text is largely unreadable, no error bars or experimental-detail are given in the abstract, and the definitions of the out-of-training regions and the likelihood protocols are not inspectable. The paper's value therefore depends on a thorough revision that makes the experiments fully reproducible.

major comments (4)

[Abstract and results tables] All reported metrics—GP RMSE 2.14e-5, DE RMSE 4.59e-4, NLL, and miscalibration values—are point estimates. No error bars, number of independent seeds, repeated-initialization statistics, dataset size, or train/test split procedure are reported. The central ranking GP > DE > SVI > MCD and the claim that DE has the best out-of-training NLL cannot be assessed for statistical significance without these quantities.
[Abstract / definition of out-of-training] The core concept of the paper, 'out-of-training inputs', is never formally defined in the accessible text. The abstract and fragments refer to 'out-of-training regions' and 'two cases', but the input ranges, the distance from the training distribution, and the procedure for constructing these regions are absent. Without this, the central out-of-training comparison is not reproducible.
[Full text / likelihood and variance construction] NLL is used to rank UQ quality, but each method constructs predictive variance differently: GP uses kernel noise, DE uses ensemble variance, MCD uses Monte Carlo dropout variance, and SVI uses a variational posterior. The visible text does not specify a shared observation-noise or variance-calibration protocol. A narrow but overconfident predictive distribution can achieve artificially low NLL, especially on deterministic targets. The authors must report the exact likelihood definitions, ensemble sizes, dropout rates, SVI parameters, and include calibration curves or reliability diagrams and a sensitivity analysis with respect to variance scaling.
[Full text / manuscript integrity] The supplied full text is largely unreadable and contains repeated 'arXiv:2508.16882v1 [eess.IV] 23 Aug 2025' headers, which do not match the claimed paper ID. Equations, table entries, and method descriptions cannot be reliably verified from this text. This is a load-bearing issue for a referee: I cannot check the derivations of the reported miscalibration metrics or the numerical values in the tables. The manuscript must be provided in a clean, machine-readable form before a meaningful evaluation can be completed.

minor comments (4)

[Abstract] Typo: 'SV I > MCD' should read 'SVI > MCD'. Also, 'adaptation' is likely intended as 'adoption' in the first sentence.
[Abstract / footnote 1] The abstract cites 'a published algebraic turbulence closure1' but the footnote/reference does not appear in the supplied text. Please ensure the reference is complete and unambiguous.
[Tables/figures] The table and figure captions are unreadable in the supplied version. Each table and figure must be self-contained, with labeled metrics, units, and hyperparameter values.
[Reproducibility] No code or data availability statement is present. To support the empirical claims, please provide training/validation/test data generation scripts, model architecture details, and trained model checkpoints (or a public repository).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study is an empirical benchmark against an external published algebraic closure; no fitted input is renamed as a prediction.

full rationale

The paper is an empirical benchmark, not a derivation. It compares exact GP, Deep Ensembles, Monte-Carlo Dropout, and Stochastic Variational Inference on a fixed published algebraic turbulence closure used as the reference truth for both accuracy and UQ metrics. No target quantity is defined in terms of a predicted quantity, no fitted parameter is relabeled as a prediction, and no load-bearing self-citation is evident: the algebraic closure is an external published surrogate, and the NN methods are trained to match it rather than being defined by it. The abstract's accuracy ranking (GP > DE > SVI > MCD) and NLL ranking are measurement claims over held-out regions, not constructions. The concern that NLL comparisons may be sensitive to differing variance parameterizations, or that the algebraic closure is an imperfect ground truth for real turbulence, is a validity/external-benchmark concern, not circularity. The supplied full text is garbled and carries an unrelated arXiv ID, which prevents equation-level inspection, but nothing in the accessible text exhibits a specific reduction of a claimed prediction to its own inputs. Accordingly, the appropriate finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim rests on (1) hyperparameter choices that are not stated in the abstract, and (2) the adequacy of the reference closure and of the constructed out-of-training split as a proxy for deployment. No new physical entities are introduced; the paper is a benchmark of existing methods.

free parameters (1)

Unreported hyperparameters (ensemble size, dropout rate, SVI parameters, GP kernel params) = not stated in abstract
The abstract gives RMSE and NLL outcomes but no hyperparameter values. The accuracy and calibration ranking could depend on these choices, and they are unverifiable from the corrupted full text.

assumptions (3)

domain assumption The cited published algebraic turbulence closure is an adequate ground-truth reference for closure accuracy and UQ calibration.
The entire RMSE and miscalibration benchmark measures fidelity to this reference; its own error against real turbulence is not discussed at abstract level.
domain assumption The out-of-training input regions used in the two test cases faithfully represent extrapolation beyond the training distribution.
The paper's central contribution is out-of-training UQ, and the validity of that comparison depends on these regions actually lying outside the training regime, which cannot be verified from the abstract.
domain assumption Standard Bayesian approximation claims (dropout as approximate inference, variational posterior in SVI) are accepted as valid epistemic-UQ estimators.
The paper compares MCD and SVI as epistemic UQ methods without, at abstract level, questioning the known approximations these methods make.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Quantifying Out-of-Training Uncertainty of Neural-Network based Turbulence Closures." pith.science (2026). https://pith.science/paper/WJPLB4CW

@misc{pith2026250816891,
  author       = {Pith},
  title        = {Pith review of: Quantifying Out-of-Training Uncertainty of Neural-Network based Turbulence Closures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WJPLB4CW}},
  note         = {Machine review of arXiv:2508.16891}
}

abstract

Neural-Network (NN) based turbulence closures have been developed for being used as pre-trained surrogates for traditional turbulence closures, with the aim to increase computational efficiency and prediction accuracy of CFD simulations. The bottleneck to the widespread adaptation of these ML-based closures is the relative lack of uncertainty quantification (UQ) for these models. Especially, quantifying uncertainties associated with out-of-training inputs, that is when the ML-based turbulence closures are queried on inputs outside their training data regime. In the current paper, a published algebraic turbulence closure1 has been utilized to compare the quality of epistemic UQ between three NN-based methods and Gaussian Process (GP). The three NN-based methods explored are Deep Ensembles (DE), Monte-Carlo Dropout (MCD), and Stochastic Variational Inference (SVI). In the in-training results, we find the exact GP performs the best in accuracy with a Root Mean Squared Error (RMSE) of $2.14 \cdot 10^{-5}$ followed by the DE with an RMSE of $4.59 \cdot 10^{-4}$. Next, the paper discusses the performance of the four methods for quantifying out-of-training uncertainties. For performance, the Exact GP yet again is the best in performance, but has similar performance to the DE in the out-of-training regions. In UQ accuracy for the out-of-training case, SVI and DE hold the best miscalibration error for one of the cases. However, the DE performs the best in Negative Log-Likelihood for both out-of-training cases. We observe that for the current problem, in terms of accuracy GP > DE > SV I > MCD. The DE results are relatively robust and provide intuitive UQ estimates, despite performing naive ensembling. In terms of computational cost, the GP is significantly higher than the NN-based methods with a $O(n^3)$ computational complexity for each training step

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

1 extracted references · 1 canonical work pages

[1]

�� ...

work page Pith review arXiv 2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

�� ...

work page Pith review arXiv 2025