{"id":"83da8b14-5872-4ed7-9c6c-f2bed25097fc","arxiv_id":"2508.16891","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":1,"one_line_summary":"On this turbulence closure benchmark, exact GP beats deep ensembles, SVI, and MC dropout in accuracy, while deep ensembles give the best out-of-training log-likelihood at far lower cost.","lead":"This paper compares four ways to estimate uncertainty in neural network based turbulence closures, especially for inputs outside the training data. It finds a Gaussian process is most accurate but expensive, while deep ensembles give nearly as good uncertainty estimates at much lower cost.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DE's best out-of-training NLL may be an artifact of uncalibrated predictive variances; without a shared noise-level protocol, NLL rankings do not establish better UQ.","rationale":"The reader's verdict (UNVERDICTED) is appropriate because the full text is corrupt and the central claim cannot be independently checked. The reader's identified weakest assumption—that the benchmark uses the algebraic closure as ground truth—is a legitimate external-validity limitation, but it is not the most load-bearing concern for the paper's internal claim: the paper explicitly frames the comparison as surrogate-model fidelity to that closure, so the closure is the intended target. A more pressing concern is whether the NLL comparison is methodologically fair, since NLL depends heavily on how predictive variance is parameterized and calibrated. If the variance protocols differ, the out-of-training NLL ranking could be an artifact, which would directly undermine the conclusion that DE is the best NN-based UQ method. The proposed test—recomputing NLL after equalizing variance calibration—would settle this. Since neither this nor any other technical detail can be verified from the corrupted text, the verdict remains UNVERDICTED, unchanged from the reader's assessment.","tokens_in":18320,"tokens_out":4986,"duration_ms":66993,"concrete_test":"Obtain the uncorrupted full text. Locate the exact predictive-variance construction for each of DE, MCD, SVI, and GP. Recompute the out-of-training NLL after replacing each method's predictive variance with a common, holdout-tuned variance (e.g., the empirical residual variance on a validation subset). If DE's NLL advantage over GP/SVI disappears or reverses under equalized variance, the reported ranking reflects a variance-calibration artifact rather than a genuine UQ superiority.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central practical claim is that deep ensembles provide 'intuitive UQ estimates' and that DE has the best out-of-training negative log-likelihood among NN-based methods. NLL compares predictive densities, but each method constructs predictive variance differently: GP learns a kernel noise hyperparameter, DE uses ensemble variance, MCD uses Monte Carlo dropout variance, and SVI uses a variational posterior. If the observation-noise level or the variance calibration is not handled comparably across methods, a method with narrow but overconfident predictive distributions can achieve artificially low NLL, particularly on out-of-training points where the target is deterministic (the algebraic closure output). The abstract reports only aggregate RMSE and NLL; it does not show calibration curves, reliability diagrams, or a sensitivity analysis with respect to variance scaling. The supplied full text is corrupted beyond usability and carries an arXiv ID from another paper (2508.16882), so the exact likelihood definitions, training splits, and out-of-training regions cannot be inspected. Therefore the claim that DE provides trustworthy UQ in out-of-training regimes is not yet substantiated by the accessible evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript benchmarks four uncertainty quantification (UQ) methods—exact Gaussian Process (GP), Deep Ensembles (DE), Monte-Carlo Dropout (MCD), and Stochastic Variational Inference (SVI)—as epistemic-UQ wrappers for a neural-network surrogate of a published algebraic turbulence closure. It reports in-training accuracy (GP RMSE 2.14e-5; DE RMSE 4.59e-4), out-of-training accuracy, miscalibration error, negative log-likelihood (NLL), and training-cost complexity. The main claimed conclusions are: GP is the most accurate but computationally expensive; DE is the best NN-based method on out-of-training NLL in both tested cases; SVI and DE have the best miscalibration error in one of the cases; overall accuracy ranking is GP > DE > SVI > MCD.","tokens_in":18506,"tokens_out":5257,"duration_ms":67036,"significance":"The problem is timely and practically important: ML-based turbulence closures are proposed as cheap surrogates, but their reliability outside the training distribution is a known bottleneck. If the reported results are reproducible, the paper would provide a useful practical message: deep ensembles can approximate GP-level accuracy and give reasonable out-of-training uncertainty at much lower cost than an exact GP. The benchmark design has a genuine strength: it compares against an external deterministic reference (the algebraic closure), so accuracy and calibration are not circularly defined. However, the contribution is currently conditional: the supplied full text is largely unreadable, no error bars or experimental-detail are given in the abstract, and the definitions of the out-of-training regions and the likelihood protocols are not inspectable. The paper's value therefore depends on a thorough revision that makes the experiments fully reproducible.","major_comments":[{"comment":"All reported metrics—GP RMSE 2.14e-5, DE RMSE 4.59e-4, NLL, and miscalibration values—are point estimates. No error bars, number of independent seeds, repeated-initialization statistics, dataset size, or train/test split procedure are reported. The central ranking GP > DE > SVI > MCD and the claim that DE has the best out-of-training NLL cannot be assessed for statistical significance without these quantities.","section":"Abstract and results tables"},{"comment":"The core concept of the paper, 'out-of-training inputs', is never formally defined in the accessible text. The abstract and fragments refer to 'out-of-training regions' and 'two cases', but the input ranges, the distance from the training distribution, and the procedure for constructing these regions are absent. Without this, the central out-of-training comparison is not reproducible.","section":"Abstract / definition of out-of-training"},{"comment":"NLL is used to rank UQ quality, but each method constructs predictive variance differently: GP uses kernel noise, DE uses ensemble variance, MCD uses Monte Carlo dropout variance, and SVI uses a variational posterior. The visible text does not specify a shared observation-noise or variance-calibration protocol. A narrow but overconfident predictive distribution can achieve artificially low NLL, especially on deterministic targets. The authors must report the exact likelihood definitions, ensemble sizes, dropout rates, SVI parameters, and include calibration curves or reliability diagrams and a sensitivity analysis with respect to variance scaling.","section":"Full text / likelihood and variance construction"},{"comment":"The supplied full text is largely unreadable and contains repeated 'arXiv:2508.16882v1 [eess.IV] 23 Aug 2025' headers, which do not match the claimed paper ID. Equations, table entries, and method descriptions cannot be reliably verified from this text. This is a load-bearing issue for a referee: I cannot check the derivations of the reported miscalibration metrics or the numerical values in the tables. The manuscript must be provided in a clean, machine-readable form before a meaningful evaluation can be completed.","section":"Full text / manuscript integrity"}],"minor_comments":[{"comment":"Typo: 'SV I > MCD' should read 'SVI > MCD'. Also, 'adaptation' is likely intended as 'adoption' in the first sentence.","section":"Abstract"},{"comment":"The abstract cites 'a published algebraic turbulence closure1' but the footnote/reference does not appear in the supplied text. Please ensure the reference is complete and unambiguous.","section":"Abstract / footnote 1"},{"comment":"The table and figure captions are unreadable in the supplied version. Each table and figure must be self-contained, with labeled metrics, units, and hyperparameter values.","section":"Tables/figures"},{"comment":"No code or data availability statement is present. To support the empirical claims, please provide training/validation/test data generation scripts, model architecture details, and trained model checkpoints (or a public repository).","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The full text provided to me appears to be a corrupted extraction; it carries an arXiv ID from another paper (2508.16882v1) and is mostly unreadable. If this is an input-pipeline artifact rather than the actual arXiv submission, the paper should be re-evaluated from the clean PDF. My major comments about the full text are based on the supplied material. The scientific claims in the abstract are plausible and the benchmark idea is sound, but the missing statistical details, undefined out-of-training regions, and unverified text block major revision before a final decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — the abstract of this one is in good shape; the body is not. The supplied full text is mangled and even carries the arXiv ID of a different paper, so I could only review the abstract. That caps everything below.\n\nWhat the abstract actually reports: a four-way epistemic-UQ comparison (exact GP, deep ensembles, MC dropout, SVI) on a published algebraic turbulence closure, with an explicit out-of-training focus. The claimed ranking — GP beats DE beats SVI beats MCD on accuracy, and DE has the best out-of-training NLL among the NN methods — is concretely useful if it holds. It's not a novel method paper, but it answers a practical question: can a cheap deep ensemble replace the expensive exact GP for this closure? That's worth knowing for CFD practitioners.\n\nWhat the abstract does well is keep the scope honest: it frames the comparison as fidelity to the reference closure, reports the GP's O(n^3) cost, and doesn't overclaim miscalibration results (SVI and DE are only best for one of the two cases). No obvious internal contradiction between the stated numbers and the rankings.\n\nThe soft spots are real but mostly unverifiable from the abstract. No error bars, no dataset size, no train/test split description, no definition of 'out-of-training regions.' That alone would make me cautious about the ranking's robustness. The stress-test concern about NLL is also fair: each method constructs predictive variance differently, and without a shared observation-noise protocol, DE's best NLL could just reflect a narrow, overconfident predictive distribution. The abstract doesn't include calibration curves or any variance-scaling sensitivity check. Also, the whole benchmark is against the reference closure, not against measured turbulence data, so the 'trustworthy uncertainty' claim only extends to how well each method mimics that surrogate.\n\nI can't judge the methods' implementation, the hyperparameters, or the actual tables, because the body is unreadable. If the authors supply a clean version, I'd want a referee to confirm the NLL computation, ask for calibration plots, and check the out-of-training region construction. The paper's a legitimate benchmark, not a breakthrough, and the abstract alone is not enough to cite.\n\nRecommendation: tell the authors to fix the file and resubmit; with a readable body, this deserves a serious referee. Without one, I'd desk-reject the corrupted submission.","headline":"Readable abstract, unreadable body; the abstract suggests a useful benchmark result, but the NLL claim needs a variance-protocol check before it supports the DE recommendation.","tokens_in":19079,"tokens_out":2815,"would_cite":false,"duration_ms":34104,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deep ensembles rival exact Gaussian processes for turbulence-closure uncertainty.","keywords":["epistemic uncertainty","turbulence closure","deep ensembles","Gaussian process","Monte-Carlo dropout","stochastic variational inference","out-of-distribution generalization","CFD surrogate modeling"],"falsifier":"Run the same four methods on the same training region but replace the reference output with direct numerical simulation or experimental measurements on a flow regime where the algebraic closure is known to be inaccurate; if deep ensembles no longer match or beat the exact GP's out-of-training negative log-likelihood, the paper's practical equivalence claim is specific to surrogate mimicry, not to genuine extrapolation.","tokens_in":18157,"feed_emoji":"🌊","tokens_out":7322,"duration_ms":72506,"temperature":0.7,"pith_summary":"This paper asks which uncertainty-quantification method a practitioner should attach to a neural-network surrogate for a turbulence closure when the surrogate is queried outside its training regime. Using a published algebraic turbulence closure as the reference output, the authors compare exact Gaussian processes (GP), deep ensembles (DE), Monte-Carlo dropout (MCD), and stochastic variational inference (SVI). They find the exact GP is the most accurate, but deep ensembles come close, deliver the best out-of-training negative log-likelihood of all four methods, and avoid the GP's O(n^3) per-step cost. The paper concludes that simple retraining with different seeds, rather than bespoke Bayesian machinery, is a practical source of epistemic uncertainty for these surrogates.","feed_headline":"Swap Gaussian process for deep ensembles in turbulence closures","feed_subtitle":"A four-way benchmark finds naive ensembling matches GP accuracy and tops out-of-training likelihood at a fraction of the cost.","key_machinery":"The benchmark setup: a published algebraic turbulence closure acts as the reference output; each method is trained on the same in-distribution input region and evaluated on two out-of-training input regions. The central objects are the four epistemic-UQ mechanisms — exact GP (posterior variance from kernel regression), deep ensembles (variance across independently initialized and trained networks), Monte-Carlo dropout (variance across stochastic forward passes), and stochastic variational inference (approximate posterior over weights) — compared by RMSE, negative log-likelihood, and miscalibration error. The mechanism carrying the result is the ensemble's spread: it is the only NN-based unce","core_discovery":"The paper's central empirical claim is a ranking: for a neural-network surrogate of an algebraic turbulence closure, predictive accuracy follows GP > DE > SVI > MCD, but the accuracy gap between the exact GP and deep ensembles is small enough that the two are effectively tied in out-of-training regions. More importantly for UQ, deep ensembles produce the lowest negative log-likelihood on both out-of-training test cases, indicating the most accurate probabilistic predictions on those regions, while SVI and DE share the best miscalibration error on one of the two cases. Since the GP costs O(n^3) per training step and the neural-network methods scale far more cheaply, the paper argues deep ense","pith_inferences":["The paper's ground truth is itself a surrogate, so the ranking measures fidelity to that reference closure, not to physical turbulence; if the reference closure fails in the out-of-training regime, all four methods fail together, and DE's NLL lead may not survive contact with measured data.","The same comparison on a known PDE solver with test parameters just outside the training box would show whether DE's extrapolation behavior generalizes to other smooth scalar regressors; a positive result would strengthen the case that ensemble variance is a universal out-of-distribution signal.","Recomputing the metrics after applying conformal prediction or a quantile recalibration to DE's variance would reveal how much of GP's remaining edge is a calibration artifact; if recalibrated DE still leads, the practical argument becomes stronger."],"forward_implications":["Practitioners building ML surrogate closures can adopt deep ensembles as the default epistemic-UQ method, getting near-GP accuracy and the best out-of-training NLL at a fraction of the training cost.","The exact GP's O(n^3) per-step complexity is prohibitive for CFD-scale training sets, so its accuracy edge does not translate to practical deployment; deep ensembles inherit the role of calibration reference.","Monte-Carlo dropout and stochastic variational inference underperform naive ensembling on this benchmark, cautioning against assuming that fancier approximate inference is automatically better.","The DE result suggests that out-of-training uncertainty in turbulence closure surrogates can be obtained by retraining with different seeds, a simple post-hoc procedure, instead of bespoke Bayesian machinery."],"supporting_citations":[],"fun_headline_variants":["Deep ensembles match GP, cut cost in turbulence UQ","For ML turbulence closures, deep ensembles win out-of-training likelihood","Naive ensembling: robust UQ at a fraction of GP cost","Deep ensembles edge GP on out-of-training turbulence UQ"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The benchmark treats the cited published algebraic turbulence closure as the ground truth for both accuracy and calibration; if that closure is inaccurate in the out-of-training regimes, the reported rankings show which method best mimics the surrogate, not which method gives trustworthy physical uncertainty.","fun_headline_variants_meta":{"raw":{"variants":["Deep ensembles match GP, cut cost in turbulence UQ","For ML turbulence closures, deep ensembles win out-of-training likelihood","Naive ensembling: robust UQ at a fraction of GP cost","Deep ensembles edge GP on out-of-training turbulence UQ"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000247,"raw_usage":{"total_tokens":1454,"prompt_tokens":891,"completion_tokens":563,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":491}},"tokens_in":635,"tokens_out":563,"duration_ms":7002,"temperature":1.0,"reasoning_tokens":491,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:07:32.176700+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same four methods on the same training region but replace the reference output with direct numerical simulation or experimental measurements on a flow regime where the algebraic closure is known to be inaccurate; if deep ensembles no longer match or beat the exact GP's out-of-training negative log-likelihood, the paper's practical equivalence claim is specific to surrogate mimicry, not to genuine extrapolation.","supporting_citations":[],"review_version":1}