{"id":"3dde8248-2b1d-433b-8a41-6035c86f5b7c","arxiv_id":"2608.05995","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A posterior-risk definition of uncertainty that subsumes Bayesian and frequentist views is used to build a semi-synthetic GP benchmark with exact oracle aleatoric and epistemic targets.","lead":"The authors define predictive uncertainty as the expected loss of a predictor under the posterior distribution of plausible ground-truth functions, formally nesting Bayesian posterior variance and frequentist excess risk. They build a semi-synthetic GP benchmark with exact oracle aleatoric and epistemic targets and find that accurate prediction does not imply reliable uncertainty disentanglement.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark rankings may be specific to the correctly specified GP oracle; transfer to realistic misspecified settings is untested and load-bearing for the paper's practical recommendations.","rationale":"I read the paper in good faith. The theoretical core is sound: Definition 2.2 is a clean application of posterior predictive risk, Proposition 2.3 is a standard Bregman bias-variance identity (with the centroid equal to the expectation under the relevant convexity conditions), and the special-case recoveries in Section 2.4 are correct. The authors also state the key limitation themselves in Section 6, which is credit-worthy. The most load-bearing concern is that the benchmark's practical conclusions are only as good as the transfer of rankings from the GP oracle to realistic settings. This is the reader's weakest_assumption, and I agree. Other concerns raised by the reader—unverifiable code artifact, undocumented dev/held-out split, and unclear novelty relative to Kotelevskii et al. (2024)—are secondary: they affect reproducibility and attribution but do not threaten the internal validity of the theoretical claim. The proposed test would settle the transfer question directly. Since the paper's own limitation statement already qualifies the claims, the verdict remains CONDITIONAL; my read does not change the reader's verdict.","tokens_in":33300,"tokens_out":18046,"duration_ms":162941,"concrete_test":"Generate targets from a GP with kernel K1 (e.g., Matérn-5/2) but compute the oracle posterior m_S and sigma_f^2 using a different kernel K2 (e.g., RBF), holding the trained predictors fixed. Recompute the primary Spearman-rank metric of Figure 2 for all methods under the misspecified oracle and compare the resulting method ordering to the correctly specified ordering. If Kendall's tau between the two orderings is below 0.5, the rankings are oracle-specific; if it is high, the transfer concern is mitigated. As a complementary check, repeat with a non-GP generative process (e.g., a random neural network with known weights plus noise) and a posterior computed by MCMC.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's oracle targets are internally valid: data are drawn from a GP prior and the same prior is used to compute p(f|S), so Eq. (4) correctly computes sigma_eps^2 + sigma_f^2 + (fhat - m_S)^2. The load-bearing concern is external validity of the method rankings. Every oracle epistemic target used to rank methods encodes the structure of one specific, smooth Gaussian prior over functions. Real-world epistemic uncertainty need not have that structure, and the paper's practical takeaways (deep ensembles, FSP-Laplace, CatBoost-KGB as reliable starting points; EDL strongest in calibration) are derived entirely from rankings against this GP oracle. The paper acknowledges this in Section 6 as an open limitation but does not test it. Because the central contribution is an evaluation benchmark 'beyond proxies,' the usefulness of the rankings depends on whether they are preserved under misspecification of the oracle's prior or under non-GP generative processes. Without such a test, the benchmark establishes internal consistency but not external relevance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a unified definition of uncertainty as sample-conditional posterior risk: for a fixed input x and observed sample S, the uncertainty of a deployed predictor f_hat is the expected loss under the posterior distribution over plausible ground-truth functions (Eq. 2). Under squared loss, this decomposes into aleatoric uncertainty sigma_eps^2(x), posterior variance sigma_f^2(x), and an estimator-dependent squared bias term (f_hat(x)-m_S(x))^2 (Eq. 4). The paper shows that the Bayesian posterior-variance view and the frequentist pointwise-excess-risk view are recovered as special cases, and it develops a semi-synthetic benchmark in which real covariates are paired with targets generated from a Gaussian-process prior, so that oracle aleatoric and epistemic targets are analytically computable. The empirical study evaluates a range of uncertainty-disentanglement methods on a development suite and a held-out benchmark suite, using Spearman rank correlation against the oracle targets as the primary metric, and reports ablations over architecture, activation, tuning objective, and optimizer. The headline findings are that predictive accuracy does not guarantee good uncertainty disentanglement, that ensemble-style neural methods and several tree-based methods provide the strongest epistemic rankings, and that EDL is strongest in aleatoric/epistemic calibration.","tokens_in":33461,"tokens_out":6864,"duration_ms":79548,"significance":"The theoretical core of the paper is sound and valuable. The derivation from Eq. (2) to Eq. (4) correctly applies the Bregman bias-variance identity, and the special-case recoveries of posterior-variance-based and excess-risk-based epistemic uncertainty are clean and clearly stated. The explicit separation of estimator-dependent bias from posterior spread is a useful conceptual contribution, as is the point that standard posterior-spread measures can miss misspecification and optimization error. The benchmark is also carefully constructed: the GP-based oracle targets are correctly specified by construction, the development/held-out split is a genuine strength, and the empirical study reports standard errors and multiple diagnostics, including risk-coverage curves, calibration plots, and a decomposition of epistemic estimates into posterior-variance and bias components. If the method rankings are taken as internally valid, the paper provides concrete practical information about which uncertainty estimators align with a well-defined oracle target.","major_comments":[{"comment":"The benchmark's oracle targets are internally valid because the data-generating GP prior and the posterior used for computing Eq. (4) coincide, but the paper's practical recommendations in Section 7 (deep ensembles, FSP-Laplace, and CatBoost-KGB as reliable starting points) are derived solely from rankings against this correctly specified GP oracle. Since the stated contribution is an evaluation 'beyond proxies,' the external relevance of these rankings is load-bearing. The paper should add a misspecification stress test, for example by generating targets from a different kernel than the one used for the posterior, or from a non-GP function class, and showing whether the method rankings are preserved. Without such a test, the benchmark establishes internal consistency but does not substantiate the general practical takeaways.","section":"Sections 4.1, 5.2, 6, and 7"},{"comment":"The oracle epistemic target used for ranking methods is method-specific: it contains the term (f_hat(x)-m_S(x))^2, so the target depends on the predictor being evaluated. A method with a predictive mean close to the GP posterior mean faces a target dominated by sigma_f^2, whereas a method with a biased mean faces a target with a substantial additional squared-bias component. Cross-method comparisons of Spearman rank correlation against such method-specific targets can therefore conflate the quality of the uncertainty estimate with differences in the composition of the target. The paper should report, alongside Figure 2, correlations against a common target such as sigma_f^2 alone, or otherwise demonstrate that the main method rankings are stable when the target composition changes. Figure 13 partially addresses this issue, but it is relegated to the appendix and does not directly test the stability of the headline ranking.","section":"Eq. (4) and Section 5.2"},{"comment":"Several of the methods that rank well in the main benchmark are themselves GP-based or GP-motivated (notably CatBoost-KGB, which explicitly samples from a GP posterior, and FSP-Laplace, which regularizes toward a GP prior). Given that the oracle targets are computed from a GP posterior, these methods may benefit from an advantage that is inherent to the evaluation protocol rather than to their general uncertainty-disentanglement quality. The paper should either analyze this potential confound explicitly, for example by comparing method rankings on tasks where the prior is deliberately misspecified, or temper the recommendations so that they are clearly restricted to the correctly specified GP setting.","section":"Sections 4.1 and 5.2"}],"minor_comments":[{"comment":"The empirical benchmark is instantiated only under squared loss, although Appendix A derives decompositions for CRPS and negative log-likelihood. The main text should state more prominently that all benchmark conclusions are specific to the squared-loss risk functional, since Implication 2 notes that the semantics of aleatoric and epistemic uncertainty depend on the chosen loss or scoring rule.","section":"Section 1 and Appendix A"},{"comment":"The text says that 'the ablations ran in between 3-9 hours while the final evaluation took 4hours'; '4hours' should be '4 hours', and 'ran in between' is awkwardly phrased.","section":"Section 5.1"},{"comment":"The name 'SW AG' appears with a space in multiple places (e.g., Section 5 and Appendix D.4), which is inconsistent with the standard notation 'SWAG' used elsewhere; please standardize.","section":"Throughout"},{"comment":"The reported Kendall's W value of approximately 0.46 for method-rank preservation across datasets is given without a confidence interval or significance test; since the number of datasets is small (seven), the uncertainty in W should be quantified or at least discussed.","section":"Section 5.2"},{"comment":"The dataset presets report 'Posterior std median' values but do not explain how this quantity is computed or how it should be interpreted relative to the aleatoric std. A brief definition in the appendix would improve reproducibility.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid contribution to uncertainty quantification benchmarking, and the theoretical derivation is sound. The main concern is that the empirical rankings, which drive the practical recommendations, are obtained entirely from a correctly specified GP oracle, and the paper does not test transfer to misspecified or non-GP settings. This is a fixable limitation within the manuscript's scope: adding a misspecification robustness experiment would substantially strengthen the external validity of the benchmark. I would not recommend rejection, but the current version does not yet fully support the general practical claims made in the conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: the theory is clean, the benchmark is transparent, and the one load-bearing weakness is acknowledged but untested external validity.\n\nWhat's new: the sample-conditional posterior risk definition (Eq. 2) is a correct and useful unification of the Bayesian posterior-variance and frequentist excess-risk views, and the paper makes explicit which assumption recovers which view — predictor equals posterior mean for the former, Dirac posterior for the latter. The Bregman decomposition itself comes from Pfau and Adlam et al., but the estimator-aware, sample-conditional framing combined with oracle GP targets is new. The empirical study is genuinely careful: development/held-out separation, error bars, ablations over width, depth, activation, optimizer, and tuning objective, plus a diagnostic decomposition showing most epistemic methods track posterior variance while showing weak sensitivity to squared bias. That last finding is the paper's most useful empirical contribution.\n\nSoft spots, in proportion. The main one is external validity. The oracle targets are internally correct — data are drawn from a GP prior and the same prior is used to compute p(f|S), so Eq. (4) is exactly right under the generative process. But the method rankings are measured against one smooth Gaussian prior, and whether those rankings survive prior misspecification or non-GP generative processes is untested. The authors flag this in Section 6 but do not test it. It is a real limitation, not a fatal flaw: the paper positions the benchmark as a controlled reference setting, and the definition itself does not depend on the GP.\n\nMinor: the empirical benchmark instantiates only squared loss (CRPS and NLL specializations appear in the appendix but are not evaluated); the code artifact is promised but not verifiable from the preprint; and the novelty boundary relative to Kotelevskii et al. (2024) could use one clarifying sentence. The reader's concern about the dev/held-out split is unfounded — Section 5 states it clearly: design choices are fixed on a seven-dataset development suite, then the held-out suite is used only for the final comparison.\n\nWho this is for: researchers working on uncertainty disentanglement, evidential regression, or UQ benchmarking. The benchmark is a plausible reference point even if the rankings turn out to be setting-specific. This deserves a serious referee; I would ask the authors to test robustness under prior misspecification and to make the code verifiable.","headline":"Sound theory and a transparent oracle benchmark for uncertainty disentanglement, with untested external validity as the main caveat.","tokens_in":34026,"tokens_out":3313,"would_cite":true,"duration_ms":30519,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper defines uncertainty as a predictor's expected loss under the posterior over plausible ground-truth functions, unifying the Bayesian variance view and the frequentist excess-risk view, and uses GP semi-synthetic regression to…","keywords":["uncertainty disentanglement","epistemic uncertainty","aleatoric uncertainty","posterior risk","Bregman divergence","Gaussian process prior","semi-synthetic benchmark","oracle evaluation"],"falsifier":"Re-run the benchmark suite with targets drawn from a non-GP generative process (for example, a Bayesian neural network prior or a Student-t noise model) and compute approximate oracle targets by Hamiltonian Monte Carlo; if the methods that rank highest under the GP oracle do not also lead in Spearman correlation against these approximate targets, the transfer assumption fails. A cheaper check is to recompute oracle rankings under CRPS on the existing GP presets; if method rankings flip substantially, the current conclusions are specific to squared loss.","tokens_in":33081,"feed_emoji":"🎯","tokens_out":8277,"duration_ms":69670,"temperature":0.7,"pith_summary":"Uncertainty estimates are only useful if they reflect what a deployed predictor does not know, but the field has no agreed target for epistemic uncertainty, and benchmarks substitute proxy tasks such as out-of-distribution detection. This paper proposes sample-conditional posterior risk as that target: the expected loss of the deployed predictor under the distribution of ground-truth functions that remain plausible given the observed data. Under squared loss the target splits cleanly into aleatoric noise, posterior variance over functions, and a squared estimator-bias term, which shows that Bayesian posterior variance and frequentist excess risk are two ends of one spectrum. The paper adds a benchmark in which targets are generated from a known Gaussian-process prior, so oracle aleatoric and epistemic uncertainty are exactly computable, and uses it to show that predictive accuracy does not imply trustworthy disentanglement.","feed_headline":"Epistemic uncertainty equals posterior variance plus estimator bias","feed_subtitle":"A semi-synthetic GP benchmark scores models against true uncertainty targets and finds accuracy does not imply reliable disentanglement.","key_machinery":"The load-bearing object is the sample-conditional posterior risk defined in Eq. (2), $R_S(\\hat f;x) = \\mathbb{E}_{f \\sim \\Pi(\\cdot|S)} \\mathbb{E}_{Y \\sim P(\\cdot|x,f)}[\\ell(\\hat f(x), Y)]$, together with its Bregman decomposition (Proposition 2.3), which splits the risk into a Bayes-error (aleatoric) term, a generalized variance over the posterior, and a generalized bias of the estimator. Under squared loss this collapses to the identity $R_S(\\hat f;x) = \\sigma_\\epsilon^2(x) + \\sigma_f^2(x) + (\\hat f(x)-m_S(x))^2$, which is what turns the definition into a computable evaluation target: in the semi-synthetic GP setting the posterior $\\Pi(\\cdot|S)$ is the same Gaussian process that generated the targets, so $\\sigma_\\epsilon^2(x)$, $\\sigma_f^2(x)$, and $m_S(x)$ are all known analytically. The benchmark estimates each method's reported epistemic and aleatoric uncertainty and scores them by Spearman rank correlation against these oracle targets, with ablation studies of architecture, activation, tuning objective, and optimizer serving to control the many degrees of freedom that otherwise dominate method comparisons.","core_discovery":"The central claim is that uncertainty disentanglement should be anchored to sample-conditional pointwise posterior risk, $R_S(\\hat f; x) = \\mathbb{E}_{Y \\sim P(\\cdot|x,S)}[\\ell(\\hat f(x), Y)]$, the expected loss of the actually deployed predictor under the posterior over plausible latent ground-truth functions given the observed sample $S$. For squared loss with additive noise this becomes $$R_S(\\hat f; x) = \\sigma_\\$epsilon^{2}$(x) + \\$sigma_f^{2}$(x) + (\\hat f(x) - m_S(x))^2,$$ so aleatoric uncertainty is the noise variance, and epistemic uncertainty is the sum of the posterior variance $\\sigma_f^2(x)$ and the squared deviation of the estimator from the posterior mean $m_S(x)$. The Bayesian posterior-variance view and the frequentist pointwise-excess-risk view are both recovered as special cases: the former when the deployed predictor equals the posterior mean, the latter when the posterior collapses to a Dirac measure at one ground-truth function. The paper then constructs semi-synthetic regression datasets with real covariates and GP-sampled targets, making the oracle epistemic and aleatoric targets exactly computable, and finds that current methods, evaluated by rank correlation against these targets, are far from oracle-level, generally track the posterior-variance component more than the bias component, and perform unevenly across datasets.","pith_inferences":["Beyond the paper: recomputing oracle rankings under CRPS or NLL (whose decompositions the appendix derives) would directly test whether the benchmark conclusions depend on the squared-loss choice, and method rankings could plausibly shift.","Beyond the paper: the implicit transfer claim is that GP-oracle rankings predict performance in realistic misspecified settings; this could be tested by generating data from a deliberately non-GP process (e.g., a Bayesian neural network prior or heavy-tailed noise) and checking whether the top-ranked methods under the GP oracle still lead against approximate MCMC targets.","Beyond the paper: the definition suggests a practical diagnostic for safety-critical deployment, namely computing the squared-bias term $(\\hat f(x)-m_S(x))^2$ whenever a posterior approximation is available, since methods that ignore it will systematically understate epistemic uncertainty under misspecification.","Beyond the paper: the benchmark's oracle targets could be reused for model selection or early stopping, because they separate irreducible noise from reducible estimator error pointwise in a way proxy tasks cannot."],"forward_implications":["If posterior risk is the right target, then a method that reports only posterior spread can understate epistemic uncertainty whenever the deployed predictor deviates from the posterior mean, and a method that reports only pointwise excess risk can understate uncertainty in data-sparse regions.","Oracle evaluation without proxies is feasible in controlled settings: real covariates with GP-generated targets and a correctly specified posterior yield exact targets for aleatoric and epistemic uncertainty at every evaluation point.","Accurate point prediction does not imply trustworthy uncertainty disentanglement: in the benchmark, models with similar predictive performance differ substantially in their Spearman correlation against the oracle targets.","Deep ensembles, FSP-Laplace, and CatBoost-KGB emerge as the most reliable general-purpose starting points among tested methods, while aleatoric rankings remain weakly separated and dataset-dependent.","Most epistemic estimates align more with the posterior-variance component than with the full posterior excess-risk target, meaning current methods largely miss the estimator-bias contribution to epistemic uncertainty."],"supporting_citations":[{"why":"Supplies the pointwise-risk view of uncertainty that Definition 2.1 extends to the posterior-risk setting.","marker":"[Lahlou et al., 2021]"},{"why":"Shows how predictive uncertainty measures can be generated from risk, the tradition this unified definition continues.","marker":"[Kotelevskii et al., 2024]"},{"why":"Provides the Bregman bias–variance identity used in Proposition 2.3 to decompose posterior risk.","marker":"[Pfau, 2025]"},{"why":"Frames bias–variance trade-offs for Bregman divergences, the decomposition framework the paper applies to uncertainty.","marker":"[Adlam et al., 2022]"},{"why":"Defines the aleatoric/epistemic distinction the paper aims to unify.","marker":"[Hüllermeier and Waegeman, 2021]"},{"why":"Argues oracle epistemic uncertainty is hard to define, motivating the construction of explicit oracle targets.","marker":"[Jürgens et al., 2024]"},{"why":"Documents why ML models fail to fully capture epistemic uncertainty, a benchmark baseline the paper addresses.","marker":"[Jiménez et al., 2025]"},{"why":"Prior uncertainty-disentanglement benchmark that relies on proxy tasks, which the paper replaces with oracle targets.","marker":"[Mucsányi et al., 2024]"},{"why":"Supplies the estimation/approximation-error decomposition used in the diagnostic appendix.","marker":"[Vapnik, 2000]"},{"why":"Defines deep ensembles, the method family the benchmark identifies among the strongest epistemic performers.","marker":"[Lakshminarayanan et al., 2017]"}],"fun_headline_variants":["Uncertainty as posterior risk: a unified view","Posterior risk unifies aleatoric and epistemic uncertainty","Accuracy doesn't guarantee reliable uncertainty disentanglement","Semi-synthetic benchmark reveals uncertainty estimation gaps","From proxies to posterior risk: better uncertainty evaluation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole empirical ranking rests on the assumption that methods ranked against a perfectly specified GP oracle, where the posterior used for evaluation is the same distribution that generated the targets, will perform similarly in realistic settings where the model is misspecified and no exact posterior exists; the paper states this transfer is untested.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty as posterior risk: a unified view","Posterior risk unifies aleatoric and epistemic uncertainty","Accuracy doesn't guarantee reliable uncertainty disentanglement","Semi-synthetic benchmark reveals uncertainty estimation gaps","From proxies to posterior risk: better uncertainty evaluation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000292,"raw_usage":{"total_tokens":1769,"prompt_tokens":1076,"completion_tokens":693,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":692,"completion_tokens_details":{"reasoning_tokens":620}},"tokens_in":692,"tokens_out":693,"duration_ms":6043,"temperature":1.0,"reasoning_tokens":620,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T19:36:34.198566+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the benchmark suite with targets drawn from a non-GP generative process (for example, a Bayesian neural network prior or a Student-t noise model) and compute approximate oracle targets by Hamiltonian Monte Carlo; if the methods that rank highest under the GP oracle do not also lead in Spearman correlation against these approximate targets, the transfer assumption fails. A cheaper check is to recompute oracle rankings under CRPS on the existing GP presets; if method rankings flip substantially, the current conclusions are specific to squared loss.","supporting_citations":[{"cited_title":", urldate =","cited_arxiv_id":null,"evidence_quote":"Supplies the estimation/approximation-error decomposition used in the diagnostic appendix."}],"review_version":1}