arxiv: 2605.06413 · v1 · submitted 2026-05-07 · 📊 stat.ML · cs.LG

Recognition: unknown

Decoupled PFNs: Identifiable Epistemic-Aleatoric Decomposition via Structured Synthetic Priors

Richard Bergna , Stefan Depeweg , Jos\'e Miguel Hern\'andez-Lobato

Authors on Pith no claims yet

Pith reviewed 2026-05-08 04:51 UTC · model grok-4.3

classification 📊 stat.ML cs.LG

keywords Prior-Fitted Networksepistemic uncertaintyaleatoric uncertaintyBayesian optimizationuncertainty decompositionmeta-learningsynthetic priors

0 comments

The pith

By controlling synthetic data generation, Prior-Fitted Networks can learn identifiable decompositions of epistemic and aleatoric uncertainty for better decision-making.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper demonstrates that separating epistemic uncertainty about the underlying function from aleatoric observation noise is generally impossible using only the posterior predictive distribution. Prior-Fitted Networks overcome this by meta-learning over synthetic tasks where the generator provides explicit labels for the noiseless signal and the noise variance at each query point. This allows training a model with two separate output heads, one for the latent signal distribution and one for the aleatoric noise, whose convolution yields the full predictive. The resulting epistemic uncertainty can then be used directly in acquisition functions for tasks like Bayesian optimization, avoiding wasteful exploration of regions with high noise rather than high model uncertainty. Sympathetic readers would care because this addresses a key limitation in applying Bayesian methods to real-world sequential decisions under noise.

Core claim

We show that this epistemic--aleatoric split is not identifiable in general from the posterior predictive distribution alone, even when that distribution is known exactly. We then exploit a distinctive advantage of PFNs: because the synthetic data-generating process is under our control, each task can contain an explicit latent signal and noise function, and the generator can provide query-level labels for both the noiseless target and the observation-noise variance. We use these labels to train a decoupled PFN with separate latent-signal and aleatoric heads. The observation-level predictive is induced by convolving the latent signal distribution with the learned noise model.

What carries the argument

Decoupled Prior-Fitted Network architecture with separate heads for the latent signal distribution and the aleatoric noise variance, trained using structured synthetic priors that supply explicit labels for both.

If this is right

Using only epistemic uncertainty for acquisition functions prevents over-exploration in high-noise areas during Bayesian optimization and active learning.
Decoupled PFNs achieve better performance than standard observation-level PFNs in hyperparameter optimization tasks.
In synthetic Bayesian optimization benchmarks, the decoupled approach obtains the highest average rank among compared models.
The convolution of latent and noise distributions provides a principled way to recover the full posterior predictive from the decomposed components.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

This decomposition technique could be adapted to other amortized inference models beyond PFNs to improve uncertainty-aware decision making.
Testing the transfer on real datasets with heteroscedastic noise would reveal how well synthetic training generalizes when ground-truth signals are unavailable.
If successful, it might reduce the sample complexity needed for effective exploration in noisy environments.

Load-bearing premise

The epistemic-aleatoric decomposition learned on synthetic tasks with explicit labels will transfer to real tasks where only noisy observations are available and the true latent signal cannot be observed.

What would settle it

Training a decoupled PFN on synthetic data and then evaluating whether its epistemic uncertainty estimates lead to superior Bayesian optimization performance compared to total variance on a set of real-world noisy functions with known ground-truth optima.

Figures

Figures reproduced from arXiv: 2605.06413 by Jos\'e Miguel Hern\'andez-Lobato, Richard Bergna, Stefan Depeweg.

**Figure 1.** Figure 1: Why decoupling matters for acquisition. In a heteroscedastic 1D task with an unsupported region (grey), total predictive uncertainty can draw a standard PFN toward high-noise observations. By separating latent-signal uncertainty from observation noise, our decoupled PFN enables epistemic-LCB to target the unsupported region instead. Stars mark selected queries. the epistemic–aleatoric split is not identif… view at source ↗

**Figure 2.** Figure 2: Average-rank summary across sequential optimisation benchmarks. view at source ↗

**Figure 3.** Figure 3: Simple regret over 100 BO steps on all 8 core HPO benchmarks, mean ± SE over 10 seeds. Dec-ICL variants (red family, solid) consistently place among the top methods across LGBM and XGBoost tasks. The RF/Diabetes benchmark is visibly noisier due to its small dataset (442 examples) and stochastic train/val splits; most methods converge to similar final regret, making average rank a more reliable summary tha… view at source ↗

**Figure 4.** Figure 4: Full HPO summary including all acquisition variants. Methods are sorted by average rank view at source ↗

**Figure 5.** Figure 5: Simple regret over 100 BO steps on the 5 synthetic benchmarks, mean ± SE over 10 seeds. LogEI variants (thicker solid lines) converge faster on all smooth benchmarks view at source ↗

**Figure 6.** Figure 6: Full synthetic BO summary including all acquisition variants, sorted by average rank. view at source ↗

**Figure 7.** Figure 7: Test RMSE as a function of the number of labelled points for the real-data active-learning view at source ↗

read the original abstract

Prior-Fitted Networks (PFNs) amortize Bayesian prediction by meta-learning over a synthetic task prior, but their standard output is a posterior predictive distribution over noisy observations. For sequential decision-making, such as active learning and Bayesian optimization, acquisition should prioritize epistemic uncertainty about the latent signal rather than irreducible aleatoric observation noise. We show that this epistemic--aleatoric split is not identifiable in general from the posterior predictive distribution alone, even when that distribution is known exactly. We then exploit a distinctive advantage of PFNs: because the synthetic data-generating process is under our control, each task can contain an explicit latent signal and noise function, and the generator can provide query-level labels for both the noiseless target and the observation-noise variance. We use these labels to train a decoupled PFN with separate latent-signal and aleatoric heads. The observation-level predictive is induced by convolving the latent signal distribution with the learned noise model. Empirically, epistemic-only acquisition mitigates the failure mode of total-variance exploration in noisy and heteroscedastic settings. In matched comparisons, decoupled models usually improve over tuned observation-level baselines, with the clearest gains in HPO; in broader sweeps, a decoupled model obtains the best average rank in both HPO and synthetic BO.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper uses explicit synthetic labels to train separate epistemic and aleatoric heads in PFNs, which cleanly addresses the non-identifiability of the split from the posterior predictive alone.

read the letter

The main advance is recognizing that the usual PFN posterior predictive mixes epistemic and aleatoric uncertainty in a way that cannot be disentangled from data alone, then exploiting the fact that the synthetic generator is fully controlled to supply per-query labels for the noiseless target and the noise variance. They train two heads, convolve the outputs for the final predictive, and use the epistemic head for acquisition. This is a direct, practical move that fits the PFN meta-training loop without extra machinery. The reported gains in HPO and synthetic BO align with the motivation: total-variance acquisition wastes effort on noise in heteroscedastic cases, while the decoupled version focuses on the latent signal. The identifiability observation itself is straightforward once stated and seems to hold up on the terms given. The soft spot is the transfer step. The heads are optimized on synthetic tasks where both components are observable; nothing in the setup guarantees that the learned mapping will separate signal from noise correctly on real data whose observation process may differ from the generator. Without ablations on prior structure or checks against mismatched real tasks, the empirical improvements could partly reflect prior match rather than a robust decomposition. The abstract gives no statistical tests or baseline-tuning details, so the size of the advantage is still hard to judge. This is for groups already working with PFNs in sequential decision settings. It deserves peer review because the engineering step is concrete, the problem is well-motivated, and the synthetic-label trick is a genuine PFN-specific lever even if more validation on transfer is needed.

Referee Report

3 major / 1 minor

Summary. The paper claims that epistemic-aleatoric uncertainty decomposition is not identifiable from the posterior predictive p(y|x) alone, even when known exactly. It exploits control over PFN synthetic data generators to supply per-query labels for both the noiseless latent signal and observation noise variance, training a decoupled model with separate heads whose outputs are convolved to recover the observation-level predictive. Empirically, epistemic-only acquisition improves over tuned baselines in hyperparameter optimization and synthetic Bayesian optimization, with clearest gains in HPO and best average rank in broader sweeps.

Significance. If the claims hold, the work supplies a practical route to identifiable epistemic uncertainty in amortized Bayesian predictors by leveraging the controllable synthetic prior that defines PFNs. This directly addresses a known limitation for acquisition functions in noisy or heteroscedastic sequential decision tasks. The explicit use of generator-provided labels for both signal and noise is a genuine strength that turns the synthetic nature of PFN training into an asset rather than a liability.

major comments (3)

[Abstract] Abstract and theoretical motivation: the central non-identifiability claim (that epistemic and aleatoric components cannot be recovered from p(y|x) even when the distribution is known exactly) is asserted without a derivation, counter-example construction, or formal statement of the conditions under which the result holds. This is load-bearing for the motivation of the decoupled architecture.
[Experiments] Experiments section: no statistical significance tests, no ablation on the structure or richness of the synthetic prior family, and no information on how observation-level baselines were tuned are provided. Without these, it is impossible to determine whether reported gains in HPO and BO are attributable to the epistemic-aleatoric split or to prior matching and hyperparameter advantages.
[Method] Method and transfer discussion: the assumption that a decomposition learned on synthetic tasks with explicit latent-signal labels will produce usable epistemic estimates on real tasks (where only noisy observations are available) is not accompanied by any theoretical guarantee or domain-shift experiment. This is the weakest link for the applicability claim.

minor comments (1)

The convolution step that induces the observation-level predictive from the two heads would benefit from an explicit equation or small diagram to clarify the distributional operation being performed.

Simulated Author's Rebuttal

3 responses · 1 unresolved

We thank the referee for the constructive review and for acknowledging the significance of using controllable synthetic priors to achieve identifiable epistemic-aleatoric decomposition. We address each major comment below, indicating planned revisions to strengthen the manuscript.

read point-by-point responses

Referee: [Abstract] Abstract and theoretical motivation: the central non-identifiability claim (that epistemic and aleatoric components cannot be recovered from p(y|x) even when the distribution is known exactly) is asserted without a derivation, counter-example construction, or formal statement of the conditions under which the result holds. This is load-bearing for the motivation of the decoupled architecture.

Authors: We agree that a more formal treatment would strengthen the motivation. The current manuscript motivates the claim by noting that the observation predictive is the convolution of latent signal and noise distributions, allowing multiple decompositions to produce the same marginal. In revision we will add a short proposition with proof sketch under standard assumptions (e.g., additive Gaussian noise with unknown variance) together with an explicit counter-example showing two distinct (epistemic, aleatoric) pairs that induce identical p(y|x). revision: yes
Referee: [Experiments] Experiments section: no statistical significance tests, no ablation on the structure or richness of the synthetic prior family, and no information on how observation-level baselines were tuned are provided. Without these, it is impossible to determine whether reported gains in HPO and BO are attributable to the epistemic-aleatoric split or to prior matching and hyperparameter advantages.

Authors: These omissions limit interpretability and we will correct them. We will report statistical significance via Wilcoxon signed-rank tests across repeated seeds for all main comparisons. An ablation varying synthetic prior richness (fixed vs. heteroscedastic noise families, different signal smoothness) will be added. We will also expand the experimental details to specify the hyperparameter search ranges, validation protocol, and selection criterion used for tuning the observation-level baseline PFNs, ensuring the comparison isolates the effect of the decoupled heads. revision: yes
Referee: [Method] Method and transfer discussion: the assumption that a decomposition learned on synthetic tasks with explicit latent-signal labels will produce usable epistemic estimates on real tasks (where only noisy observations are available) is not accompanied by any theoretical guarantee or domain-shift experiment. This is the weakest link for the applicability claim.

Authors: We accept that the transfer step is the most empirically grounded part of the argument. The manuscript demonstrates gains on real HPO and BO tasks, but does not claim a general theoretical guarantee, which would require unverifiable assumptions on synthetic-to-real prior alignment. We will add a controlled domain-shift experiment (train on one synthetic noise/signal family, evaluate on a deliberately mismatched family) to quantify robustness. A universal theoretical guarantee for arbitrary domain shifts lies outside the scope of the present work. revision: partial

standing simulated objections not resolved

A general theoretical guarantee for transfer of the learned decomposition to arbitrary real-world tasks under domain shift.

Circularity Check

0 steps flagged

No significant circularity; derivation relies on external synthetic labels and empirical validation

full rationale

The paper establishes non-identifiability of the epistemic-aleatoric split from p(y|x) alone as a general result, then uses explicit control over the synthetic generator to supply per-query labels for the latent signal and noise variance during meta-training. Separate heads are trained directly on these generator-provided labels, with the observation predictive formed by convolution; this does not reduce to a self-definition or fitted parameter renamed as prediction. No load-bearing self-citations appear, and results are validated against tuned baselines on HPO and BO tasks rather than being forced by the training objective. The transfer assumption to real data is explicit and externally testable, keeping the chain self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on the ability to generate synthetic tasks with query-level ground-truth labels for both the latent signal and the noise variance; these labels are treated as given by the data generator rather than learned or derived.

axioms (2)

domain assumption The synthetic data-generating process can be designed to expose separate, query-level labels for the noiseless target and the observation-noise variance.
Invoked to justify training the two separate heads; without this the decoupled supervision disappears.
domain assumption The decomposition learned on synthetic tasks transfers to real tasks that only provide noisy observations.
Required for the empirical claim that decoupled acquisition improves performance on HPO and BO benchmarks.

pith-pipeline@v0.9.0 · 5543 in / 1507 out tokens · 43745 ms · 2026-05-08T04:51:53.821804+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

23 extracted references · 6 canonical work pages · 1 internal anchor

[1]

Neural computation , volume=

Information-based objective functions for active data selection , author=. Neural computation , volume=. 1992 , publisher=

1992
[2]

Bayesian active learning for classiﬁcation and preferenc e learning,

Bayesian active learning for classification and preference learning , author=. arXiv preprint arXiv:1112.5745 , year=

work page arXiv
[3]

Journal of Global optimization , volume=

Efficient global optimization of expensive black-box functions , author=. Journal of Global optimization , volume=. 1998 , publisher=

1998
[4]

Gaussian process optimization in the bandit setting: No regret and experimental design,

Gaussian process optimization in the bandit setting: No regret and experimental design , author=. arXiv preprint arXiv:0912.3995 , year=

work page arXiv
[5]

Advances in neural information processing systems , volume=

What uncertainties do we need in bayesian deep learning for computer vision? , author=. Advances in neural information processing systems , volume=
[6]

International conference on machine learning , pages=

Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning , author=. International conference on machine learning , pages=. 2018 , organization=

2018
[7]

international conference on machine learning , pages=

Dropout as a bayesian approximation: Representing model uncertainty in deep learning , author=. international conference on machine learning , pages=. 2016 , organization=

2016
[8]

Transformers can do bayesian inference, 2024

Transformers can do bayesian inference , author=. arXiv preprint arXiv:2112.10510 , year=

work page arXiv
[9]

International Conference on Machine Learning , pages=

Statistical foundations of prior-data fitted networks , author=. International Conference on Machine Learning , pages=. 2023 , organization=

2023
[10]

TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second

Tabpfn: A transformer that solves small tabular classification problems in a second , author=. arXiv preprint arXiv:2207.01848 , year=

work page internal anchor Pith review arXiv
[11]

Nature , volume=

Accurate predictions on small data with a tabular foundation model , author=. Nature , volume=. 2025 , publisher=

2025
[12]

TabICL: A tabular foundation model for in-context learning on large data.arXiv preprint arXiv:2502.05564, 2025

Tabicl: A tabular foundation model for in-context learning on large data , author=. arXiv preprint arXiv:2502.05564 , year=

work page arXiv
[13]

International Conference on Machine Learning , pages=

Pfns4bo: In-context learning for bayesian optimization , author=. International Conference on Machine Learning , pages=. 2023 , organization=

2023
[14]

arXiv preprint arXiv:2404.16795 , year=

In-context freeze-thaw bayesian optimization for hyperparameter optimization , author=. arXiv preprint arXiv:2404.16795 , year=

work page arXiv
[15]

Advances in neural information processing systems , volume=

Simple and scalable predictive uncertainty estimation using deep ensembles , author=. Advances in neural information processing systems , volume=
[16]

Advances in neural information processing systems , volume=

Regression with input-dependent noise: A Gaussian process treatment , author=. Advances in neural information processing systems , volume=
[17]

Proceedings of the 24th international conference on Machine learning , pages=

Most likely heteroscedastic Gaussian process regression , author=. Proceedings of the 24th international conference on Machine learning , pages=
[18]

Journal of Computational and Graphical Statistics , volume=

Practical heteroscedastic Gaussian process modeling for large simulation experiments , author=. Journal of Computational and Graphical Statistics , volume=. 2018 , publisher=

2018
[19]

Journal of Artificial Intelligence Research , volume=

Hebo: Pushing the limits of sample-efficient hyper-parameter optimisation , author=. Journal of Artificial Intelligence Research , volume=
[20]

Advances in neural information processing systems , volume=

Algorithms for hyper-parameter optimization , author=. Advances in neural information processing systems , volume=
[21]

International conference on machine learning , pages=

Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures , author=. International conference on machine learning , pages=. 2013 , organization=

2013
[22]

Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages=

Optuna: A next-generation hyperparameter optimization framework , author=. Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages=
[23]

Advances in neural information processing systems , volume=

BoTorch: A framework for efficient Monte-Carlo Bayesian optimization , author=. Advances in neural information processing systems , volume=