REVIEW 3 major objections 5 minor 3 cited by
The paper proves a predictive central limit theorem under quasi-martingale conditions and uses it to decompose TabPFN's total predictive uncertainty into aleatoric and epistemic components with credible bands that meet frequentist coverage.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 04:31 UTC pith:BMGSGH7W
load-bearing objection A genuinely new supervised predictive CLT under quasi-martingale conditions, honestly applied to TabPFN, but the theorem's sufficient conditions are not met by the target model and the coverage evidence is mixed. the 3 major comments →
Uncertainty Decomposition for Bayes-Filtered Transformers via Bayesian Predictive Inference
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Theorem 4.3 is the central claim: under the quasi-martingale condition (5) and the stronger √n-weighted variant (7), the posterior law of the limiting predictive distribution F̃(x,t) given Z_{1:n} is asymptotically N_m(0, V(x,t)). The covariance is consistently estimated by V_n = (1/n)Σ k² Δ_k Δ_kᵀ, computed from the one-step predictive updates along the observed context. Since TabPFN is, in an idealized limit, a Bayesian in-context learner, this variance serves as epistemic uncertainty. The authors report near-nominal frequentist coverage across seven DGPs and an entropy-based decomposition (via a moment-matched Beta/Dirichlet approximation) that places aleatoric uncertainty near class boun
What carries the argument
The central object is the quasi-martingale predictive rule: a sequence of one-step-ahead predictive distributions P_k(x,A) whose expected absolute drift is summable. The load-bearing identity is the predictive CLT (Theorem 4.3), which makes the posterior of the limiting predictive distribution asymptotically Gaussian; the workhorse estimator is V_n = (1/n)Σ_{k=1}^n k² Δ_k Δ_kᵀ, the inflated outer product of prediction updates along the context. It requires only n forward passes and no simulation of future covariates. A moment-matched Beta (or Dirichlet) approximation converts the Gaussian law into an expected-entropy estimate of aleatoric uncertainty.
Load-bearing premise
The theorem's formal validity rests on the √n-weighted quasi-martingale condition (7), and the paper's own diagnostics for TabPFN fit drift exponents of about 0.73–0.87, below the 1.5 threshold the condition requires—so if those estimates are accurate, the CLT is not formally guaranteed for the exact model.
What would settle it
Under the paper's discrete-covariate logistic DGP, run TabPFN rollouts with larger ensembles (n_estimators≥64) and fit the drift exponent β for E|b_n|; if β is consistently below 1.5 (or below 1), the quasi-martingale condition underlying Theorem 4.3 fails for TabPFN, and the decomposition's validity reduces to the empirical coverage alone.
If this is right
- The predictive CLT extends Bayesian predictive inference to supervised settings under quasi-martingale conditions, covering predictive rules that are neither exchangeable nor exact martingales.
- For TabPFN, epistemic uncertainty bands are computable in-context—n forward passes on expanding prefixes—avoiding the need for a generative model of covariates or ensembles of rollouts.
- The asymptotic credible intervals and sup-t bands achieve near-nominal frequentist coverage at moderate context lengths across linear, dependent, Poisson, probit, and categorical DGPs.
- The entropy-based decomposition is numerically stable near p=0 or 1 and generalizes to multiclass problems via a Dirichlet moment match.
- Because the CLT only requires a compact response space and a fixed event/covariate pair, the framework applies beyond TabPFN to other supervised black-box predictive rules.
Where Pith is reading between the lines
- The size of V_n/n could serve as an in-context diagnostic of how far a given dataset is from the 'idealized' Bayesian limit: large persistent epistemic variance flags contexts where the predictive rule is still fluctuating and where extra data would shift predictions.
- Because V_n depends on row order while TabPFN's prediction itself is permutation-invariant, the estimator captures a trajectory-dependent quantity; comparing V_n across random permutations could yield a permutation-based sensitivity measure for epistemic uncertainty.
- If the quasi-martingale conditions fail for TabPFN (as the paper's power-law diagnostics hint), a natural next step is a CLT with a slower-than-√n rate; such a result would also sharpen the two estimators and explain the small-n coverage deficits visible in the Dependent and Categorical DGPs.
- The credible bands target the limiting predictive distribution F̃ rather than the true conditional law; the near-nominal coverage suggests the two coincide in the tested regimes, but applications to misspecified or non-stationary data would be the natural stress test.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper develops a Bayesian predictive inference (BPI) framework for uncertainty decomposition in TabPFN, a supervised Bayes-filtered transformer. The main theoretical contribution is a predictive central limit theorem (Theorem 4.3) for supervised settings under quasi-martingale conditions. The theorem states that, conditionally on an observed context, the limiting predictive distribution is asymptotically Gaussian with covariance determined by the volatility of predictive updates; the normalized update covariance V_n/n is then interpreted as epistemic uncertainty. The authors use this to build pointwise and simultaneous credible bands for TabPFN's limiting predictive probabilities/CDFs, and propose an entropy-based aleatoric/epistemic decomposition. The method is validated on seven synthetic DGPs and illustrated on two real datasets.
Significance. If the theoretical bridge to TabPFN were secure, this would be a genuinely useful contribution: it is one of the first attempts to give a principled, black-box epistemic/aleatoric decomposition for a foundation model that never explicitly represents a posterior. The paper is honest about the gap between its sufficient conditions and practical verification, and the appendix proofs are detailed. It also ships code and includes an external coverage benchmark. However, the central application-level claim is not formally supported: the paper's own diagnostics in Appendix B indicate that TabPFN violates the CLT's sufficient condition (7), and several simultaneous-band coverage rates in Table 1 are far from nominal. The theoretical CLT is valuable in its own right, but the advertised TabPFN uncertainty decomposition currently rests on an empirical heuristic rather than on Theorem 4.3.
major comments (3)
- [§4.3 and Appendix B.5] Theorem 4.3 requires condition (7): Σ √n E|E[Δ_n|Z_{1:n-1}]| < ∞. Under a power-law model E|b_n| ≈ C n^{-β}, this requires β > 1.5. Appendix B.5 reports β̂ ≈ 0.87 (95% CI [0.77,0.98]) for n_estimators=8 and β̂ ≈ 0.73 ([0.67,0.80]) for n_estimators=16. The authors explicitly state they do not see behaviour consistent with β > 1.5. The unweighted quasi-martingale condition (5) would require β > 1, which is also not supported by the point estimates. Thus the Gaussian posterior approximation and the interpretation of V_n/n as epistemic variance are not formally justified for TabPFN. The paper's caveat that (7) is sufficient, not necessary, is correct but shifts the burden onto empirical validation. That validation is partial: coverage of f_0 is a frequentist property and can hold even when V_n/n is not a correct posterior variance. Please either develop a theorem with weaker, verifiable cond
- [§6, Table 1] The abstract and Section 3.2 claim that the credible bands achieve near-nominal frequentist coverage as context length grows. Table 1 does not support this uniformly. For the Categorical DGP, simultaneous V_n bands have coverage 0.53 at n=200, 0.53 at n=500, and 0.87 at n=1000; simultaneous U_n bands at n=1000 are 0.80. With R=100 replicates, the Monte Carlo standard error of a simultaneous coverage rate is about 0.05, so 0.87 and 0.80 are substantially below the nominal 0.95. The paper acknowledges deviations but the abstract's wording is too strong. Moreover, coverage of the true f_0 does not directly validate the epistemic variance: conservative intervals can overcover even if V_n/n overstates or misstates the posterior variance. A more direct check of the epistemic variance (e.g., comparing V_n/n to a known posterior variance in a simple Bayesian model) would be needed to support the
- [§7 and Appendix C] The entropy-based decomposition is presented as a contribution, but the moment-matched Beta/Dirichlet approximation is not derived from the predictive CLT and has no stated error bound. In the multiclass extension, the Dirichlet parameters are set by matching only the sum of variances across classes (Eq. 16), not the individual class variances, so the estimated aleatoric entropy may be inaccurate when class variances differ. Since this decomposition is one of the three advertised contributions, it needs either a theoretical justification (e.g., an error bound or a proof that the approximation is asymptotically equivalent to the CLT-based delta method) or at least a numerical sanity check against a known posterior.
minor comments (5)
- [§4.2, Theorem 4.3] Condition (iii) assumes V(x,t) is positive definite. For atomic events, the limiting variance can be zero or singular; positive semidefinite would be the more natural assumption, with the Gaussian statement interpreted in the possibly degenerate sense.
- [§H.3] In the proof of Theorem 4.3, the step from R_n(x,t) to n Σ_{k≥n}(M_k−M_{k−1})^2 silently drops the k=n term. This is harmless only if nΔ_nΔ_n^T → 0, which follows from convergence of R_n; please make this explicit.
- [§3.2] The row-permutation procedure is described briefly. Since V_n depends on the trajectory, the choice of random permutation is an additional stochastic ingredient. For i.i.d. data this is justified, but a formal statement about the permutation's effect on the CLT conditions would be helpful, especially for non-exchangeable real-data settings.
- [§6, Table 1] The column headers are misaligned: 'Vn Point.', 'Un Point.', 'Vn Simul.', 'Un Simul.' should be associated with their Rate/Width columns. Also, 'n estimators' and 'n_estimators' are used inconsistently across the paper.
- [Appendix B] The diagnostics are run with n_estimators=8 and 16, while the rest of the paper uses n_estimators=64. The relation between ensemble size and the fitted exponents β̂ should be discussed; the observed decrease from 0.87 to 0.73 as the ensemble size increases is itself informative about the stability of the diagnostic.
Circularity Check
No significant circularity: the predictive CLT is proved from explicit quasi-martingale conditions, the variance estimator is derived in the theorem, and coverage is checked against external DGPs.
full rationale
The paper's central claim—the predictive CLT (Theorem 4.3) and the use of V_n/n as a Gaussian posterior variance for the limiting predictive F~—is derived in Appendix H from explicit sufficient conditions (5), (7), and (ii)-(iii) of Theorem 4.3. The variance estimator V_n = (1/n) Σ k² Δ_k Δ_kᵀ is not assumed to equal the posterior variance; Theorem 4.4 proves convergence of V_n to the limiting covariance V under stated conditions, and Theorem 4.5 then yields the Gaussian approximation. The proof relies on an adapted external CLT (Berti et al., 2011), not on an unverified self-citation. The BPI references (Fortini & Petrone) frame the definitions of aleatoric/epistemic uncertainty, but the central theorem is proved in the paper itself. Empirical coverage is evaluated against the true data-generating distribution f0 (Appendix F), an external benchmark, rather than against quantities fitted by the paper. The main caveat—Appendix B finds fitted drift exponents β̂ ≈ 0.73–0.87, below the β > 1.5 sufficient for condition (7)—is an assumption-validation gap or correctness risk, not circularity: the paper explicitly labels (7) as sufficient rather than necessary, and the theorem's logic does not define its conclusion into its assumptions. Therefore no load-bearing step reduces to its own inputs.
Axiom & Free-Parameter Ledger
free parameters (3)
- n_estimators =
64
- softmax temperature =
1.0
- variance clipping in Beta/Dirichlet moment matching
axioms (5)
- domain assumption TabPFN, in an idealised limit, is a Bayesian posterior predictive rule because it minimizes population log-loss on Bayes-filtered data.
- domain assumption The TabPFN predictive rule P_k(x,A) satisfies the quasi-martingale condition (5) and the stronger √n-weighted condition (7).
- domain assumption Condition (iv): for every n, X_{n+1} is conditionally independent of Y_{1:n} given X_{1:n}.
- domain assumption The response space Y is compact.
- standard math Berti et al. (2011) Proposition 1, as adapted in Theorem G.4, is valid without the assumption M_n = E[Y_{n+1}|G_n].
read the original abstract
Bayes-filtered transformers are transformers meta-learned on sequences from a prior predictive distribution to approximate the corresponding posterior predictive distribution. They output total predictive uncertainty in a single forward pass but never explicitly represent a posterior distribution, making the standard route to separating aleatoric from epistemic uncertainty unavailable. We address this challenge through the lens of Bayesian predictive inference (BPI). Our main result is a predictive Central Limit Theorem (CLT) for supervised settings under conditions that are among the weakest known in the BPI literature. The CLT characterises the posterior of the limiting predictive distribution given an observed context as asymptotically Gaussian; the variance of this Gaussian quantifies epistemic uncertainty. We apply the framework to TabPFN, a Bayes-filtered transformer that is a state-of-the-art foundation model for tabular prediction. The resulting credible bands achieve near-nominal frequentist coverage as context length grows, and the decomposition largely matches standard desiderata: epistemic uncertainty shrinks with context length and is highest in sparsely observed regions within the span of the context data, while aleatoric uncertainty dominates near decision boundaries where classes overlap.
Figures
Forward citations
Cited by 3 Pith papers
-
Elicitation without Backpropagation: Steering Model Behavior by Optimizing the Latent Posterior
Posterior Prefix Tuning steers a Bayes-filtered transformer's latent posterior by optimizing a tilted surrogate with prior samples, eliciting high-utility continuations without backpropagating through the transformer.
-
What does a Bayes-filtered transformer believe? A predictive Monte Carlo approach
Predictive Monte Carlo recovers a Bayes-filtered transformer's implicit prior and posterior over latent tasks from next-token rollouts alone; the task-diversity threshold and transient generalization appear in this re...
-
Concentration and Calibration in Predictive Bayesian Inference
Predictive Bayesian inference posteriors concentrate onto a forward-model-dependent quantity and produce miscalibrated credible sets unless the predictive model contains the true data-generating process.
Reference graph
Works this paper leans on
-
[1]
and Hu, J
Albert, J. and Hu, J. Probability and Bayesian Modeling . Texts in Statistical Science . Chapman and Hall/CRC , first edition, 2020
2020
-
[2]
Battiston, M. and Cappello, L. Bayesian predictive inference beyond martingales. arXiv:2507.21874, 2025
arXiv 2025
-
[3]
A central limit theorem and its applications to multicolor randomly reinforced urns
Berti, P., Crimaldi, I., Pratelli, L., and Rigo, P. A central limit theorem and its applications to multicolor randomly reinforced urns. Journal of Applied Probability, 48 0 (2): 0 527--546, 2011
2011
-
[4]
Fong, E. and Yiu, A. Asymptotics for parametric martingale posteriors. arXiv:2410.17692, 2024 a
arXiv 2024
-
[5]
Fong, E. and Yiu, A. Bayesian quantile estimation and regression with martingale posteriors. arXiv:2406.03358, 2024 b
arXiv 2024
-
[6]
Fong, E., Holmes, C., and Walker, S. G. Martingale posterior distributions. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 85 0 (5): 0 1357--1391, 2023
2023
-
[7]
and Petrone, S
Fortini, S. and Petrone, S. Quasi- Bayes properties of a procedure for sequential learning in mixture models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 82 0 (4): 0 1087--1114, 2020
2020
-
[8]
and Petrone, S
Fortini, S. and Petrone, S. Prediction-based uncertainty quantification for exchangeable sequences. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 381 0 (2247), 2023
2023
-
[9]
and Petrone, S
Fortini, S. and Petrone, S. Exchangeability, prediction and predictive modeling in Bayesian statistics. Statistical Science, 40 0 (1): 0 40--67, 2025
2025
-
[10]
On a notion of partially conditionally identically distributed sequences
Fortini, S., Petrone, S., and Sporysheva, P. On a notion of partially conditionally identically distributed sequences. Stochastic Processes and their Applications, 128 0 (3): 0 819--846, 2018
2018
-
[11]
K., Catt, E., Dutordoir, V., Grau-Moya , J., Orseau, L., Hutter, M., and Veness, J
Genewein, T., Deletang, G., Ruoss, A., Wenliang, L. K., Catt, E., Dutordoir, V., Grau-Moya , J., Orseau, L., Hutter, M., and Veness, J. Memory-based meta-learning on non-stationary distributions. In Proceedings of the International Conference on Machine Learning , pp.\ 11173--11195, 2023
2023
-
[12]
K., Mattern, C., Aitchison, M., and Veness, J
Grau-Moya , J., Genewein, T., Hutter, M., Orseau, L., Deletang, G., Catt, E., Ruoss, A., Wenliang, L. K., Mattern, C., Aitchison, M., and Veness, J. Learning universal predictors. In Proceedings of the International Conference on Machine Learning , pp.\ 16178--16205, 2024
2024
-
[13]
S., Wilson, A
Hamada, M. S., Wilson, A. G., Reese, C. S., and Martz, H. F. Bayesian Reliability . Springer Series in Statistics . Springer, 2008
2008
-
[14]
u ller, S., Purucker, L., Krishnakumar, A., K \
Hollmann, N., M \"u ller, S., Purucker, L., Krishnakumar, A., K \"o rfer, M., Hoo, S. B., Schirrmeister, R. T., and Hutter, F. Accurate predictions on small data with a tabular foundation model. Nature, 637 0 (8045): 0 319--326, 2025
2025
-
[15]
S., Si, J., Valdettaro, F., Chen, W., Faisal, A
Jayasekera, I. S., Si, J., Valdettaro, F., Chen, W., Faisal, A. A., and Li, Y. Variational uncertainty decomposition for in-context learning. In Advances in Neural Information Processing Systems , 2025
2025
-
[16]
Foundations of Modern Probability , volume 99 of Probability Theory and Stochastic Modelling
Kallenberg, O. Foundations of Modern Probability , volume 99 of Probability Theory and Stochastic Modelling . Springer International Publishing, 2021
2021
-
[17]
Meta-trained agents implement Bayes-optimal agents
Mikulik, V., Del \'e tang, G., McGrath, T., Genewein, T., Martic, M., Legg, S., and Ortega, P. Meta-trained agents implement Bayes-optimal agents. In Advances in Neural Information Processing Systems , volume 33, pp.\ 18691--18703, 2020
2020
-
[18]
Nagler, T. and R \"u gamer, D. Uncertainty quantification for prior-data fitted networks using martingale posteriors. arXiv:2505.11325, 2025
Pith/arXiv arXiv 2025
-
[19]
Ng, K., Fong, E., Frazier, D. T., Knoblauch, J., and Wei, S. TabMGP : Martingale posterior with TabPFN . arXiv:2510.25154, 2025
Pith/arXiv arXiv 2025
-
[20]
Olea, J. L. M. and Plagborg-M ller, M. Simultaneous confidence bands: Theory , implementation, and an application to SVARs . Journal of Applied Econometrics, 34 0 (1): 0 1--17, 2019
2019
-
[21]
Ortega, P. A., Wang, J. X., Rowland, M., Genewein, T., Kurth-Nelson , Z., Pascanu, R., Heess, N., Veness, J., Pritzel, A., Sprechmann, P., Jayakumar, S. M., McGrath, T., Miller, K., Azar, M., Osband, I., Rabinowitz, N., Gy \"o rgy, A., Chiappa, S., Osindero, S., Teh, Y. W., van Hasselt, H., de Freitas, N., Botvinick, M., and Legg, S. Meta-learning of Sequ...
Pith/arXiv arXiv 1905
-
[22]
TabPFN documentation: Intended use
PriorLabs . TabPFN documentation: Intended use. https://priorlabs.ai/getting\_started/intended\_use/\#computational-and-time-requirements, 2025
2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.