REVIEW 27 cited by
Transformers Can Do Bayesian Inference
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Currently, it is hard to reap the benefits of deep learning for Bayesian methods, which allow the explicit specification of prior knowledge and accurately capture model uncertainty. We present Prior-Data Fitted Networks (PFNs). PFNs leverage in-context learning in large-scale machine learning techniques to approximate a large set of posteriors. The only requirement for PFNs to work is the ability to sample from a prior distribution over supervised learning tasks (or functions). Our method restates the objective of posterior approximation as a supervised classification problem with a set-valued input: it repeatedly draws a task (or function) from the prior, draws a set of data points and their labels from it, masks one of the labels and learns to make probabilistic predictions for it based on the set-valued input of the rest of the data points. Presented with a set of samples from a new supervised learning task as input, PFNs make probabilistic predictions for arbitrary other data points in a single forward propagation, having learned to approximate Bayesian inference. We demonstrate that PFNs can near-perfectly mimic Gaussian processes and also enable efficient Bayesian inference for intractable problems, with over 200-fold speedups in multiple setups compared to current methods. We obtain strong results in very diverse areas such as Gaussian process regression, Bayesian neural networks, classification for small tabular data sets, and few-shot image classification, demonstrating the generality of PFNs. Code and trained PFNs are released at https://github.com/automl/TransformersCanDoBayesianInference.
Forward citations
Cited by 27 Pith papers
-
SkillTFM: Gated Skill Evolution for Training-Free Adaptation of Tabular Foundation Models
SkillTFM adapts tabular foundation models at deployment time by gating evidence-triggered repair skills, improving AUC by up to 0.142 on boundary-shifted tasks with zero observed harm.
-
TabPFN beyond Tabular Data: Calibration and Accuracy on Multimodal Embeddings
TabPFN as a zero-gradient head on frozen multimodal embeddings ranks best on NLL and ECE across 22 820 episodes while matching accuracy in mid-shot, mid-dimension regimes and also fixes miscalibration after fine-tuning.
-
TimEE: End-to-end Time Series Classification via In-Context Learning
A 4.5M-parameter transformer meta-trained on synthetic VARX-generated classification tasks achieves state-of-the-art ROC AUC on the UCR time series classification benchmark via in-context learning with no per-dataset ...
-
Foundation Models for Credit Risk Prediction: A Game Changer?
Tabular foundation models, used zero-shot, match or beat tuned gradient boosting on average in credit PD and LGD benchmarks, with a larger edge on small datasets.
-
Incremental Transformer Neural Processes
A causally masked, KV-cached Transformer Neural Process (incTNP) updates its context in linear time and matches or exceeds standard TNP accuracy on tabular and weather-stream benchmarks.
-
Learning-To-Measure: In-Context Active Feature Acquisition
One transformer, pretrained with sequence modeling, greedily acquires features to reduce prediction uncertainty across multiple tasks with missing data, matching or beating task-specific AFA baselines without per-task...
-
GraphPFN: A Prior-Data Fitted Graph Foundation Model
GraphPFN, a graph foundation model pretrained on synthetic graphs from a stochastic-block-model plus preferential-attachment prior with graph-aware structured causal models, outperforms existing graph foundation model...
-
Meta-learning ecological priors from large language models explains human learning and decision making
A meta-learned transformer trained on LLM-generated tasks (ERMI) outperforms classical cognitive models in predicting human choices across function learning, category learning, and decision making.
-
LLMs are Bayesian, In Expectation, Not in Realization
Transformers can be Bayes-competitive in prequential log loss even when their predictive distributions are not invariant to example order, provided the cumulative predictive KL to the Bayesian reference stays small.
-
Direct Regret Optimization in Bayesian Optimization
A decision transformer, trained offline on ROI-filtered, early-stopped GP-ensemble rollouts and refined by sparse real evaluations, is proposed as a non-myopic policy for Bayesian optimization.
-
Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
A Prior-data Fitted Network with a scaling-law-specific prior gives better point and uncertainty predictions for neural scaling law extrapolation than MCMC, BNSL, and LC-PFN baselines.
-
Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning
Different procedural pretraining tasks create complementary, transferable structures in a transformer's attention and MLP weights, and structures from different tasks can be combined into one initialization.
-
Amortized In-Context Bayesian Posterior Estimation
A benchmark of in-context Bayesian posterior estimators shows the reverse-KL objective with transformers and normalizing flows outperforms forward-KL neural posterior estimation on predictive and out-of-distribution tasks.
-
Zero-shot Meta-learning for Tabular Prediction Tasks with Adversarially Pre-trained Transformer
Adversarially pre-trained transformer (APT) matches top gradient-boosting models on 35 small tabular classification benchmarks and improves on TabPFN in regression, while handling datasets with any number of classes v...
-
Scalable In-Context Learning on Tabular Data via Retrieval-Augmented Large Language Models
A retrieval module that selects the most relevant training rows for each test row lets LLMs do tabular in-context learning on large training sets, with a fitted power-law improvement.
-
mPOLICE: Provable Enforcement of Multi-Region Affine Constraints in Deep Neural Networks
mPOLICE generalizes POLICE to enforce exact affine output constraints inside multiple disjoint convex regions of a ReLU network's input by giving each region its own activation pattern.
-
Task Vectors in In-Context Learning: Emergence, Formation, and Benefit
Small transformers naturally encode task information in specific layers under limited conditions; a new auxiliary loss places a strong task vector at a chosen layer and improves out-of-distribution robustness.
-
DOFEN: Deep Oblivious Forest ENsemble
DOFEN, a deep neural network that randomly assembles per-column soft conditions into relaxed oblivious decision trees and ensembles them, reaches state-of-the-art DNN performance on the Tabular Benchmark.
-
Physics-Informed Learning for Robust Acoustic Localization with Calibrated Uncertainty
A gated TabPFN corrector reduces tail errors in hyperbolic acoustic localization and adds GDOP-scaled conformal uncertainty, validated on frozen-lake field playback and forest simulation.
-
Entangled by Design: Spurious Intra-Variable Signal Routing in Tabular In-Context Learners
In-context learners route predictions through a spurious component inside a composite feature whenever that component correlates with the label, and the routing persists as context grows.
-
Procedural Pretraining: Warming Up Language Models with Abstract Data
A short warm-up on procedural data (brackets, sorting, sets) makes language models more accurate and more data-efficient on language, code, and informal math.
-
Universal pre-training by iterated random computation
Pre-training a transformer on data generated by randomly initialized LSTMs yields zero-shot in-context learning on several held-out datasets, with gains that improve with scale and faster finetuning.
-
Dynamic Estimation Loss Control in Variational Quantum Sensing via Online Conformal Inference
A dynamic variational quantum sensing method using online conformal inference controls the long-term estimation loss at a user-specified level while updating circuit and estimator parameters.
-
Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics
GoLU (x times e^{-e^{-x}}) matches or beats GELU and Swish on several benchmarks, but the gains often appear only after learning-rate tuning.
-
Evaluating Machine Learning Models for Post-Wildfire Debris-Flow Prediction
TabPFN and top tree-based models reach threat scores of about 0.62 to 0.64 for post-wildfire debris-flow prediction, with rainfall intensity and storm accumulation the most important features, and synthetic data augme...
-
From Rows to Yields: How Foundation Models for Tabular Data Simplify Crop Yield Prediction
TabPFN, a pretrained tabular transformer, achieves accuracy statistically comparable to tuned machine learning pipelines for sub-national crop yield forecasting in South Africa, with far less feature engineering and compute.
-
Clustering by Attention: Leveraging Prior Fitted Transformers for Data Partitioning
A pre-trained PFN transformer is prompted with a few labeled samples to cluster the rest of a dataset by attention, with theoretical and empirical claims that are not fully supported.
Discussion (0). Continue with ORCID to comment.