A new corpus of 108 mixed string-numeric tables shows that advanced tabular learners with basic string embeddings perform well on most real-world data, while large LLM encoders help on free-text heavy tables.
hub
Transformers Can Do Bayesian Inference
21 Pith papers cite this work. Polarity classification is still indexing.
21
Pith papers citing it
abstract
Currently, it is hard to reap the benefits of deep learning for Bayesian methods, which allow the explicit specification of prior knowledge and accurately capture model uncertainty. We present Prior-Data Fitted Networks (PFNs). PFNs leverage in-context learning in large-scale machine learning techniques to approximate a large set of posteriors. The only requirement for PFNs to work is the ability to sample from a prior distribution over supervised learning tasks (or functions). Our method restates the objective of posterior approximation as a supervised classification problem with a set-valued input: it repeatedly draws a task (or function) from the prior, draws a set of data points and their labels from it, masks one of the labels and learns to make probabilistic predictions for it based on the set-valued input of the rest of the data points. Presented with a set of samples from a new supervised learning task as input, PFNs make probabilistic predictions for arbitrary other data points in a single forward propagation, having learned to approximate Bayesian inference. We demonstrate that PFNs can near-perfectly mimic Gaussian processes and also enable efficient Bayesian inference for intractable problems, with over 200-fold speedups in multiple setups compared to current methods. We obtain strong results in very diverse areas such as Gaussian process regression, Bayesian neural networks, classification for small tabular data sets, and few-shot image classification, demonstrating the generality of PFNs. Code and trained PFNs are released at https://github.com/automl/TransformersCanDoBayesianInference.
hub tools
citation-role summary
background 2
method 1
citation-polarity summary
representative citing papers
Transformers performing in-context learning implicitly implement gradient descent, ridge regression, and least-squares predictors for linear models, with behavior shifting based on model depth, width, and data noise.
A new probing framework detects moderate parametric memorization signals in tabular in-context learning models under single-task fine-tuning, strongest on low-cardinality tasks, but signals largely disappear under realistic training.
SurvivalPFN amortizes Bayesian survival analysis for right-censored data by pretraining a prior-data fitted network on synthetic identifiable DGPs and then performing in-context inference, achieving competitive results on 61 real datasets.
The paper proposes Retrieval Augmented Forecasting (RAF) that augments time-series foundation models with retrieved similar series to improve forecasting accuracy across domains.
A 4.5M-parameter transformer meta-trained on synthetic VARX-generated classification tasks achieves state-of-the-art ROC AUC on the UCR time series classification benchmark via in-context learning with no per-dataset training.
POLAR uses pretrained predictive foundation models as fixed belief-state encoders and trains only a lightweight policy head on top for amortised Bayesian experimental design, optimisation, and active learning.
CURE is a context management policy using entropy-gated admission and redundancy-aware eviction that delivers up to 27% relative improvement over classical stream learners on seven streams while remaining robust across TFM backbones.
Complementing tabular foundation model pretraining with LSBO-specific synthetic tasks and a regularizer yields strong performance on held-out molecular optimization benchmarks.
Tabular foundation models, used zero-shot, match or beat tuned gradient boosting on average in credit PD and LGD benchmarks, with a larger edge on small datasets.
ADKO is a decentralized framework where agents share compact GP-derived tokens and LM insights to achieve collaborative Bayesian optimization with a decomposed regret bound that includes compression and approximation losses.
Pre-trained TabPFN acts as an effective training-free summary network for neural posterior estimation, matching or outperforming standard methods while preserving useful marginal and location information in the posterior.
Decoupled PFNs use controllable synthetic priors to train separate latent-signal and noise heads, making epistemic-aleatoric decomposition identifiable and improving acquisition in noisy settings.
Tabular foundation models achieve high accuracy in molecular property prediction through in-context learning, with up to 100% win rates on MoleculeACE tasks when paired with CheMeleon embeddings.
SGNNs pretrain neural networks on synthetic corpora from multiple mechanistic models and noise levels to enable robust forecasting and back-to-simulation attribution across epidemiology, ecology, and other fields.
Amortized transformer model with conditional fixed-point iterations learns SCM causal mechanisms from data and graphs, matching per-dataset baselines and outperforming in low-data regimes.
Tabular foundation models for gut microbiome data show degraded performance under perturbations like zero-imputation and sparsification even when discriminative taxa are retained, with zero-imputation most harmful and greater sensitivity than random forests.
A Graph Attention Network pretrained solely on synthetic MDPs solves held-out tabular RL benchmarks in context, outperforming UCB-VI and Q-learning online while matching VI-LCB offline.
CoMET achieves strong multimodal classification performance by composing frozen modality encoders, PCA compression, and tabular foundation models without any training, reaching state-of-the-art on diverse benchmarks including large-scale hierarchical tasks.
Preprocessing-optimized TabPFN achieves top average rank in regression on 66 NIR datasets and remains competitive on outliers and extrapolation compared to PLS, Ridge, CatBoost, and CNN-1D.
Thresholding and downsampling effectively mitigate class imbalance in PFNs for tabular classification due to their calibration and limited-data strengths.
citing papers explorer
-
STRABLE: Benchmarking Tabular Machine Learning with Strings
A new corpus of 108 mixed string-numeric tables shows that advanced tabular learners with basic string embeddings perform well on most real-world data, while large LLM encoders help on free-text heavy tables.
-
What learning algorithm is in-context learning? Investigations with linear models
Transformers performing in-context learning implicitly implement gradient descent, ridge regression, and least-squares predictors for linear models, with behavior shifting based on model depth, width, and data noise.
-
Probing Memorization of Tabular In-Context Learning
A new probing framework detects moderate parametric memorization signals in tabular in-context learning models under single-task fine-tuning, strongest on low-cardinality tasks, but signals largely disappear under realistic training.
-
SurvivalPFN: Amortizing Survival Prediction via In-Context Bayesian Inference
SurvivalPFN amortizes Bayesian survival analysis for right-censored data by pretraining a prior-data fitted network on synthetic identifiable DGPs and then performing in-context inference, achieving competitive results on 61 real datasets.
-
Retrieval Augmented Time Series Forecasting
The paper proposes Retrieval Augmented Forecasting (RAF) that augments time-series foundation models with retrieved similar series to improve forecasting accuracy across domains.
-
TimEE: End-to-end Time Series Classification via In-Context Learning
A 4.5M-parameter transformer meta-trained on synthetic VARX-generated classification tasks achieves state-of-the-art ROC AUC on the UCR time series classification benchmark via in-context learning with no per-dataset training.
-
Efficient Adaptive Data Acquisition via Pretrained Belief Representations
POLAR uses pretrained predictive foundation models as fixed belief-state encoders and trains only a lightweight policy head on top for amortised Bayesian experimental design, optimisation, and active learning.
-
Bounded Context Management for Tabular Foundation Models on Stream Learning
CURE is a context management policy using entropy-gated admission and redundancy-aware eviction that delivers up to 27% relative improvement over classical stream learners on seven streams while remaining robust across TFM backbones.
-
In-Context Learning for Latent Space Bayesian Optimization
Complementing tabular foundation model pretraining with LSBO-specific synthetic tasks and a regularizer yields strong performance on held-out molecular optimization benchmarks.
-
Foundation Models for Credit Risk Prediction: A Game Changer?
Tabular foundation models, used zero-shot, match or beat tuned gradient boosting on average in credit PD and LGD benchmarks, with a larger edge on small datasets.
-
ADKO: Agentic Decentralized Knowledge Optimization
ADKO is a decentralized framework where agents share compact GP-derived tokens and LM insights to achieve collaborative Bayesian optimization with a decomposed regret bound that includes compression and approximation losses.
-
Pre-trained Tabular Foundation Models as Versatile Summary Networks for Neural Posterior Estimation
Pre-trained TabPFN acts as an effective training-free summary network for neural posterior estimation, matching or outperforming standard methods while preserving useful marginal and location information in the posterior.
-
Decoupled PFNs: Identifiable Epistemic-Aleatoric Decomposition via Structured Synthetic Priors
Decoupled PFNs use controllable synthetic priors to train separate latent-signal and noise heads, making epistemic-aleatoric decomposition identifiable and improving acquisition in noisy settings.
-
Tabular foundation models for in-context prediction of molecular properties
Tabular foundation models achieve high accuracy in molecular property prediction through in-context learning, with up to 100% win rates on MoleculeACE tasks when paired with CheMeleon embeddings.
-
Simulation as Supervision: Mechanistic Pretraining for Scientific Discovery
SGNNs pretrain neural networks on synthetic corpora from multiple mechanistic models and noise levels to enable robust forecasting and back-to-simulation attribution across epidemiology, ecology, and other fields.
-
Amortized Inference of Causal Models via Conditional Fixed-Point Iterations
Amortized transformer model with conditional fixed-point iterations learns SCM causal mechanisms from data and graphs, matching per-dataset baselines and outperforming in low-data regimes.
-
Are Tabular Foundation Models Robust to Realistic Query Distribution Shifts in Microbiome Data?
Tabular foundation models for gut microbiome data show degraded performance under perturbations like zero-imputation and sparsification even when discriminative taxa are retained, with zero-imputation most harmful and greater sensitivity than random forests.
-
Reinforcement Learning Foundation Models Should Already Be A Thing
A Graph Attention Network pretrained solely on synthetic MDPs solves held-out tabular RL benchmarks in context, outperforming UCB-VI and Q-learning online while matching VI-LCB offline.
-
Modular Multimodal Classification Without Fine-Tuning: A Simple Compositional Approach
CoMET achieves strong multimodal classification performance by composing frozen modality encoders, PCA compression, and tabular foundation models without any training, reaching state-of-the-art on diverse benchmarks including large-scale hierarchical tasks.
-
Tabular foundation models for robust calibration of near-infrared chemical sensing data
Preprocessing-optimized TabPFN achieves top average rank in regression on 66 NIR datasets and remains competitive on outliers and extrapolation compared to PLS, Ridge, CatBoost, and CNN-1D.
-
Correcting Class Imbalance in Prior-Data Fitted Networks for Tabular Classification
Thresholding and downsampling effectively mitigate class imbalance in PFNs for tabular classification due to their calibration and limited-data strengths.