TabPFN is a Prior-Data Fitted Network that approximates Bayesian inference for small tabular classification by training a Transformer once on synthetic data drawn from a causal prior, then solves new tasks in a single forward pass without further updates.
hub
Why do tree-based models still outperform deep learning on tabular data?
24 Pith papers cite this work, alongside 147 external citations. Polarity classification is still indexing.
24
Pith papers citing it
147
external citations · Pith
abstract
While deep learning has enabled tremendous progress on text and image datasets, its superiority on tabular data is not clear. We contribute extensive benchmarks of standard and novel deep learning methods as well as tree-based models such as XGBoost and Random Forests, across a large number of datasets and hyperparameter combinations. We define a standard set of 45 datasets from varied domains with clear characteristics of tabular data and a benchmarking methodology accounting for both fitting models and finding good hyperparameters. Results show that tree-based models remain state-of-the-art on medium-sized data ($\sim$10K samples) even without accounting for their superior speed. To understand this gap, we conduct an empirical investigation into the differing inductive biases of tree-based models and Neural Networks (NNs). This leads to a series of challenges which should guide researchers aiming to build tabular-specific NNs: 1. be robust to uninformative features, 2. preserve the orientation of the data, and 3. be able to easily learn irregular functions. To stimulate research on tabular architectures, we contribute a standard benchmark and raw data for baselines: every point of a 20 000 compute hours hyperparameter search for each learner.
hub tools
citation-role summary
background 4
citation-polarity summary
fields
cs.LG 11 cs.CR 2 astro-ph.CO 1 astro-ph.HE 1 astro-ph.IM 1 cs.AI 1 cs.CL 1 cs.IR 1 cs.NE 1 hep-ph 1roles
background 4polarities
background 4representative citing papers
Introduces graph-to-image prediction of per-node dynamic stability landscapes in oscillator networks from topology, releases two 10k-graph datasets, and shows GNN-CNN models achieve good accuracy with cross-size generalization.
Schema-1 is the first Data Language Model that natively understands raw tabular data and outperforms gradient-boosted ensembles, AutoML, and prior tabular foundation models on row-level prediction and imputation tasks.
AXIL computes exact fixed-structure instance attributions for squared-error GBMs via a matrix-free O(TN) backward operator, outperforming BoostIn/TREX/LeafInfluence on 20 regression datasets.
Kolmogorov n-width theory plus PRESS statistics yield closed-form optimal spline resolution; KORE estimates bias/noise scales from two pilots and matches CV performance with far fewer fits.
Adaptive Binning improves tabular SSL by coupling feature discretization to training via representation-aware curriculum learning and a heterogeneity-aware objective, yielding gains on medical datasets without per-dataset tuning.
A LightGBM classifier trained on NWAY Bayesian matches identifies true Chandra-Gaia counterparts for 113k X-ray sources, flags 7k ambiguous cases, and attributes half of 20k separation-only matches to chance coincidences, validated at 95% on COUP without positional features.
LimiX-2M outperforms larger TabPFN-v2 and TabICL models on tabular benchmarks by expanding scalars into RBF features and using a reordered S->N->F attention block.
ASD-Bench evaluates 17 ML and deep learning models on 4,068 AQ-10 records across child, adolescent, and adult cohorts, showing high adult performance, harder adolescent classification, shifting feature importance, and dissociation between accuracy and calibration.
L2C2 is a deep RL framework that learns to clean tabular data by aligning it to the synthetic prior of tabular foundation models, yielding higher accuracy on some benchmarks and cross-dataset policy transfer.
UniRec unifies heterogeneous recommendation modalities via specialized encoders, triplet representations, and hierarchical modeling to outperform prior multimodal LLM recommenders by up to 15% on benchmarks.
XGBoost classifier filters interlopers in CSST slitless spectroscopy simulations, retaining 42% of galaxies with 96.6% accurate redshifts and 0.13% outliers.
TabICL scales in-context learning to large tabular data via column-then-row attention for row embeddings followed by a transformer, matching TabPFNv2 speed and performance while outperforming it and CatBoost on datasets over 10K samples.
Reciprocal co-training links an LM and a random forest via RL so LM embeddings enrich the forest and forest probabilities reward LM updates, yielding gains on three medical datasets.
Weakly supervised classifiers trained on background-versus-mixture samples can identify anomalous gamma-ray sources without labeled signal templates, approaching supervised performance in controlled benchmarks.
A Fréchet-based random-effects algorithm with M-estimation consistency guarantees is proposed for modeling non-Euclidean random objects in general metric spaces.
Gradient boosting produces risk scores with competitive accuracy but 60% fewer rules on classification tasks and 16% fewer on time-to-event tasks than regression-based methods like AutoScore.
Scaling experiments on structured medical claims data show task-dependent saturation: disease incidence prediction benefits from models up to 101M parameters while medication prediction saturates at 11M, with all models outperforming a LightGBM baseline.
A blockchain-anchored explainable ML system delivers tamper-evident fraud detection with F1 of 0.895 and sub-25ms latency on Layer-2 networks.
A combined kitchen sink observable set of Energy Flow Polynomials and subjettiness variables outperforms standard baselines in sensitivity to a wide range of resonant signals, with new public benchmarks released and an attribute bagging variant reducing training cost.
Cooperative coevolution and monolithic evolution achieve similar performance gains over baselines in low-label semi-supervised tabular classification.
TabPFNv2.5 delivers 40x faster inference than Random Forest at 97% binary accuracy on TON IoT data, enabling a hybrid pipeline for real-time IoT threat screening in smart cities.
An XGBoost model reproduces convective cell frequency near cold fronts with high skill but underestimates counts at the surface front, depending most on CAPE and time of day.
Cost-aware execution filters enable selected machine learning strategies, particularly long-only XGBoost, to achieve over 65% annualized returns and Sharpe ratios above 1 in hourly BTC trading despite 10bp costs.
citing papers explorer
-
TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
TabPFN is a Prior-Data Fitted Network that approximates Bayesian inference for small tabular classification by training a Transformer once on synthetic data drawn from a causal prior, then solves new tasks in a single forward pass without further updates.
-
Learning Dynamic Stability Landscapes in Synchronization Networks
Introduces graph-to-image prediction of per-node dynamic stability landscapes in oscillator networks from topology, releases two 10k-graph datasets, and shows GNN-CNN models achieve good accuracy with cross-size generalization.
-
Data Language Models: A New Foundation Model Class for Tabular Data
Schema-1 is the first Data Language Model that natively understands raw tabular data and outperforms gradient-boosted ensembles, AutoML, and prior tabular foundation models on row-level prediction and imputation tasks.
-
AXIL: Exact Instance Attribution for Gradient Boosting
AXIL computes exact fixed-structure instance attributions for squared-error GBMs via a matrix-free O(TN) backward operator, outperforming BoostIn/TREX/LeafInfluence on 20 regression datasets.
-
Solve for the Hyperparameter, Skip the Search: Kolmogorov-Optimal Scaling Laws for Spline Regression
Kolmogorov n-width theory plus PRESS statistics yield closed-form optimal spline resolution; KORE estimates bias/noise scales from two pilots and matches CV performance with far fewer fits.
-
When, Where, and How: Adaptive Binning for Tabular Self-Supervised Learning
Adaptive Binning improves tabular SSL by coupling feature discretization to training via representation-aware curriculum learning and a heterogeneity-aware objective, yielding gains on medical datasets without per-dataset tuning.
-
The Chandra-Gaia Catalog of Counterparts: Resolving ambiguous Gaia matches to X-ray sources in the Chandra Source Catalog using Machine Learning
A LightGBM classifier trained on NWAY Bayesian matches identifies true Chandra-Gaia counterparts for 113k X-ray sources, flags 7k ambiguous cases, and attributes half of 20k separation-only matches to chance coincidences, validated at 95% on COUP without positional features.
-
LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models
LimiX-2M outperforms larger TabPFN-v2 and TabICL models on tabular benchmarks by expanding scalars into RBF features and using a reordered S->N->F attention block.
-
ASD-Bench: A Four-Axis Comprehensive Benchmark of AI Models for Autism Spectrum Disorder
ASD-Bench evaluates 17 ML and deep learning models on 4,068 AQ-10 records across child, adolescent, and adult cohorts, showing high adult performance, harder adolescent classification, shifting feature importance, and dissociation between accuracy and calibration.
-
Prior-Aligned Data Cleaning for Tabular Foundation Models
L2C2 is a deep RL framework that learns to clean tabular data by aligning it to the synthetic prior of tabular foundation models, yielding higher accuracy on some benchmarks and cross-dataset policy transfer.
-
UniRec: Unified Multimodal Encoding for LLM-Based Recommendations
UniRec unifies heterogeneous recommendation modalities via specialized encoders, triplet representations, and hierarchical modeling to outperform prior multimodal LLM recommenders by up to 15% on benchmarks.
-
Filtering Interlopers with Photometry and Diagnostic Features: A Machine Learning Framework Validated with CSST Slitless Spectroscopy
XGBoost classifier filters interlopers in CSST slitless spectroscopy simulations, retaining 42% of galaxies with 96.6% accurate redshifts and 0.13% outliers.
-
TabICL: A Tabular Foundation Model for In-Context Learning on Large Data
TabICL scales in-context learning to large tabular data via column-then-row attention for row embeddings followed by a transformer, matching TabPFNv2 speed and performance while outperforming it and CatBoost on datasets over 10K samples.
-
Reciprocal Co-Training (RCT): Coupling Gradient-Based and Non-Differentiable Models via Reinforcement Learning
Reciprocal co-training links an LM and a random forest via RL so LM embeddings enrich the forest and forest probabilities reward LM updates, yielding gains on three medical datasets.
-
Weakly supervised machine learning for model-agnostic searches of new phenomena in the $\gamma$-ray sky
Weakly supervised classifiers trained on background-versus-mixture samples can identify anomalous gamma-ray sources without labeled signal templates, approaching supervised performance in controlled benchmarks.
-
Random-Effects Algorithm for Random Objects in Metric Spaces
A Fréchet-based random-effects algorithm with M-estimation consistency guarantees is proposed for modeling non-Euclidean random objects in general metric spaces.
-
Gradient Boosted Risk Scores
Gradient boosting produces risk scores with competitive accuracy but 60% fewer rules on classification tasks and 16% fewer on time-to-event tasks than regression-based methods like AutoScore.
-
A Nationwide Japanese Medical Claims Foundation Model: Balancing Model Scaling and Task-Specific Computational Efficiency
Scaling experiments on structured medical claims data show task-dependent saturation: disease incidence prediction benefits from models up to 101M parameters while medication prediction saturates at 11M, with all models outperforming a LightGBM baseline.
-
Who Audits the Auditor? Tamper-Proof Fraud Detection with Blockchain-Anchored Explainable ML
A blockchain-anchored explainable ML system delivers tamper-evident fraud detection with F1 of 0.895 and sub-25ms latency on Layer-2 networks.
-
Kitchen Sink Anomaly Detection
A combined kitchen sink observable set of Energy Flow Polynomials and subjettiness variables outperforms standard baselines in sensitivity to a wide range of resonant signals, with new public benchmarks released and an attribute bagging variant reducing training cost.
-
Cooperative Coevolution versus Monolithic Evolutionary Search for Semi-Supervised Tabular Classification
Cooperative coevolution and monolithic evolution achieve similar performance gains over baselines in low-label semi-supervised tabular classification.
-
Optimizing IoT Intrusion Detection with Tabular Foundation Models for Smart City Forensics
TabPFNv2.5 delivers 40x faster inference than Random Forest at 97% binary accuracy on TON IoT data, enabling a hybrid pipeline for real-time IoT threat screening in smart cities.
-
Modelling convective cell occurrence in proximity to cold fronts using extreme gradient boosting
An XGBoost model reproduces convective cell frequency near cold fronts with high skill but underestimates counts at the surface front, depending most on CAPE and time of day.
-
Machine Learning-Based Bitcoin Trading Under Transaction Costs: Evidence From Walk-Forward Forecasting
Cost-aware execution filters enable selected machine learning strategies, particularly long-only XGBoost, to achieve over 65% annualized returns and Sharpe ratios above 1 in hourly BTC trading despite 10bp costs.