REVIEW 24 cited by
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Why do tree-based models still outperform deep learning on tabular data?
read the original abstract
While deep learning has enabled tremendous progress on text and image datasets, its superiority on tabular data is not clear. We contribute extensive benchmarks of standard and novel deep learning methods as well as tree-based models such as XGBoost and Random Forests, across a large number of datasets and hyperparameter combinations. We define a standard set of 45 datasets from varied domains with clear characteristics of tabular data and a benchmarking methodology accounting for both fitting models and finding good hyperparameters. Results show that tree-based models remain state-of-the-art on medium-sized data ($\sim$10K samples) even without accounting for their superior speed. To understand this gap, we conduct an empirical investigation into the differing inductive biases of tree-based models and Neural Networks (NNs). This leads to a series of challenges which should guide researchers aiming to build tabular-specific NNs: 1. be robust to uninformative features, 2. preserve the orientation of the data, and 3. be able to easily learn irregular functions. To stimulate research on tabular architectures, we contribute a standard benchmark and raw data for baselines: every point of a 20 000 compute hours hyperparameter search for each learner.
Forward citations
Cited by 24 Pith papers
-
TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
TabPFN is a Prior-Data Fitted Network that approximates Bayesian inference for small tabular classification by training a Transformer once on synthetic data drawn from a causal prior, then solves new tasks in a single...
-
Learning Dynamic Stability Landscapes in Synchronization Networks
Introduces graph-to-image prediction of per-node dynamic stability landscapes in oscillator networks from topology, releases two 10k-graph datasets, and shows GNN-CNN models achieve good accuracy with cross-size gener...
-
Data Language Models: A New Foundation Model Class for Tabular Data
Schema-1 is the first Data Language Model that natively understands raw tabular data and outperforms gradient-boosted ensembles, AutoML, and prior tabular foundation models on row-level prediction and imputation tasks.
-
Reciprocal Co-Training (RCT): Coupling Gradient-Based and Non-Differentiable Models via Reinforcement Learning
RCT couples an LLM and Random Forest via RL feedback so each augments the other's features and rewards, producing consistent gains on three medical datasets.
-
AXIL: Exact Instance Attribution for Gradient Boosting
AXIL computes exact fixed-structure instance attributions for squared-error GBMs via a matrix-free O(TN) backward operator, outperforming BoostIn/TREX/LeafInfluence on 20 regression datasets.
-
Solve for the Hyperparameter, Skip the Search: Kolmogorov-Optimal Scaling Laws for Spline Regression
Kolmogorov n-width theory plus PRESS statistics yield closed-form optimal spline resolution; KORE estimates bias/noise scales from two pilots and matches CV performance with far fewer fits.
-
When, Where, and How: Adaptive Binning for Tabular Self-Supervised Learning
Adaptive Binning improves tabular SSL by coupling feature discretization to training via representation-aware curriculum learning and a heterogeneity-aware objective, yielding gains on medical datasets without per-dat...
-
The Chandra-Gaia Catalog of Counterparts: Resolving ambiguous Gaia matches to X-ray sources in the Chandra Source Catalog using Machine Learning
A LightGBM classifier trained on NWAY Bayesian matches identifies true Chandra-Gaia counterparts for 113k X-ray sources, flags 7k ambiguous cases, and attributes half of 20k separation-only matches to chance coinciden...
-
LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models
LimiX-2M outperforms larger TabPFN-v2 and TabICL models on tabular benchmarks by expanding scalars into RBF features and using a reordered S->N->F attention block.
-
ASD-Bench: A Four-Axis Comprehensive Benchmark of AI Models for Autism Spectrum Disorder
ASD-Bench evaluates 17 ML and deep learning models on 4,068 AQ-10 records across child, adolescent, and adult cohorts, showing high adult performance, harder adolescent classification, shifting feature importance, and...
-
Prior-Aligned Data Cleaning for Tabular Foundation Models
L2C2 is a deep RL framework that learns to clean tabular data by aligning it to the synthetic prior of tabular foundation models, yielding higher accuracy on some benchmarks and cross-dataset policy transfer.
-
UniRec: Unified Multimodal Encoding for LLM-Based Recommendations
UniRec unifies heterogeneous recommendation modalities via specialized encoders, triplet representations, and hierarchical modeling to outperform prior multimodal LLM recommenders by up to 15% on benchmarks.
-
Filtering Interlopers with Photometry and Diagnostic Features: A Machine Learning Framework Validated with CSST Slitless Spectroscopy
XGBoost classifier filters interlopers in CSST slitless spectroscopy simulations, retaining 42% of galaxies with 96.6% accurate redshifts and 0.13% outliers.
-
TabICL: A Tabular Foundation Model for In-Context Learning on Large Data
TabICL scales in-context learning to large tabular data via column-then-row attention for row embeddings followed by a transformer, matching TabPFNv2 speed and performance while outperforming it and CatBoost on datase...
-
Weakly supervised machine learning for model-agnostic searches of new phenomena in the $\gamma$-ray sky
Weakly supervised classifiers trained on background-versus-mixture samples can identify anomalous gamma-ray sources without labeled signal templates, approaching supervised performance in controlled benchmarks.
-
Random-Effects Algorithm for Random Objects in Metric Spaces
A Fréchet-based random-effects algorithm with M-estimation consistency guarantees is proposed for modeling non-Euclidean random objects in general metric spaces.
-
Gradient Boosted Risk Scores
Gradient boosting produces risk scores with competitive accuracy but 60% fewer rules on classification tasks and 16% fewer on time-to-event tasks than regression-based methods like AutoScore.
-
A Nationwide Japanese Medical Claims Foundation Model: Balancing Model Scaling and Task-Specific Computational Efficiency
Scaling experiments on structured medical claims data show task-dependent saturation: disease incidence prediction benefits from models up to 101M parameters while medication prediction saturates at 11M, with all mode...
-
Who Audits the Auditor? Tamper-Proof Fraud Detection with Blockchain-Anchored Explainable ML
A blockchain-anchored explainable ML system delivers tamper-evident fraud detection with F1 of 0.895 and sub-25ms latency on Layer-2 networks.
-
Kitchen Sink Anomaly Detection
A combined kitchen sink observable set of Energy Flow Polynomials and subjettiness variables outperforms standard baselines in sensitivity to a wide range of resonant signals, with new public benchmarks released and a...
-
Cooperative Coevolution versus Monolithic Evolutionary Search for Semi-Supervised Tabular Classification
Cooperative coevolution and monolithic evolution achieve similar performance gains over baselines in low-label semi-supervised tabular classification.
-
Optimizing IoT Intrusion Detection with Tabular Foundation Models for Smart City Forensics
TabPFNv2.5 delivers 40x faster inference than Random Forest at 97% binary accuracy on TON IoT data, enabling a hybrid pipeline for real-time IoT threat screening in smart cities.
-
Modelling convective cell occurrence in proximity to cold fronts using extreme gradient boosting
An XGBoost model reproduces convective cell frequency near cold fronts with high skill but underestimates counts at the surface front, depending most on CAPE and time of day.
-
Machine Learning-Based Bitcoin Trading Under Transaction Costs: Evidence From Walk-Forward Forecasting
Cost-aware execution filters enable selected machine learning strategies, particularly long-only XGBoost, to achieve over 65% annualized returns and Sharpe ratios above 1 in hourly BTC trading despite 10bp costs.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.