Pith. sign in

REVIEW 9 cited by

Large Scale Transfer Learning for Tabular Data via Language Modeling

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.12031 v2 pith:7XSJ2CHZ submitted 2024-06-17 cs.LG cs.AIcs.CL

Josh Gardner , Juan C. Perdomo , Ludwig Schmidt This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords tabulardatalanguagepredictiondatasetslargemodelmodels
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Tabular data -- structured, heterogeneous, spreadsheet-style data with rows and columns -- is widely used in practice across many domains. However, while recent foundation models have reduced the need for developing task-specific datasets and predictors in domains such as language modeling and computer vision, this transfer learning paradigm has not had similar impact in the tabular domain. In this work, we seek to narrow this gap and present TabuLa-8B, a language model for tabular prediction. We define a process for extracting a large, high-quality training dataset from the TabLib corpus, proposing methods for tabular data filtering and quality control. Using the resulting dataset, which comprises over 2.1B rows from over 4M unique tables, we fine-tune a Llama 3-8B large language model (LLM) for tabular data prediction (classification and binned regression) using a novel packing and attention scheme for tabular prediction. Through evaluation across a test suite of 329 datasets, we find that TabuLa-8B has zero-shot accuracy on unseen tables that is over 15 percentage points (pp) higher than random guessing, a feat that is not possible with existing state-of-the-art tabular prediction models (e.g. XGBoost, TabPFN). In the few-shot setting (1-32 shots), without any fine-tuning on the target datasets, TabuLa-8B is 5-15 pp more accurate than XGBoost and TabPFN models that are explicitly trained on equal, or even up to 16x more data. We release our model, code, and data along with the publication of this paper.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Algorithmic Recourse of In-Context Learning for Tabular Data

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    The paper delivers the first theoretical analysis and practical zeroth-order framework for algorithmic recourse under in-context learning for tabular prediction.

  2. Basis Transformers for Multi-Task Tabular Regression

    cs.LG 2025-06 conditional novelty 7.0 of 10

    Basis transformers beat fine-tuned LLMs on 34 multi-task tabular regression datasets while using five times fewer parameters and no data preprocessing.

  3. Large Language Models are Good Relational Learners

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Rel-LLM combines a GNN encoder with a frozen LLM via soft prompts and masked attribute pretraining, reporting improved average performance on RelBench relational database tasks.

  4. CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation

    cs.LG 2025-06 conditional novelty 6.0 of 10

    CACTI combines median-truncated copy masking with language-model column embeddings to improve tabular imputation accuracy across MCAR, MAR, and MNAR missingness.

  5. Preserving AUC Fairness in Learning with Noisy Protected Groups

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A distributionally robust AUC-fairness method with TV-distance bounds that preserves group fairness when protected-group labels are noisy.

  6. Zero-shot Meta-learning for Tabular Prediction Tasks with Adversarially Pre-trained Transformer

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Adversarially pre-trained transformer (APT) matches top gradient-boosting models on 35 small tabular classification benchmarks and improves on TabPFN in regression, while handling datasets with any number of classes v...

  7. TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems

    cs.LG 2025-02 conditional novelty 6.0 of 10

    BETA augments TabPFN with encoder fine-tuning and bagging to reduce both bias and variance, achieving SOTA accuracy on 200+ tabular benchmarks while scaling to larger and higher-dimensional data.

  8. TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments

    cs.LG 2025-01 conditional novelty 6.0 of 10

    TabFSBench simulates feature removal in tabular data across four scenarios and 24 models, reporting that most models degrade, tabular LLMs are most robust, and that closed-environment performance correlates with featu...

  9. Tackling prediction tasks in relational databases with LLMs

    cs.LG 2024-11 conditional novelty 5.0 of 10

    Pre-trained LLMs, fed serialized relational rows with related examples, achieve competitive AUROC/MAE on RelBench without fine-tuning, but the headline comparison is weakened by pretraining contamination on Formula 1 tasks.

Pith tools