Pith. sign in

REVIEW 4 cited by

Fine-tuned In-Context Learning Transformers are Excellent Tabular Data Classifiers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.13396 v2 pith:CYQ3QSIZ submitted 2024-05-22 cs.LG stat.ML

Felix den Breejen , Sangmin Bae , Stephen Cha , Se-Young Yun This is my paper
classification cs.LGstat.ML
keywords fine-tuningperformancedatasetscomplexdatadatasettabpfnboundaries
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The recently introduced TabPFN pretrains an In-Context Learning (ICL) transformer on synthetic data to perform tabular data classification. In this work, we extend TabPFN to the fine-tuning setting, resulting in a significant performance boost. We also discover that fine-tuning enables ICL-transformers to create complex decision boundaries, a property regular neural networks do not have. Based on this observation, we propose to pretrain ICL-transformers on a new forest dataset generator which creates datasets that are unrealistic, but have complex decision boundaries. TabForest, the ICL-transformer pretrained on this dataset generator, shows better fine-tuning performance when pretrained on more complex datasets. Additionally, TabForest outperforms TabPFN on some real-world datasets when fine-tuning, despite having lower zero-shot performance due to the unrealistic nature of the pretraining datasets. By combining both dataset generators, we create TabForestPFN, an ICL-transformer that achieves excellent fine-tuning performance and good zero-shot performance.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Algorithmic Recourse of In-Context Learning for Tabular Data

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    The paper delivers the first theoretical analysis and practical zeroth-order framework for algorithmic recourse under in-context learning for tabular prediction.

  2. PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models

    cs.DB 2026-02 conditional novelty 7.0 of 10

    PLUREL synthetic relational databases produce power-law scaling for Relational Transformer pretraining and, combined with real-data continued pretraining, improve zero-shot forecasting.

  3. Real-TabPFN: Improving Tabular Foundation Models via Continued Pre-training With Real-World Data

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Continuing the pre-training of TabPFN on 71 curated real-world tables raises its average normalized ROC-AUC from 0.954 to 0.976 on 29 AutoML Benchmark datasets.

  4. Fabrication of nano-diamonds with a single NV center: Towards matter-wave interferometry with massive objects

    quant-ph 2025-08 unverdicted novelty 3.0 of 10

    The authors describe design considerations and fabrication of 40x65x80 nm nanodiamond pillars with single NV centers as a step toward matter-wave interferometry of massive objects.

Pith tools